The prevalence of motion sickness among passengers using personal electronic devices, such as smartphones, during vehicle journeys has become a growing concern. This issue is expected to intensify with the increasing adoption of assistant or automated driving functions, which may lead to non-driving tasks (NDT) being performed by all on-board passengers, including the user in the “driver” seat during conditionally or fully automated driving modes. This trend presents challenges related to motion sickness, particularly in terms of specific performance requirements for non-driving tasks. In response to the need to alleviate passenger motion sickness, we have developed an easy-to-understand animation cue app that can be conveniently implemented on smartphones. The motion cue conveys information about vehicle accelerations, including their directions and magnitudes, using the metaphors of traffic signal colors and backward-moving lane lines, either in straight or curved lane driving. Following several rounds of improvements based on moving-base simulator and real car experiments, finally a successful cue design was found, which could significantly alleviate motion sickness of passengers while engaging in NDT, with minimal impact on their NDT performances. However, the study also revealed limitations to the sickness-alleviating capability of our motion cue design, including potential lack of universal acceptance among different users and reduced effectiveness in severely uncomfortable driving conditions. This work may provide valuable insights for further visual cue improvements that can contribute in future carsickness-proof vehicles.

A robotic locker system is needed where automated storage and retrieval of items are required without the need for staff presence. For example, a robot can provide 7/24 available services of medical items pick-up and return, during the COVID-19 pandemic (or under other emergencies). A robotic locker system is usually equipped with a user-friendly intuitive interface (e.g., a touchscreen); meanwhile, the robot desires a multitask agent that can observe, understand, and operate the locker’s interface to complete many tasks of storing/accessing/shipping items. In this article, we study building a multitask agent for interacting with robotic locker systems, called AgentMario. Without human intervention for a specific task, AgentMario decomposes solving a task into learning basic skills (states or user interfaces) and planning over the skills (finding the next state/interface). When the agent is solving a task, our search algorithm walks on the finite state machine graph and generates the proper plans (operation sequence) for the agent. In experiments, our method accomplishes four diverse tasks of picking-up/storing/dropping-off/shipping items. By employing image recognition and mechanical automation technologies, we implement AgentMario with a robot arm to enable contactless operation over the locker’s interface. Experimental results show that our method outperforms baselines in most tasks by a large margin.

Optical see-through head-mounted displays (OHMDs) blend digital content with the physical world, presenting unique color management challenges. Previous literature suggests using green as the main color, but this severely limits creative freedom. To address this, we conducted an empirical study with 30 participants, evaluating 216 colors under various OHMD usage conditions. Based on the results, we propose color guidelines indicating each hue's clear and comfortable saturation and brightness ranges, along with clarity and comfort scores across hues for different devices and lighting conditions. Our color guidelines expand the usable color palette, offering designers a wider range of color options. These guidelines were used and iteratively refined through feedback in a workshop with 12 designers, integrating them into practical design workflows. The resulting comprehensive color guide provides a valuable resource for OHMD interface designers, enhancing both the aesthetic possibilities and functional effectiveness of augmented reality experiences.

Technology integration in educational settings has led to the development of novel sensor-based tools that enable students to measure and interact with their environment. Although reports from using such tools can be positive, evaluations are often conducted under controlled conditions and short timeframes. There is a need for longitudinal data collected in realistic classroom settings. However, sustained and authentic classroom use requires technology platforms to be seen by teachers as both easy to use and of value. We describe our development of a sensor-based platform to support science teaching that followed a 14-month design process. We share insights from this design and development approach, and report findings from a six-month large-scale evaluation involving 35 schools and 1245 students. We share lessons learnt, including that technology integration is not an educational goal per se and that technology should be a transparent tool to enable students to achieve their learning goals.

We present ANISMA, a software and hardware toolkit to prototype on-skin haptic devices that generate skin deformation stimuli like pressure, stretch, and motion using shape-memory alloys (SMAs). Our toolkit embeds expert knowledge that makes SMA spring actuators more accessible to human–computer interaction (HCI) researchers. Using our software tool, users can design different actuator layouts, program their spatio-temporal actuation and preview the resulting deformation behavior to verify a design at an early stage. Our toolkit allows exporting the actuator layout and 3D printing it directly on skin adhesive. To test different actuation sequences on the skin, a user can connect the SMA actuators to our customized driver board and reprogram them using our visual programming interface. We report a technical analysis, verify the perceptibility of essential ANISMA skin deformation devices with 8 participants, and evaluate ANISMA regarding its usability and supported creativity with 12 HCI researchers in a creative design task.

The sense of touch offers interesting possibilities as a robust and ubiquitous communication channel. In this paper, we present OM, a tool that enables users to design subjective vibrotactile expressions associated with contextualised information relevant to them. OM consists of a pair of wrist-worn devices that can reproduce complex vibrotactile symbols and a companion editor smartphone app that allows users to create, customise and store personalised expressions. We studied OM in real-world contexts by allowing 13 participants to explore the functionalities of OM throughout their daily interactions with complete autonomy. We highlight relevant scenarios, design considerations, and future directions towards a tool that can help people unveil an alternative, ubiquitous and private communication system accessible to all.

With voice user interfaces (VUIs) becoming ubiquitous and speech synthesis technology maturing, it is possible to synthesise voices to resemble our friends and relatives (which we will collectively call 'kin') and use them on VUIs. However, designing such interfaces and investigating how the familiarity of kin voices affect user perceptions remain under-explored. Our surveys and interviews with 25 users revealed that VUIs using kin voices were perceived as more engaging, persuasive and safer yet eerier than VUIs using common virtual assistant voices. We then developed a technology probe, KinVoice, an Alexa-based VUI that was deployed in three households over two weeks. Users set reminders using KinVoice, which in turn, gave the reminders in synthesised kin voices. This was to explore users' needs, uncover challenges involved and inspire new applications. We discuss design guidelines for integrating familiar kin voices into VUIs, applications that benefit from its usage, and implications for balancing voice realism and usability with security and diversification.

Speech Emotion Recognition (SER) has been a challenging task on which researchers have been working for decades. Recently, Deep Learning (DL) based approaches have been shown to perform well in SER tasks; however, it has been noticed that their superior performance is limited to the distribution of the data used to train the model. In this paper, we present an analysis of using autoencoders to improve the generalisability of DL based SER solutions. We train a sparse autoencoder using a large speech corpus extracted from social media. Later, the trained encoder part of the autoencoder is reused as the input to a long short-term memory (LSTM) network, and the encoder-LSTM modal is re-trained on an aggregation of five commonly used speech emotion corpora. Our evaluation uses an unseen corpus in the training & validation stages to simulate 'in the wild' condition and analyse the generalisability of our solution. A performance comparison is carried out between the encoder based model and a model trained without an encoder. Our results show that the autoencoder based model improves the unweighted accuracy of the unseen corpus by 8%, indicating autoencoder based pre-training can improve the generalisability of DL based SER solutions.

We present MAGHair, a novel wearable technique that provides subtle haptic sensation by stimulating the body hair without touching the skin. Our approach builds on previous research in magnetic hair stimulation and magnetic locomotion. We use magnetic cosmetics to augment the body hair, which can then be stimulated by a wearable apparatus that combines electromagnets and permanent magnets. We provide technical insights on the implementation of a fully functional wrist-worn form factor and early adaptations into other form factors. In addition, we provide a workflow for evaluating and characterizing the magnetic cosmetic recipes. Finally, we evaluate MAGHair, which demonstrated that users could detect the sensation of hair movement that they described as gentle and unique.

In this paper, we present M-Hair, a novel method for providing tactile feedback by stimulating only the body hair without touching the skin. It works by applying passive magnetic materials to the body hair, which is actuated by external magnetic fields. Our user study suggested that the value of the M-hair mechanism is in inducing affective sensations such as pleasantness, rather than effectively discriminating features such as shape, size, and direction. This work invites future research to use this method in applications that induce emotional responses or affective states, and as a research tool for investigations of this novel sensation.

Currently, smartwatches are equipped with Photoplethysmography (PPG) sensors to measure Heart Rate (HR) and Heart Rate Variability (HRV). However, PPG sensors consume considerably high energy, making it impractical to monitor HR & HRV continuously for an extended period. Utilising low power accelerometers to estimate HR has been broadly discussed in previous decades. Inspired by prior work, we introduce CompRate, an alternative method to measure HR continuously for an extended period in low-intensity physical activities. CompRate model calibrated for individual users only has an average performance of Root Mean Squared Error (RMSE) 1.58 Beats Per Minute (BPM). Further, CompRate used 3.75 times less energy compared to the built-in PPG sensor. We also demonstrate that CompRate model can be extended to predict HRV. We will demonstrate CompRate in several application scenarios: self-awareness of fatigue and just-in-time interruption while driving; enabling teachers to be aware of students’ mental effort during a learning activity; and the broadcasting of the location of live victims in a disaster situation.

Conventionally, keys on a physical keyboard have only two states: "released'' and "pressed''. As such, various techniques, such as hotkeys, are designed to enhance the keyboard expressiveness. Realizing that user inevitably perform touch actions during keystrokes, we propose GestAKey, leveraging location and motion of the touch on individual keycaps to augment the functionalities of existing keystrokes. With a log study, we collected touch data for both normal usage (typing and hotkeys) and while performing touch gestures (location and motion), which are analyzed to assess the viability of augmenting keystrokes with simultaneous gestures. A controlled experiment was conducted to compare GestAKey with existing keyboard interaction techniques, in terms of efficiency and learnability. The results show that GestAKey has comparable performance with hotkey. We further discuss the insights of integrating such touch modality into existing keyboard interaction, and demonstrate several usage scenarios.

Binocular disparity allows interesting visual effects visible only to people with stereoscopic 3D displays. Here, we studied and applied one such effect, binocular luster, to the application of digital colorblind aids with active shutter 3D. We developed two prototype techniques, ColorBless and PatternBless, to investigate the effectiveness of such aids and to explore the potential applications of a luster effect in stereoscopic 3D beyond highlighting. User studies and interviews revealed that luster-based aids were fast and required lower cognitive effort than existing aids and were preferred over other aids by the majority of colorblind participants. We infer design implications of a luster effect from the study and propose potential applications in augmented visualization.

Stroke gestures are intuitive and efficient but often require gesture-capable input hardware such as a touchscreen. In this paper, we present GestKeyboard, a novel technique for gesturing over an ordinary, unmodified physical keyboard that remains the major input modality for existing desktop and laptop computers. We discuss an exploratory study for understanding the design space of gesturing on a physical keyboard and our algorithms for detecting gestures in a modeless way, without interfering with the keyboard's major functionality such as text entry and shortcuts activation. We explored various features for detecting gestures from a keyboard event stream. Our experiment based on the data collected from 10 participants indicated it is feasible to reliably detect gestures from normal keyboard use, 95% detection accuracy within a maximum latency of 200ms.

Touch-sensitive surfaces have become a predominant input medium for computing devices. In particular, multitouch capability of these devices has given rise to developing rich interaction vocabularies for “real” direct manipulation of user interfaces. However, the richness and flexibility of touch interaction often comes with significant complexity for programming these behaviors. Particularly, finger touches, though intuitive, are imprecise and lead to ambiguity. Touch input often involves coordinated movements of multiple fingers as opposed to the single pointer of a traditional WIMP interface. It is challenging in not only detecting the intended motion carried out by these fingers but also in determining the target objects being manipulated due to multiple focus points. Currently, developers often need to build touch behaviors by dealing with raw touch events that is effort consuming and error-prone. In this article, we present Touch, a tool that allows developers to easily specify their desired touch behaviors by demonstrating them live on a touch-sensitive device or selecting them from a list of common behaviors. Developers can then integrate these touch behaviors into their application as resources and via an API exposed by our runtime framework. The integrated tool support enables developers to think and program optimistically about how these touch interactions should behave, without worrying about underlying complexity and technical details in detecting target behaviors and invoking application logic. We discuss the design of several novel inference algorithms that underlie these tool supports and evaluate them against a multitouch dataset that we collected from end users. We also demonstrate the usefulness of our system via an example application.

Researchers have explored a variety of technologies that enable a single display to simultaneously present different content when viewed from different angles or by different people. These displays provide new functionalities such as personalized views for multiple users, privacy protection, and stereoscopic 3D displays. However, current multi-view displays rely on special hardware, thus significantly limiting their availability to consumers and adoption in everyday scenarios. In this paper, we present a pure software solution (i.e. with no hardware modification) that allows us to present two independent views concurrently on the most widely used and affordable type of LCD screen, namely Twisted Nematic (TN). We achieve this by exploiting a technical limitation of the technology which causes these LCDs to show varying brightness and color depending on the viewing angle. We describe our technical solution as well as demonstrate example applications in everyday scenarios.

Human stereo vision processes the two different images seen by the two eyes to generate depth sensation. While current stereoscopic display technologies look at how to faithfully simulate the stereo viewing experience, we took a look out of this scope, to explore how we may present binocular image pairs that differ in other ways to create novel visual experience. This paper presents several interesting techniques we explored, and discusses their potential applications according to an informal user study.

Abstract One of the fundamental tasks for analytic activity is retrieving (i.e., reading) the value of a particular quantity in an information visualization. However, few previous studies have compared user performance in such value retrieval tasks for different visualizations. We present an experimental comparison of user performance (time and error distance) across four multivariate data visualizations. Three variants of scatterplot (SCP) visualizations, namely SCPs with common vertical axes (SCP‐common), SCPs with a staircase layout (SCP‐staircase), and SCPs with rotated axes between neighboring cells (SCP‐rotated), and a baseline parallel coordinate plots (PCP) were compared. Results show that the baseline PCP is better than SCP‐rotated and SCP‐staircase under all conditions, while the difference between SCP‐common and PCP depends on the dimensionality and density of the dataset. PCP shows advantages over SCP‐common when the dimensionality and density of the dataset are low, but SCP‐common eventually outperforms PCP as data dimensionality and density increase. The results suggest guidelines for the use of SCPs and PCPs that can benefit future researchers and practitioners.

Browsing the web has been shown to be a highly recurrent activity. Aimed to optimize the browsing experience, extensive previous research has been carried out on users' revisitation behavior. However, the conventional definition for revisitation, which only considers page loading activities by monitoring http requests initiated by the browser, largely underestimates users' intended revisitation activities with tabbed browsers. Thus, we introduce a goal-oriented definition and a refined revisitation measurement based on page viewings in tabbed browsers. An empirical analysis of statistics taken from a client-side log study showed that although the overall revisitation rate remained relatively constant, tabbed browsing has introduced new behaviors warrant future investigations.