Head-mounted displays increasingly require text entry that goes beyond character input to support real-world writing and editing. Existing HMD text entry research and systems focus primarily on entry speed and accuracy, while offering limited support for cursor control, navigation, and command execution. Midair and voice-based techniques address some needs but suffer from fatigue, limited affordances, and poor discoverability of editing functions. This work investigates one-handed thumb-to-finger text composition for HMDs, combining on-body input with hand-proximate visual interfaces that provide tactile feedback and spatial guidance. We map the design space of such systems across input, editing, command invocation, and visualization. Through participatory design workshops with 18 designers, we elicit concrete concepts for one-handed text composition on HMDs. From these artifacts, we derive five implementable design guidelines focused on discoverability and complex text composition.

Conventional Mixed Reality (MR) workspaces are frequently organized in cockpit-like layouts, where multiple floating windows surround the user. While this configuration facilitates access to digital content, it often induces occlusion, reducing understanding of the physical environment and limiting access to real-world objects. To overcome this challenge, we present the Contour-Adaptive Mixed Environment Overlays (CAMEO), a contour-adaptive MR interface that drapes virtual windows onto physical surfaces. This design integrates digital content with nearby items, thereby improving users’ visual access to background objects and supporting interaction with them. We evaluate CAMEO in two controlled studies. The first demonstrates that draping reduces hand-movement detours relative to flat mid-air surfaces, enabling more direct interaction with nearby items. The second shows that controlled window deformation does not significantly impair text legibility when compared to flat surfaces. Together, these findings contribute a novel design paradigm for MR workspaces that balances immersion, readability, and environmental understanding.

Shape-changing displays typically lose pixel density as surface area expands, limiting their usability. We introduce MorphSkein, a shape-changing after-image display that preserves initial density (1.44 px/cm 2 ) across naturally occurring axisymmetric shapes generated by spinning cables (troposkeins). The system uses a telescopic pole and four LED-strip rewinders on a rotating base. As the strips spin, centrifugal force forms troposkeins, and persistence of vision creates 360°-visible displays, while adjusting pole height and strip lengths changes their shape. As surface area grows, pixel density is preserved vertically by releasing new rows from the rewinders and horizontally by rendering extra columns per revolution with the strips. This keeps comparable density along the central horizontal line of the display, with naturally higher density toward the top and bottom where the troposkein curves inward. Because the technique relies on a mathematical model assuming ideal troposkein geometry, angular velocity becomes critical: incorrect speeds degrade pixel density accuracy, axisymmetric shape fidelity, or both. Interpolation of experimental data shows that 69.44% of reachable troposkein configurations achieve ⩾90% density accuracy and shape fidelity for at least one operating speed. Remaining cases degrade due to insufficient motor speed or limited MCU speed and LED refresh rate. Limitations and improvements are discussed.

Hand Proximate User Interfaces (HPUI) on Head Mounted Displays (HMD) leverage hand tracking to anchor content on the hand and interact with it using thumb-to-finger interactions. Similar to many other interaction techniques on HMDs, HPUI realizes these interactions by combining simple geometry in game engines. This, in turn, leads to accidental triggers, akin to the "fat-finger problem" on touch screens. To address this, we explore and provide insight into how the thumb’s surface interacts when using HPUI by approximating the thumb’s surface with a large number of raycasts. We observe that different regions of the thumb are used when interacting with different parts of the hand. The results also highlight the need to consider the temporal component. We then propose approaches to improving the precision of thumb-to-finger interactions on HPUI and show that these improve target selection accuracy with denser target layouts.

Attentional lapses while individuals are engaged in activities can have critical effects on their performance. EEG sensors offer the potential to monitor brain activity and detect such lapses in-situ. However, advances in automatic detection are limited, notably due to the scarcity of validated EEG data for training. In this work, we explore the design space of lapses-of-attention detectors using EEG signals, framing it as a binary classification problem. We introduce an EEG dataset with two validated attention levels acquired through a controlled experiment (N = 24) involving reading tasks with and without auditory distractions. We evaluated fifteen detectors using three different EEG feature extraction techniques, and five classifier models. Models using filterbank-CSP features yielded the highest median per-participant detection accuracy of 96%. Limited-resource analyses further indicate the Beta frequency band is the most informative for attention detection, and highlight detection can be achieved with only four EEG channels. Taken together, our findings inform on the feasibility and design of automatic attention detection for brain-computer interfaces utilizing simpler EEG devices.

Designing a comfortable, familiar, and efficient one-handed text entry method for Head-Mounted Displays (HMDs) remains a significant challenge. Existing midair typing systems induce fatigue, while novel techniques often demand extensive training or sacrifice input efficiency. Consequently, we introduce ThumbSwype , a novel thumb-to-finger text entry technique that adapts smartphone swipe typing for HMDs. Users see the traditional QWERTY keyboard overlaid on their index, middle, and ring fingers, allowing them to perform swipe gestures with their thumb to type words. In an evaluation study (N=16) , participants achieved a mean of 14.52 words per minute (WPM), which is 63.8% of their smartphone swipe-typing performance, with a peak average of 20.2 WPM. We compare ThumbSwype’s performance with related work, and discuss directions for future improvement.

Micro Data Videos (mDVs) are up to one-minute data-driven vertical video clips for mobile devices. Widely adopted on social media platforms, mDVs hold significant potential for disseminating information. Despite their growing prevalence, little is known about their components and how they are designed. Thus, two studies were conducted. Study 1 analyzed 40 mDVs and revealed their narrative components. Study 2 examined, through design sessions with design experts, how such components are assembled to craft storyboards for mDVs. The diverse narrative styles of mDVs render them flexible and suitable for multiple topics and purposes. Further, many include a “Linker" directing viewers to external online resources. Participants approached their design in a structural yet iterative manner with emphasis on setting up a “hook” in the opening seconds to capture attention. We summarize and share common design practices used in creating mDVs, an increasingly important medium in data storytelling.

Foot-based input can serve as a supplementary or alternative approach to text entry in virtual reality (VR). This work explores the feasibility and design of foot-based techniques that are hands-free. We first conducted a preliminary study to assess foot-based text entry in standing and seated positions with tap and swipe input approaches. The findings showed that foot-based text input was feasible, with the possibility for performance and usability improvements. We then developed three foot-based techniques, including two tap-based techniques (FeetSymTap and FeetAsymTap) and one swipe-based technique (FeetGestureTap), and evaluated their performance via another user study. The results show that the two tap-based techniques supported entry rates of 11.12 WPM and 10.80 WPM, while the swipe-based technique led to 9.16 WPM. Our findings provide a solid foundation for the future design and implementation of foot-based text entry in VR and have the potential to be extended to MR and AR.

Optical-see-through (OST) augmented reality headsets offer users the flexibility to access relevant data visualizations anytime and anywhere. However, the appearance of content displayed on OST displays varies in colour and transparency depending on the environment they are viewed in, potentially leading to interpretation challenges. We present the findings of a psychophysical study (N = 24), aimed at assessing the impact of two environmental factors – lighting intensity and background colour – on user performance and colour perception accuracy in a visualization and colour-matching task using an OST headset. Our results suggest the effect of background colour on visualization interpretation is notable only under bright lighting conditions. Interestingly, participants perceived low-colour-contrast scenarios as more challenging, although their performance did not decline. Additionally, visualization colours were perceptibly and distinctly mismatched, but did not blend with the background colours. Finally, we discuss visual comfort and colour coding in the context of designing ubiquitous visualizations on OST displays, highlighting open challenges.

We explore the value of Hand Proximate User Interfaces (HPUIs) for in-situ interactive head-mounted systems, i.e. systems designed to support a user’s primary activity. HPUIs are unencumbered, single-handed, and have tactile and proprioceptive affordances. This allows novice and expert users to rely on the visual cues available in the UI, but to then gradually interact eyes-free such that they minimize interrupting the user’s core task. Prior work on HPUI falls short of validating this premise. We address this gap with comparative analysis in the context of a compound task, where one hand is involved in a primary task and the second interacts with a secondary but supporting task (e.g. selecting items from a menu). We compare HPUI with mid-air direct interactions, a common form of interacting with head-mounted displays. The results show that HPUI performs similarly to mid-air but with better eyes-free affordances and user preferences.

Large curved displays are becoming increasingly popular due to their ability to provide users with a wider field of view and a more immersive experience compared to flat displays. Current interaction techniques for large curved displays often assume a user is positioned at the display's centre, crucially failing to accommodate general use conditions where the user may move during use. In this work, we investigated how user position impacts pointing interaction on large curved displays and evaluated cursor enhancement techniques to provide faster and more accurate performance across positions. To this effect, we conducted two user studies. First, we evaluated the effects of user position on pointing performance on a large semi-circular display (3m-tall, 3270R curvature) through a 2D Fitts' Law selection task. Our results indicate that as users move away from the display, their pointing speed significantly increases (at least by 9%), but accuracy decreases (by at least 6%). Additionally, we observed participants were slower when pointing from laterally offset positions. Secondly, we explored which pointing techniques providing motor- and visual-space enhancements best afford effective pointing performance across user positions. Across a total of six techniques tested, we found that a combination of acceleration and distance-based adjustments with cursor enlargement significantly improves target selection speed and accuracy across different user positions. Results further show techniques with visual-space enhancements (e.g., cursor enlargement) are significantly faster and more accurate than their non-visually-enhanced counterparts. Based on our results we provide design recommendations for implementing cursor enhancement techniques for large curved displays.

Head Mounted Displays (HMDs) are gaining more public attention. With the advancement of tracking technologies, they are incorporating unencumbered interaction techniques to address the need for user-friendly and efficient interaction techniques for day-to-day activities. While there is a good understanding of the different interaction techniques individually, very little research has been done to compare them directly. This would be vital to understanding their strengths and weaknesses in different contexts and building better synergies among them. This paper uses a target selection task to compare the performance and user preferences for four interaction techniques: gaze-pinch, ray pointer, hand-proximate user interface, and direct mid-air interactions. Results indicate that the gaze-pinch interaction technique required significantly more time to complete the task than the others, whose time to complete was similar. However, in terms of preferences and errors, the interaction techniques mostly performed similar.

Natural gestures are crucial for mid-air interaction, but predicting and managing muscle fatigue is challenging. Existing torque-based models are limited in their ability to model above-shoulder interactions and to account for fatigue recovery. We introduce a new hybrid model, NICER , which combines a torque-based approach with a new term derived from the empirical measurement of muscle contraction and a recovery factor to account for decreasing fatigue during rest. We evaluated NICER in a mid-air selection task using two interaction methods with different degrees of perceived fatigue. Results show that NICER can accurately model above-shoulder interactions as well as reflect fatigue recovery during rest periods. Moreover, both interaction methods show a stronger correlation with subjective fatigue measurement ( ρ = 0.978/0.976) than a previous model, Cumulative Fatigue ( ρ = 0.966/0.923), confirming that NICER is a powerful analytical tool to predict fatigue across a variety of gesture-based interactive applications.

Our study investigates the design practices of Hand-Proximate User Interfaces (HPUI) which are displayed on and around a user’s hand in a head-mounted display (HMD). Specifically, we examine one-handed inputs where the main mode of interaction is thumb-to-finger contact. Our focus is on the user interface (UI) design of these displays, and we aim to develop design guidelines and heuristics for this novel design space. To achieve this, we conducted a participatory design study involving 15 participants who provided feedback on 120 different design examples, as well as their thoughts surrounding the HPUI design. Participants favored designs that were ergonomically comfortable and flexible, and those that provided clear visibility regardless of hand positioning. Based on this feedback, we developed 7 design guidelines for Hand Proximate User Interfaces. In applying these guidelines we find that common application interfaces can easily be accommodated using HPUI for use on head-mounted displays.

Three state virtual keyboards which differentiate contact events between released, touched, and pressed states have the potential to improve overall typing experience and reduce the gap between virtual keyboards and physical keyboards. Incorporating force sensitivity, three-state virtual keyboards can utilize a force threshold to better classify a contact event. However, our limited knowledge of how force plays a role during typing on virtual keyboards limits further progress. Through a series of studies we observe that using a uniform threshold is not an optimal approach. Furthermore, the force being applied while typing varies significantly across the keys and among participants. As such, we propose three different approaches to further improve the uniform threshold. We show that a carefully selected non-uniform threshold function could be sufficient in delineating typing events on a three-state keyboard. Finally, we conclude our work with lessons learned, suggestion for future improvements, and comparisons with current methods available.

Data physicalisation is a technique that encodes data through the geometric and material properties of an artefact, allowing users to engage with data in a more immersive and multi-sensory way. However, current methods of data physicalisation are limited in terms of their reconfigurability and the types of materials that can be used. Acoustophoresis—a method of suspending and manipulating materials using sound waves—offers a promising solution to these challenges. In this paper, we present DataLev, a design space and platform for creating reconfigurable, multimodal data physicalisations with enriched materiality using acoustophoresis. We demonstrate the capabilities of DataLev through eight examples and evaluate its performance in terms of reconfigurability and materiality. Our work offers a new approach to data physicalisation, enabling designers to create more dynamic, engaging, and expressive artefacts.

Data Videos (DV) have been used in a diverse set of fields. However, the possibility of utilizing them with social robots for further improving the viewer’s engagement is yet to be examined. While social robots have been used in various presentation-related applications, there is also a lack of design instructions on how to better utilize them. Hence with this early work, we explore the possibility of using social robots as potential DV presenters through; a quantitative analysis of the factors of visible presenters in DVs, and a testing phase of these factors via four group design sessions involving experienced designers. From the DV analysis, we identified 12 unique techniques across four main factors. The observations from the group design sessions show that these findings overlap with the design practices of experienced designers when designing robotic presentations.

Research suggest negative feedback could guide users’ behaviours effectively in Human-AI interactions. However, providing negative feedback, relative to positive counterparts, can be more challenging in any type of communication. This paper delves into the potential of a social robot in delivering negative feedback for improving the in-class learning experience for children. With child participants (12 and younger), we conducted three co-design studies to investigate their preferred facial expressions of a social robot, Haru, which can identify them being distracted (i.e., undesirable behaviour), and redirect their attention back to their task with the facial expressions. Altogether, results indicated that children do not want to see conventional punishing expressions (e.g., angry faces) as a reaction to their undesirable behaviour. Instead, they preferred pleasant ones (e.g., funny, cute). Further, the importance of using realistic stimuli for studies and the co-design approach, as well as the challenges of interpreting children’s drawing responses, are discussed.

Tabletop social robots are becoming increasingly common, not only as social companions but as presenters and orators of information. We present an exploration of utilizing robots as a multimodal presentation tool to communicate data-driven facts. Our exploration is inspired by the wealth of research on data videos (DVs) as these have become mainstream sources for swiftly conveying data-driven information to a mass audience. We first analyze 48 DVs that contain visible narrators (presenters who are visible in the video frames) as our source for understanding the techniques used to convey factual information via presenters. Twelve dimensions across four factors (presenter-grounded; narrative-grounded; viewer-engagement-related; and data-visualization-related) were identified. These factors were carefully arranged in designing presenters to engage the audience with the video content. We adapt these findings to the design of an expressive social tabletop robot that can communicate data-driven knowledge to its audience. Supported by four design sessions with expert content creators and designers, we provide nine design implications for designing multimodal presentations with an expressive tabletop social robot. We conclude with the possible application potentials of this unique data presentation modality.

Conditionally automated vehicles provide unique opportunities for drivers to engage in non-driving-related tasks (NDRTs); however, drivers must remain prepared to respond to take-over requests. This paper explores design challenges and potential solutions for supporting reading as an NDRT in SAE Level 3 vehicles. Specifically, we assess two prominent text-presentation techniques: vertical scrolling text presentation (VSTP) and rapid serial visual presentation (RSVP), exploring both in conjunction with their integration with auditory speech displays (ASD). A driving simulation study involving N = 32 participants revealed that RSVP surpassed VSTP in regaining situational awareness, as indicated by lower average braking actuation, and was also preferred by participants. The integration of ASDs with both techniques reduced perceived cognitive workload and improved the user experience, albeit with compromised lateral control. Our findings can help advance the design of human-centered interfaces for reading in conditionally automated vehicles.

Virtual Reality (VR) is increasingly being adopted in fitness, gaming, and workplace productivity applications for its natural interaction with body movement. A widely accepted method for quantifying the physical fatigue caused by VR interactions is through metrics such as Consumed Endurance (CE). Proposed in 2014, CE calculates the shoulder torque to infer endurance time (ET)—i.e. the maximum amount of time a pose can be maintained—during mid-air interactions. This model remains widely cited but has not been closely examined beyond its initial evaluation, leaving untested assumptions about exertion from low-intensity interactions and its basis on torque. In this paper, we present two VR studies where we (1) collect a baseline dataset that replicates the foundation of CE and (2) extend the initial evaluation in a pointing task from a two-dimensional (2D) screen to a three-dimensional (3D) immersive environment. Our baseline dataset collected from a high-precision tracking system found that the CE model overestimates ET for low-exertion interactions. Further, our studies reveal that a biomechanical model based on only torque cannot account for additional exertion measured when the shoulder angle exceeds 90° elevation. Based on these findings, we propose a revised formulation of CE to highlight the need for a hybrid approach in future fatigue modelling.

Virtual Reality (VR) headsets equipped with multiple cameras enable hands-only teleportation techniques without requiring any physical controller. Hands-only teleportation is an effective alternative to controllers for navigation tasks in virtual reality - allowing users to move from one point to another instantaneously. However, the current implementation of hands-only techniques does not consider users' physical attributes (e.g., arm's reach). Thus, a hands-only teleportation technique can lead to different user experiences based on physical attributes. We propose PAWS, a personalized arm and wrist-based teleportation technique that incorporates users' physical attributes for improved teleportation experiences. We first evaluate different degrees of teleportation personalization with no-, partial, and full personalization. We find that full personalization offers faster locomotion - but at the cost of degraded performances with distant targets due to increased sensitivity. We hence further explore different combinations of mapping functions (e.g., sigmoid, quadratic) to personalize motor movements and find that asymmetric functions result in improved performance. Overall, our results show that PAWS helps users to navigate quickly in virtual environments.

The current pandemic showed us the importance of swiftly disseminating data-based information to the masses of people. This study explores an affect-centered narrative to convey data-driven messages regarding contact tracing apps (CTAs) using video as a medium (i.e., data video). A between-subjects online study compared the effect of three storytelling approaches on viewers' perception. A video developed by Google was selected as the baseline video (Control Condition; 2min 23s) due to its high quality and relevance to CTAs. The central messages of this baseline video were; a) how CTAs work, and b) how safe and effective CTAs are. Infographics supporting these messages were then added to the baseline video (the second condition; 3min 19s); this was a simple data video (DV), and it did not intend to induce specific emotional experiences in participants (i.e., cognition-centered video). Finally, an affect-focused DV (AFDV) was also created by emphasizing the emotion-based narrative aspect of the message (the third condition; 4min 6s). In this video, three cute human-like cartoon characters were introduced. Viewers in this condition needed to process both cognitive and affective information. Note all three videos (i.e., control video, DV, and AFDV) conveyed identical messages. Participants watched one of these three videos only once, and we explored the video effect on their perception. Our results repeatedly indicated the potential benefits of including affect in data storytelling.

Data Videos (DVs), or animated infographics that tell stories with data, are becoming increasingly popular. Despite their potential to induce attitude change, little is explored about how to produce effective DVs. This paper describes two studies that explored factors linked to the potential of health DVs to improve viewers’ behavioural change intentions. We investigated: 1) how viewers’ affect is linked to their behavioural change intentions; 2) how these affect are linked to the viewers’ personality traits; 3) which attributes of DVs are linked to their persuasive potential. Results from both studies indicated that viewers’ negative affect lowered their behavioural change intentions. Individuals with higher neuroticism exhibited higher negative affect and were harder to convince. Finally, Study 2 proved that providing any solutions to the health problem, presented in the DV, made the viewers perceive the videos as more actionable while lowering their negative affect, and importantly, induced higher behavioural change intentions.

Advanced technologies are increasingly enabling the creation of interactive devices with non-rectangular form-factors but it is currently unclear what alternative form-factors are desirable for end-users. We contribute an understanding of the interplay between the rationale for the form factors of such devices and their interactive content through think-aloud design sessions in which participants could mold devices as they wished using clay. We analysed their qualitative reflections on how the shapes affected interaction. Using thematic analysis, we identified shape features desirable on handheld freeform devices and discuss the particularity of three themes central to the choice of form factors: freeform dexterity, shape features discoverability and shape adaptability (to the task and context). In a second study following the same experimental set-up, we focused on the trade off between dexterity and discoverability and the relation to the concept of affordance. Our work reveals the shape features that impact the most the choice of grasps on freeform devices from which we derive design guidelines for the design of such devices.

We present EdgeSelect, a linear target selection interaction technique that utilizes a small portion of the smartwatch display, explicitly designed to mitigate the ‘fat finger’ and screen occlusion problems, two of the most common and well-known challenges when interacting with small displays. To design our technique, we first conducted a user study to answer which segments of the smartwatch display have the least screen occlusion while users are interacting with it. We use results from the first experiment to introduce EdgeSelect, a three-layer non-linear interaction technique, which can be used to interact with multiple co-adjacent graphs on the smartwatch by using a region that is the least prone to finger occlusion. In a second experiment, we explore the density limits of the targets possible with EdgeSelect. Finally, we demonstrate the generalizability of EdgeSelect to interact with various types of content.

Teleportation, a widely used locomotion technique in Virtual Reality (VR), is used to move users through a virtual environment. Until recently, handheld controllers have been used for teleportation, where users use controllers to point to a location and perform an action (e.g., button press) to be instantly moved to the targeted location. Recent advancements in VR hand tracking enable users to move through and interact with the virtual world without controllers. This opens the opportunity for compelling alternatives to explore hand tracking-based teleportation techniques for more natural, intuitive and immersive interactions. Prior work mostly explores using arm movement for teleportation as an alternative to using the controller. In this paper, we design and evaluate WriArm, a VR locomotion technique that leverages both wrist and arm movement for VR teleportation. We first conduct a design study to find suitable hand gesture sets that can be mapped to teleportation activities such as activation, pointing, confirmation and cancellation for WriArm and arm-based techniques. Based on the results, we conduct a study comparing users’ performance while navigating tasks with the two techniques and three gesture sets. Results show that WriArm improves navigation efficiency by allowing users to navigate the environment quickly. We conclude with design guidelines for arm and wrist-based teleportation in VR.

Supporting eyes-free interaction, mobility and encumbrance, while providing a broad set of commands on a smartwatch display is a difficult, yet important, task. Bezel-to-bezel (B2B) gestures are valuable for rapid command invocation during eyes-free operation, however we lack knowledge regarding B2B interactions on circular devices during common usage scenarios. We aim to improve our understanding of the dynamics of B2B interactions in these scenarios by conducting two studies and a third analysis: First, we explore the performance of B2B in a seated position; second, we explore the effect of mobility and encumbrance on the B2B interaction; finally, we improve on the B2B accuracies by calculating features and utilizing machine learning. With the limited interaction capabilities on smartwatches and the importance of the scenario of use, we conclude with applications and design guidelines for improved utilization of B2B that enables effective smartwatch control while in common, mobile and eyes-free scenarios.

We present BezelGlide, a novel suite of bezel interaction techniques, designed to minimize screen occlusion and ‘fat finger’ effects, when interacting with common graphs on smartwatches. To explore the design of BezelGlide, we conducted two user studies. First, we quantified the amount of screen occlusion experienced when interacting with the smartwatch bezel. Next, we designed two techniques that involve gliding the finger along the smartwatch bezel for graph interaction. Full BezelGlide (FBG) and Partial BezelGlide (PBG), use the full or a portion of the bezel, respectively, to reduce screen occlusion while scanning a line chart for data. In the common value detection task, we find that PBG outperforms FBG and Shift, a touchscreen occlusion-free technique, both quantitatively and subjectively, also while mobile. We finally illustrate the generzability potential of PBG to interact with common graph types making it a valuable interaction technique for smartwatch users.

We designed a mid-air input space for restful interactions on the couch. We observed people gesturing in various postures on a couch and found that posture affects the choice of arm motions when no constraints are imposed by a system. Study participants that sat with the arm rested were more likely to use the forearm and wrist, as opposed to the whole arm. We investigate how a spherical input space, where forearm angles are mapped to screen coordinates, can facilitate restful mid-air input in multiple postures. We present two controlled studies. In the first, we examine how a spherical space compares with a planar space in an elbow-anchored setup, with a shoulder-level input space as baseline. In the second, we examine the performance of a spherical input space in four common couch postures that set unique constraints to the arm. We observe that a spherical model that captures forearm movement facilitates comfortable input across different seated postures.

The tabletop robot Haru, used for affective telepresence research, enables a teleoperator to communicate affects from a distance. The robot’s expressiveness offers myriad ways of communicating affects through the execution of emotive routines. The teleoperator reacts to input modalities such as the user’s facial expression, gestures and speech-based intent as perceived by the robot’s perception system. However, due to the sheer number of routines to select from, the task of choosing the appropriate or the most preferred routine is becoming cumbersome. In this paper, we propose a human-in-the-loop reinforcement learning mechanism in which an agent learns the teleoperator’s selection preference as a function of the input modalities and aids the routine selection process by narrowing it to n-best optimal choices. Our experimental results show that with only a few number of interactions from the teleoperator, the system can learn to recommend optimal routine behaviors for all perceived modalities, which greatly reduces the workload of the teleoperator.

Identifying cybersickness in virtual reality (VR) applications such as games in a fast, precise, non-intrusive, and non-disruptive way remains challenging. Several factors can cause cybersickness, and their identification will help find its origins and prevent or minimize it. One such factor is virtual movement. Movement, whether physical or virtual, can be represented in different forms. One way to represent and store it is with a temporally annotated point sequence. Because a sequence is memory-consuming, it is often preferable to save it in a compressed form. Compression allows redundant data to be eliminated while still preserving changes in speed and direction. Since changes in direction and velocity in VR can be associated with cybersickness, changes in compression rate can likely indicate changes in cybersickness levels. In this research, we explore whether quantifying changes in virtual movement can be used to estimate variation in cybersickness levels of VR users. We investigate the correlation between changes in the compression rate of movement data in two VR games with changes in players’ cybersickness levels captured during gameplay. Our results show (1) a clear correlation between changes in compression rate and cybersickness, and (2) that a machine learning approach can be used to identify these changes. Finally, results from a second experiment show that our approach is feasible for cybersickness inference in games and other VR applications that involve movement.

Fingertip computing has seen increased interest through miniaturized smart-rings for augmenting digital peripherals. One key advantages of such always-available input devices is the non-necessity to hold a device for interaction, as it remains affixed to a finger for access when needed. Such a wearable device makes it possible to interaction with content even when the hand is encumbered, by grasping or holding objects. Our investigation aims at understanding the properties of this fundamental smart-ring advantage. We designed a smart-ring prototype, ARO (in-Air, on-Ring, on-Object interaction), which facilitates input while grasping objects. To better identify interaction possibilities, we present the results of an elicitation study through which we grouped various forms of micro-gestures possible with ARO while holding objects under different grasp requirements. We then explored the ability for users to perform different navigation tasks (i.e. zooming and panning) using the smart-ring with encumbered hands. In our studies, users were most efficient when using either In-air or On-ring interactions, in comparisons to gestures detected On-object. Furthermore, In-air was the most preferred by our participants. Based on our findings, we conclude with recommendations for the design of future smart-rings and fingertip devices at large, to allow efficient interaction while hands are encumbered.

Multiple embedded sensors enable smartwatch apps to amass large amounts of interrelated time-series data simultaneously, such as heart rate, oxygen levels or steps walked. Visualizing multiple interlinked datasets is possible on smartphones but remains challenging on small smartwatch displays. We propose a new technique, the Space-Filling Line Graph (SF-LG), that preserves the key visual properties of time-series graphs while making available space on the display to augment such graphs with additional information. Results from our first study (N=30) suggest that, while SF-LG makes available additional space on the small display, it also enables effective (i.e. quick and accurate) comprehension of key line graph tasks. We next implement a greedy algorithm to embed auxiliary information in the most suitable regions on the display. In a second study (N=27), we find that participants are efficient at locating and linking interrelated content using SF-LG in comparison to two baselines approaches. We conclude with guidelines for smartwatch space maximization for visual displays.

We describe a method of teaching a robot its empathic behavioural response from its interaction with people. We used the input modalities such as relative spatial information, facial expressions, body gestures and speech information as perception input that triggers the robot’s empathic response. First, we bootstrap the training through a pre-learning mechanism in which training is conducted by users who know the robotic system. This phase provides simulation-based training using a simple graphical user interface to simulate the input, rewards and correction feedback. In the second phase, we developed an online learning scheme for naive users to personalize their robot further, building on top of the bootstrapped model. Here, we developed a natural user interface that enables natural human-robot interaction via the suite of sensors that allows the users to provide evaluative feedback during the interaction with the robot. We evaluated the system and our results show that bootstrapping is an efficient tool to hasten the robot’s learning while online learning provided some form of personalization in the real environment with naive users.

We explore the design of Hand Proximate User Interfaces (HPUIs) for head-mounted displays (HMDs) to facilitate near-body interactions with the display directly projected on, or around the user's hand. We focus on single-handed input, while taking into consideration the hand anatomy which distorts naturally when the user interacts with the display. Through two user studies, we explore the potential for discrete as well as continuous input. For discrete input, HPUIs favor targets that are directly on the fingers (as opposed to off-finger) as they offer tactile feedback. We demonstrate that continuous interaction is also possible, and is as effective on the fingers as in the off-finger space between the index finger and thumb. We also find that with continuous input, content is more easily controlled when the interaction occurs in the vertical or horizontal axes, and less with diagonal movements. We conclude with applications and recommendations for the design of future HPUIs.

Visual interfaces are important in human motion to capture it, to visualize it, and to facilitate motion-based interactive systems. This workshop aims at providing a platform for researchers, designers and users to discuss the challenges related to the design of visual interfaces for motion-based interaction, in terms of visualization (e.g. graphical user interface, multimodal feedback, evaluation) and processing (e.g., data collection, treatment, interpretation, recognition) of human movement (e.g., motor skills, amplitude of movements, limitations). We will share experiences, lessons learned and elaborate tools for developing all the possible applications going forward.

The use of Virtual Reality (VR) in applications such as data analysis, artistic creation, and clinical settings requires high precision input. However, the current design of handheld controllers, where wrist rotation is the primary input approach, does not exploit the human fingers' capability for dexterous movements for high precision pointing and selection. To address this issue, we investigated the characteristics and potential of using a pen as a VR input device. We conducted two studies. The first examined which pen grip allowed the largest range of motion---we found a tripod grip at the rear end of the shaft met this criterion. The second study investigated target selection via 'poking' and ray-casting, where we found the pen grip outperformed the traditional wrist-based input in both cases. Finally, we demonstrate potential applications enabled by VR pen input and grip postures.

Foldable handheld displays have the potential to offer a rich interaction space, particularly as they fold into a convex form factor, for collocated multi-user interactions. In this paper, we explore Tent mode, a convex configuration of a foldable device partitioned into a primary and a secondary display, as well as a tertiary, Edge display that sits at the intersection of the two. We specifically explore the design space for a wide range of scenarios, such as co-browsing a gallery or co-planning a trip. Through a first collection of interviews, end-users identified a suite of apps that could leverage Tent mode for multi-user interactions. Based on these results we propose an interaction design space that builds on unique Tent mode properties, such as folding, flattening or tilting the device, and the interplay between the three sub-displays. We examine how end-users exploit this rich interaction space when presented with a set of collaborative tasks through a user study, and elicit potential interaction techniques. We implemented these interaction techniques and report on the preliminary user feedback we collected. Finally, we discuss the design implications for collocated interaction in Tent mode configurations.

Extended Reality (XR) systems (which encapsulate AR, VR and MR) is an emerging field which enables the development of novel visualization and interaction techniques. To develop and to assess such techniques, researchers and designers have to face choices in terms of which development tools to adopt, and with very little information about how such tools support some of the very basic tasks for information visualization, such as selecting data items, linking and navigating. As a solution, we propose Flex-ER, a flexible web-based environment that enables users to prototype, debug and share experimental conditions and results. Flex-ER enables users to quickly switch between hardware platforms and input modalities by using a JSON specification that supports both defining interaction techniques and tasks at a low cost. We demonstrate the flexibility of the environment through three task design examples: brushing, linking and navigating. A qualitative user study suggest that Flex-ER can be helpful to prototype and explore different interaction techniques for immersive analytics.

Hierarchy structures such as file systems are widespread interfaces for item retrieval and selection tasks. Some hierarchies can be modified by end-users, such as application launchers on smartphones or pictures in a file folder. These modifiable hierarchies cannot benefit from an optimization made beforehand as their content, unknown during the design process, is constantly evolving. We hence propose an analytic model which designers can integrate in their system to recommend a range of local structure modifications (e.g., creating new folders) to end-users. Proposing a range of modifications gives flexibility to end-users regarding their own meaningful grouping and labeling choices to follow a recommendation. A first experiment confirms that the recommendations built on our model can lead to modified hierarchies resulting in faster theoretical selection times. A second experiment confirms that the theoretical selection times fit empirical selection times in different hierarchy visual layouts: linear, radial, and grid.

Intensive exploration and navigation of hierarchical lists on smartphones can be tedious and time-consuming as it often requires users to frequently switch between multiple views. To overcome this limitation, we present PinchList, a novel interaction design that leverages pinch gestures to support seamless exploration of multi-level list items in hierarchical views. With PinchList, sub-lists are accessed with a pinch-out gesture whereas a pinch-in gesture navigates back to the previous level. Additionally, pinch and flick gestures are used to navigate lists consisting of more than two levels. We conduct a user study to refine the design parameters of PinchList such as a suitable item size, and quantitatively evaluate the target acquisition performance using pinch-in/out gestures in both scrolling and non-scrolling conditions. In a second study, we compare the performance of PinchList in a hierarchal navigation task with two commonly used touch interfaces for list browsing: pagination and expand-and-collapse interfaces. The results reveal that PinchList is significantly faster than other two interfaces in accessing items located in hierarchical list views. Finally, we demonstrate that PinchList enables a host of novel applications in list-based interaction?

With upcoming breakthroughs in free-form display technologies, new user interface design challenges have emerged. Here, we investigate a question, which has been widely explored on traditional GUIs but unexplored on non-rectangular interfaces: what are the user strategies in terms of visual search when information is not presented in a traditional rectangular layout? To achieve this, we present two complementary studies investigating eye movements in different visual search tasks. Our results unveil which areas are seen first according to different visual structures. By doing so we address the question of where to place relevant content for the UI designers of non-rectangular displays.

Autostereograms are 2D images that can reveal 3D content when viewed with a specific eye convergence, without using extra-apparatus. We contribute to autostereogram studies from an HCI perspective. We explore touch inputs and output design options when interacting with autostereograms on smartphones. We found that an interactive help (i.e. to control the autostereogram stereo-separation), a color-based feedback (i.e. highlight of the screen), and a direct touch input can provide support for faster and more accurate interaction than a static help (i.e. static dots indicating the stereo-separation), an animated feedback (i.e., a 'pressed' effect), and an indirect input. In addition, results reveal that participants learn to perceive smaller and smaller autostereogram content faster with practice. This learning effect transfers across display devices (smartphone to desktop screen).

Pictographic representations and animation techniques are commonly incorporated into narrative visualizations such as data videos. General belief is that these techniques may enhance the viewer experience, thus appealing to a broad audience and enticing the viewer to consume the entire video. However, no study has formally assessed the effect of these techniques on data insight communication and viewer engagement. In this paper, we first propose a scale-based questionnaire covering five factors of viewer engagement we identified from multiple application domains such as game design and marketing. We then validate this questionnaire through a crowdsourcing study on Amazon's Mechanical Turk to assess the effect of animation and pictographs in data videos. Our results reveal that each technique has an effect on viewer engagement, impacting different factors. In addition, insights from these studies lead to design considerations for authoring engaging data videos.

The use of crowdsourcing platforms for data collection in HCI research is attractive in their ability to provide rapid access to large and diverse participant samples. As a result, several researchers have conducted studies investigating the similarities and differences between data collected through crowdsourcing and more traditional, laboratory-style data collection. We add to this body of research by examining the feasibility of conducting social acceptability studies via crowdsourcing. Social acceptability can be a key determinant for the early adoption of emerging technologies, and as such, we focus our investigation on social acceptability for Head-Worn Display (HWD) input modalities. Our results indicate that data collected via a crowdsourced experiment and a laboratory-style setting did not differ at a statistically significant level. These results provide initial support for crowdsourcing platforms as viable options for conducting social acceptability research.

Selecting an item of interest on smartwatches can be tedious and time-consuming as it involves a series of swipe and tap actions. We present PageFlip, a novel method that combines into a single action multiple touch operations such as command invocation and value selection for efficient interaction on smartwatches. PageFlip operates with a page flip gesture that starts by dragging the UI from a corner of the device. We first design PageFlip by examining its key design factors such as corners, drag directions and drag distances. We next compare PageFlip to a functionally equivalent radial menu and a standard swipe and tap method. Results reveal that PageFlip improves efficiency for both discrete and continuous selection tasks. Finally, we demonstrate novel smartwatch interaction opportunities and a set of applications that can benefit from PageFlip.

Smartwatches enable rapid access to information anytime and anywhere. However, current smartwatch content navigation techniques, for panning and zooming, were directly adopted from those used on smartphones. These techniques are cumbersome when performed on small smartwatch screens and have not been evaluated for their support in mobility and encumbrance contexts (when the user's hands are busy). We studied the effect of mobility and encumbrance on common content navigation techniques and found a significant decrease in performance as the pace of mobility increases or when the user was encumbered with busy hands. Based on these initial findings, we proposed a design space which would improve efficiency when navigation techniques, such as panning and zooming, are employed in mobility contexts. Our results reveal that our design space can effectively be used to create novel interaction techniques that improve smartwatch content navigation in mobility and encumbrance contexts.

We investigated interactive agents using subliminal priming - the act of exposing a person to stimuli that they may not consciously notice, but are still processed subliminally in their mind - in an attempt to shape a person's mood and behavior. We present an overview of the psychology of subliminal priming from the perspective of how it applies to human-agent interaction, including a discussion of the potential ethical and practical implications. We further present the results from two exploratory studies (one in-lab, one crowdsourced) that present potential subliminal-priming interfaces. Our results suggest that subliminal priming may impact how participants perceive an agent and how much they enjoy a task, but we failed to find any effect of priming on participant mood or agent persuasiveness. This work aims to raise awareness of the dangers of subliminal methods of priming and contributes to the discussion on the ethics of social agents.

We present PathWord (PATH passWORD), a multimodal digit entry method for ad-hoc authentication based on known digits shape and user relative eye movements. PathWord is a touch-free, gaze-based input modality, which attempts to decrease shoulder surfing attacks when unlocking a system using PINs. The system uses a modified web camera to detect the user's eye. This enables suppressing direct touch, making it difficult for passer-bys to be aware of the input digits, thus reducing shoulder surfing and smudge attacks. In addition to showing high accuracy rates (Study 1: 87.1% successful entries) and strong confidentiality through detailed evaluations with 42 participants (Study 2), we demonstrate how PathWord considerably diminishes the potential of stolen passwords (on average 2.38% stolen passwords with PathWord vs. over 90% with traditional PIN screen). We show use-cases of PathWord and discuss its advantages over traditional input modalities. We envision PathWord as a method to foster confidence while unlocking a system through gaze gestures.

Current haptic devices are often bulky and rigid, making them unsuitable for ubiquitous interaction and scenarios where the user must also interact with the real world. To address this gap, we propose HydroRing, an unobtrusive, finger-worn device that can provide the tactile sensations of pressure, vibration, and temperature on the fingertip, enabling mixed-reality haptic interactions. Different from previous explorations, HydroRing in active mode delivers sensations using liquid travelling through a thin, flexible latex tube worn across the fingerpad, and has minimal impact on a user's dexterity and their perception of stimuli in passive mode. Two studies evaluated participants' ability to perceive and recognize sensations generated by the device, as well as their ability to perceive physical stimuli while wearing the device. We conclude by exploring several applications leveraging this mixed-reality haptics approach.

The popularity of head-worn displays (HWD) technologies such as Virtual Reality (VR) and Augmented Reality (AR) headsets is growing rapidly. To predict their commercial success, it is essential to understand the acceptability of these new technologies, along with new methods to interact with them. In this vein, the evaluation of social acceptability of interactions with these technologies has received significant attention, particularly from the performer's (i.e., user's) viewpoint. However, little work has considered social acceptability concerns from observers' (i.e., spectators') perspective. Although HWDs are designed to be personal devices, interacting with their interfaces are often quite noticeable, making them an ideal platform to contrast performer and observer perspectives on social acceptability. Through two studies, this paper contrasts performers' and observers' perspectives of social acceptability interactions with HWDs under different social contexts. Results indicate similarities as well as differences, in acceptability, and advocate for the importance of including both perspectives when exploring social acceptability of emerging technologies. We provide guidelines for understanding social acceptability specifically from the observers' perspective, thus complementing our current practices used for understanding the acceptability of interacting with these devices.

In recent years, around device input has emerged as a complement to standard touch input, albeit in limited tasks and contexts, such as for item selection or map navigation. We push the boundaries for around device interactions to facilitate an entire smartphone application: browsing through large information lists to make a decision. To this end, we present AirPanes, a novel technique that allows two-handed in-air interactions, conjointly with touch input to perform analytic tasks, such as making a purchase decision. AirPanes resolves the inefficiencies of having to switch between multiple views or panes in common smartphone applications. We explore the design factors that make AirPanes efficient. In a controlled study, we find that AirPanes is on average 50% more efficient that standard touch input for an analytic task. We offer recommendations for implementing AirPanes in a broad range of applications.

The rapid evolution of multi-display environments (MDEs) has created a vacuum in need of novel input devices to optimize interaction in MDEs. In this paper, we propose TDome, a novel touch-enabled 6DOF input and output device to facilitate interactions in MDEs. TDome offers a private display as output, and multiple degrees of freedom as input by combining touch gestures on the display with physical rotation, roll and translation manipulations of the device. TDome allows versatile interactions that address major MDE tasks, which we illustrate through various proof-of-concept implementations: detect surrounding displays, select one display, transfer data across displays, reach distant displays and perform private interactions. We explore TDome's usability and suitability for MDEs through three user studies. First we explore combined physical+touch gestures from which we discard uncomfortable combinations. We experimentally validate their feasibility and come up with a set of 71 combined gestures that are comfortable and ensure a high success rate, i.e. that can be easily performed and efficiently detected. Finally, we collect user feedback to identify natural mappings between gestures and MDE interactions.

Graphical user interfaces are composed of varying elements (text, images, etc.) whose visual arrangement has been relatively well established in the context of rectangular interfaces. The advent of non-rectangular displays questions this knowledge. In this paper we study how traditional content layouts can be adapted to fit different non-rectangular displays. We performed a first qualitative study where graphic designers fitted text and images into different non-rectangular displays. From the analysis of their output we generalize and adapt ten composition principles that have been proposed in the literature for rectangular displays. We evaluate the revised principles through a paired comparison questionnaire where 57 participants compared pairs of layouts. Using the Bradley-Terry-Luce model to analyze our data we show that some results contradict current conventions on visual design for rectangular displays. We then extracted the most interesting cases and conducted a follow up study with additional shapes to investigate how the principles generalize. From these results we propose a set of guidelines for designing visual content for non-rectangular displays.

We propose the concept of a guiding system specifically designed for semaphoric gaze gestures, i.e. gestures defining a vocabulary to trigger commands via the gaze modality. Our design exploration considers fundamental gaze gesture phases: Exploration, Guidance, and Return. A first experiment reveals that Guidance with dynamic elements moving along 2D paths is efficient and resistant to visual complexity. A second experiment reveals that a Rapid Serial Visual Presentation of command names during Exploration allows for more than 30% faster command retrievals than a standard visual search. To resume the task where the guide was triggered, labels moving from the outward extremity of 2D paths toward the guide center leads to efficient and accurate origin retrieval during the Return phase. We evaluate our resulting Gaze Gesture Guiding system, G3, for interacting with distant objects in an office environment using a head-mounted display. Users report positively on their experience with both semaphoric gaze gestures and G3.

We present a set of mini-map alternatives for indicating the relative locations of robot team members in a tele-operation interface, and evaluation results showing that these can perform as well as mini-maps while being less intrusive. Teleoperation operators often work with a team of robots to improve task effectiveness. Maintaining awareness of where robot team members are, relative to oneself, is important for team effectiveness, such as for deciding which robot may help with a task, may be best suited to investigate a point of interest, or to determine where one should move next. We explore the use of established interface techniques from mobile computing for supporting teleoperators in maintaining peripheral awareness of robot team members' relative locations. We evaluate the nontrivial adoption of these techniques to teleoperation, comparing to an overview mini-map base case. Our results indicate that in-feed embedded indicators perform comparatively well to mini-maps, while being less obtrusive, indicating that they are a viable alternative for teleoperation interfaces.

We present SoundCraft, a smartwatch prototype embedded with a microphone array, that localizes angularly, in azimuth and elevation, acoustic signatures: non-vocal acoustics that are produced using our hands. Acoustic signatures are common in our daily lives, such as when snapping or rubbing our fingers, tapping on objects or even when using an auxiliary object to generate the sound. We demonstrate that we can capture and leverage the spatial location of such naturally occurring acoustics using our prototype. We describe our algorithm, which we adopt from the MUltiple SIgnal Classification (MUSIC) technique [31], that enables robust localization and classification of the acoustics when the microphones are required to be placed at close proximity. SoundCraft enables a rich set of spatial interaction techniques, including quick access to smartwatch content, rapid command invocation, in-situ sketching, and also multi-user around device interaction. Via a series of user studies, we validate SoundCraft's localization and classification capabilities in non-noisy and noisy environments.

Data videos, or short data-driven motion graphics, are an increasingly popular medium for storytelling. However, creating data videos is difficult as it involves pulling together a unique combination of skills. We introduce DataClips, an authoring tool aimed at lowering the barriers to crafting data videos. DataClips allows non-experts to assemble data-driven "clips" together to form longer sequences. We constructed the library of data clips by analyzing the composition of over 70 data videos produced by reputable sources such as The New York Times and The Guardian. We demonstrate that DataClips can reproduce over 90% of our data videos corpus. We also report on a qualitative study comparing the authoring process and outcome achieved by (1) non-experts using DataClips, and (2) experts using Adobe Illustrator and After Effects to create data-driven clips. Results indicated that non-experts are able to learn and use DataClips with a short training period. In the span of one hour, they were able to produce more videos than experts using a professional editing tool, and their clips were rated similarly by an independent audience.

Mobile devices have begun to raise questions around the potential for overuse when in the presence of family or friends. As such, we conducted a diary and interview study to understand how people use mobile devices in the presence of others at home, and how this shapes their behavior and household dynamics. Results show that family members become frustrated when others do non-urgent activities on their phones in the presence of others. Yet people often guess at what others are doing because of the personal nature of mobile devices. In some cases, people developed strategies to provide a greater sense of activity awareness to combat the problem. Mobile phone usage was sometimes perceived as beneficial by providing a mechanism for needed disengagement from family members. These findings suggest several opportunities for redesigning mobile device software to mitigate emergent frustrations, and open up new opportunities for nurturing social interactions among family members.

Emerging technologies allow for the creation of non-rectangular displays with unlimited constraints in shape. However, the introduction of such displays radically deviates from the prevailing tradition of placing content on rectangular screens and raises fundamental design questions. Among these is the foremost question of how to legibly present text. We address this fundamental concern through a multi-part exploration that includes: (1) a focus-group study from which we collected free-form display scenarios and extracted display shape properties; (2) a framework that identifies different mappings of text onto a non-rectangular shape and formulates hypotheses concerning legibility for different display shape properties; and (3) a series of quantitative text legibility studies to assess our hypotheses. Or results agree with and extend upon other findings in the existing literature on text legibility, but they also uncover unique instances in which different rules need to be applied for non-rectangular displays. These results also provide guidelines for the design of visual interfaces.

We propose and study a new input modality, WristWhirl, that uses the wrist as an always-available joystick to perform one-handed continuous input on smartwatches. We explore the influence of the wrist's bio-mechanical properties for performing gestures to interact with a smartwatch, both while standing still and walking. Through a user study, we examine the impact of performing 8 distinct gestures (4 directional marks, and 4 free-form shapes) on the stability of the watch surface. Participants were able to perform directional marks using the wrist as a joystick at an average rate of half a second and free-form shapes at an average rate of approximately 1.5secs. The free-form shapes could be recognized by a $1 gesture recognizer with an accuracy of 93.8% and by three human inspectors with an accuracy of 85%. From these results, we designed and implemented a proof-of-concept device by augmenting the watchband using an array of proximity sensors, which can be used to draw gestures with high quality. Finally, we demonstrate a number of scenarios that benefit from one-handed continuous input on smartwatches using WristWhirl.

Data videos, motion graphics that incorporate visualizations about facts, are increasingly gaining popularity as a means of telling stories with data. However, very little is systematically recorded about (a) what elements are featured in data videos and (b) the processes used to create them. In this article, we provide initial insights to build this knowledge. We first report on a qualitative analysis of 50 professionally designed data videos, extracting and exposing their most salient constituents. Second, we report on a series of workshops with experienced storytellers from cinematography, graphics design and screenplay writing. We provided them with a set of data facts and visualizations and observed them create storyboards for data videos. From these exploratory studies, we derive broader implications for the design of an authoring tool to enable a wide audience to create data videos. Our findings highlight the importance of providing a flexible tool supporting a non-linear creation process and allowing users to iteratively go back to different phases of the process.

Motivated by a rise in the variety and number of mobile devices that users carry, we investigate scenarios when operating these devices in a spatially interlinked manner can lead to interfaces that generate new advantages. Our exploration is focused on the design of SAMMI, a spatially-aware multi-device interface to assist with analytic map navigation tasks, where, in addition to browsing the workspace, the user has to make a decision based on the content embedded in the map. We focus primarily on the design space for spatially interlinking a smartphone with a smartwatch. As both smart devices are spatially tracked, the user can browse information by moving either device in the workspace. We identify several design factors for SAMMI and through a first study we explore how best to combine these for efficient map navigation. In a second study we compare SAMMI to the common Flick-&-Pinch gestures for an analytic map navigation task. Our results reveal that SAMMI is an efficient spatial navigation interface, and by means of an additional spatially tracked display, can facilitate quick information retrieval and comparisons. We finally demonstrate other potential use cases for SAMMI that extend beyond map navigation to facilitate interaction with spatial workspaces.

Distributed display environments (DDEs) allow use of various specialized devices but challenge designers to provide a clean flow of data across multiple displays. Upcoming consumer-ready head-worn displays (HWDs) can play a central role in unifying the interaction experience in such ecosystems. In this paper, we report on the design and development of Gluey, a user interface that acts as a 'glue' to facilitate seamless input transitions and data movement across displays. Based on requirements we refine for such an interface, Gluey leverages inherent headworn display attributes such as field-of-view tracking and an always-available canvas to redirect input and migrate content across multiple displays, while minimizing device switching costs. We implemented a functional prototype integrating Gluey's numerous interaction possibilities. From our experience in this integration and from user evaluation results, we identify the open challenges in using HWDs to unify the interaction experience in DDEs.

The orientation and repositioning of physical artefacts (such as paper documents) to afford shared viewing of content, or to steer the attention of others to specific details, is known as micro-mobility. But the role of grasp in micro-mobility has rarely been considered, much less sensed by devices. We therefore employ capacitive grip sensing and inertial motion to explore the design space of combined grasp + micro-mobility by considering three classes of technique in the context of active reading. Single user, single device techniques support grip-influenced behaviors such as bookmarking a page with a finger, but combine this with physical embodiment to allow flipping back to a previous location. Multiple user, single device techniques, such as passing a tablet to another user or working side-by-side on a single device, add fresh nuances of expression to co-located collaboration. And single user, multiple device techniques afford facile cross-referencing of content across devices. Founded on observations of grasp and micro-mobility, these techniques open up new possibilities for both individual and collaborative interaction with electronic documents.

GPS, RFID, and other technologies have made it increasingly common to track the positions of people and objects over time as they move through two-dimensional spaces. Visualizing such spatio-temporal movement data is challenging because each person or object involves three variables (two spatial variables as a function of the time variable), and simply plotting the data on a 2D geographic map can result in overplotting and occlusion that hides details. This also makes it difficult to understand correlations between space and time. Software such as GeoTime can display such data with a three-dimensional visualization, where the 3rd dimension is used for time. This allows for the disambiguation of spatially overlapping trajectories, and in theory, should make the data clearer. However, previous experimental comparisons of 2D and 3D visualizations have so far found little advantage in 3D visualizations, possibly due to the increased complexity of navigating and understanding a 3D view. We present a new controlled experimental comparison of 2D and 3D visualizations, involving commonly performed tasks that have not been tested before, and find advantages in 3D visualizations for more complex tasks. In particular, we tease out the effects of various basic interactions and find that the 2D view relies significantly on "scrubbing" the timeline, whereas the 3D view relies mainly on 3D camera navigation. Our work helps to improve understanding of 2D and 3D visualizations of spatio-temporal data, particularly with respect to interactivity.

Users of optical see-through head-mounted displays (OHMD) perceive color as a blend of the display color and the background. Color-blending is a major usability challenge as it leads to loss of color encodings and poor text legibility. Color correction aims at mitigating color blending by producing an alternative color which, when blended with the background, more closely approximates the color originally intended. In this paper we present an end-to-end approach to the color blending problem addressing the distortions introduced by the transparent material of the display efficiently and in real time. We also present a user evaluation of correction efficiency. Finally, we present a graphics library called SmartColor showcasing the use of color correction for different types of display content. SmartColor uses color correction to provide three management strategies: correction, contrast, and show-up-on-contrast. Correction determines the alternate color which best preserves the original color. Contrast determines the color which best supports text legibility while preserving as much of the original hue. Show-up-on-contrast makes a component visible when a related component does not have enough contrast to be legible. We describe SmartColor's architecture and illustrate the color strategies for various types of display content.

As a novel class of mobile devices with rich interaction capabilities we introduce tPads -- transparent display tablets. tPads are the result of a systematic design investigation into the ways and benefits of interacting with transparent mobiles which goes beyond traditional mobile interactions and augmented reality (AR) applications. Through a user-centered design process we explored interaction techniques for transparent-display mobiles and classified them into four categories: overlay, dual display & input, surface capture and model-based interactions. We investigated the technical feasibility of such interactions by designing and building two touch-enabled semi-transparent tablets called tPads and a range of tPad applications. Further, a user study shows that tPad interactions applied to everyday mobile tasks (application switching and image capture) outperform current mobile interactions and were preferred by users. Our hands-on design process and experimental evaluation demonstrate that transparent displays provide valuable interaction opportunities for mobile devices.

Crowd sourced crisis mapping is a relatively new phenomenon and platform that enables the collection and visualization of real-time crisis data submitted by users through social media tools and cellular technologies. Crisis maps are generally used by both state and non-state actors for sense-making and as a reference point for action. The current crisis map visualizations only show the location from which documents such as reports or short messages have been generated. Such a limited representation fails to immediately show important content, such as themes from a document and their changes over time. As a result, sense-making becomes time-consuming and cognitively demanding. We present a new visualization approach, Geo-Temporal Tag Visualization (GTViz), which treats the tags on the crowd sourced reports as spatio-temporal textual datasets. GTViz embeds a map to represent the spatial properties of the dataset and compactly shows the changes of key terms from the data source over time. We demonstrate the value of GTViz through two case studies and a controlled user study.

As wearable computing goes mainstream, we must improve the state of interface design to keep users productive with natural-feeling interactions. We present the Personal Cockpit, a solution for mobile multitasking on head-worn displays. We appropriate empty space around the user to situate virtual windows for use with direct input. Through a design-space exploration, we run a series of user studies to fine-tune our layout of the Personal Cockpit. In our final evaluation, we compare our design against two baseline interfaces for switching between everyday mobile applications. This comparison highlights the deficiencies of current view-fixed displays, as the Personal Cockpit provides a 40% improvement in application switching time. We demonstrate of several useful implementations and a discussion of important problems for future implementation of our design on current and near-future wearable devices.

Mid-air interactions are prone to fatigue and lead to a feeling of heaviness in the upper limbs, a condition casually termed as the gorilla-arm effect. Designers have often associated limitations of their mid-air interactions with arm fatigue, but do not possess a quantitative method to assess and therefore mitigate it. In this paper we propose a novel metric, Consumed Endurance (CE), derived from the biomechanical structure of the upper arm and aimed at characterizing the gorilla-arm effect. We present a method to capture CE in a non-intrusive manner using an off-the-shelf camera-based skeleton tracking system, and demonstrate that CE correlates strongly with the Borg CR10 scale of perceived exertion. We show how designers can use CE as a complementary metric for evaluating existing and designing novel mid-air interactions, including tasks with repetitive input such as mid-air text-entry. Finally, we propose a series of guidelines for the design of fatigue-efficient mid-air interfaces.

We propose the use of Hand-to-Face input, a method to interact with head-worn displays (HWDs) that involves contact with the face. We explore Hand-to-Face interaction to find suitable techniques for common mobile tasks. We evaluate this form of interaction with document navigation tasks and examine its social acceptability. In a first study, users identify the cheek and forehead as predominant areas for interaction and agree on gestures for tasks involving continuous input, such as document navigation. These results guide the design of several Hand-to-Face navigation techniques and reveal that gestures performed on the cheek are more efficient and less tiring than interactions directly on the HWD. Initial results on the social acceptability of Hand-to-Face input allow us to further refine our design choices, and reveal unforeseen results: some gestures are considered culturally inappropriate and gender plays a role in selection of specific Hand-to-Face interactions. From our overall results, we provide a set of guidelines for developing effective Hand-to-Face interaction techniques.

Users of optical see-through head-mounted displays (OHMD) perceive color as a blend of the display color and the background. Color-blending is a major usability challenge as it leads to loss of color encodings and poor text legibility. Color correction aims at mitigating color blending by producing an alternative color which, when blended with the background, more closely approaches the color originally intended. To date, approaches to color correction do not yield optimal results or do not work in real-time. This paper makes two contributions. First, we present QuickCorrection, a realtime color correction algorithm based on display profiles. We describe the algorithm, measure its accuracy and analyze two implementations for the OpenGL graphics pipeline. Second, we present SmartColor, a middleware for color management of userinterface components in OHMD. SmartColor uses color correction to provide three management strategies: correction, contrast, and show-up-on-contrast. Correction determines the alternate color which best preserves the original color. Contrast determines the color which best warranties text legibility while preserving as much of the original hue. Show-up-on-contrast makes a component visible when a related component does not have enough contrast to be legible. We describe the SmartColor's architecture and illustrate the color strategies for various types of display content.

Several research groups have demonstrated advantages of extending a mobile device's input vocabulary with in-air gestures. Such gestures show promise but are not yet being integrated onto commercial devices. One reason for this might be the uncertainty about users' perceptions regarding the social acceptance of such around-device gestures. In three studies, performed in public settings, we explore users' and spectators' attitudes about using around-device gestures in public. The results show that people are concerned about others' reactions. They are also sensitive and selective regarding where and in front of whom they would feel comfortable using around-device gestures. However, acceptance and comfort are strongly linked to gesture characteristics, such as, gesture size, duration and in-air position. Based on our findings we present recommendations for around-device input designers and suggest new approaches for evaluating the social acceptability of novel input methods.

We present a two-part exploration on mobile true-3D displays, i.e. displaying volumetric 3D content in mid-air. We first identify and study the parameters of a mobile true-3D projection, in terms of the projection's distance to the phone, angle to the phone, display volume and position within the display. We identify suitable parameters and constraints, which we propose as requirements for developing mobile true-3D systems. We build on the first outcomes to explore methods for coordinating the display configurations of the mobile true-3D setup. We explore the resulting design space through two applications: 3D map navigation and 3D interior design. We discuss the implications of our results for the future design of mobile true-3D displays.

We explore grip and motion sensing to afford new techniques that leverage how users naturally manipulate tablet and stylus devices during pen + touch interaction. We can detect whether the user holds the pen in a writing grip or tucked between his fingers. We can distinguish bare-handed inputs, such as drag and pinch gestures produced by the nonpreferred hand, from touch gestures produced by the hand holding the pen, which necessarily impart a detectable motion signal to the stylus. We can sense which hand grips the tablet, and determine the screen's relative orientation to the pen. By selectively combining these signals and using them to complement one another, we can tailor interaction to the context, such as by ignoring unintentional touch inputs while writing, or supporting contextually-appropriate tools such as a magnifier for detailed stroke work that appears when the user pinches with the pen tucked between his fingers. These and other techniques can be used to impart new, previously unanticipated subtleties to pen + touch interaction on tablets.

Applying visual analytics to large software systems can help users comprehend the wealth of information produced by source repository mining. One concept of interest is the co-evolution of test code with source code, or how source and test files develop together over time. For example, understanding how the testing pace compares to the development pace can help test managers gauge the effectiveness of their testing strategy. A useful concept that has yet to be effectively incorporated into a co-evolution visualization is co-change. Co-change is a quantity that identifies correlations between software artifacts, and we propose using this to organize our visualization in order to enrich the analysis of co-evolution. In this paper, we create, implement, and study an interactive visual analytics tool that displays source and test file changes over time (co-evolution) while grouping files that change together (co-change). Our new technique improves the analyst's ability to infer information about the software development process and its relationship to testing. We discuss the development of our system and the results of a small pilot study with three participants. Our findings show that our visualization can lead to inferences that are not easily made using other techniques alone.

Exploring information content on mobile devices can be tedious and time consuming. We present Around-Device Binning, or AD-Binning, a novel mobile user interface that allows users to off-load mobile content in the space around the device. We informed our implementation of AD-Binning by exploring various design factors, such as the minimum around-device target size, suitable item selection methods, and techniques for placing content in off-screen space. In a task requiring exploration, we find that AD-Binning improves browsing efficiency by avoiding the minute selection and flicking mechanisms needed for on-screen interaction. We conclude with design guidelines for off screen content storage and browsing.

Mobile device use while walking, or eyes-busy mobile interaction, is a leading cause of life-threatening pedestrian collisions. We introduce CrashAlert, a system that augments mobile devices with a depth camera, to provide distance and location visual cues of obstacles on the user's path. In a realistic environment outside the lab, CrashAlert users improve their handling of potential collisions, dodging and slowing down for simple ones while lifting their head in more complex situations. Qualitative results outline the value of extending users' peripheral alertness in eyes-busy mobile interaction through non-intrusive depth cues, as used in CrashAlert. We present the design features of our system and lessons learned from our evaluation.

Rich interaction with high-resolution wall displays is not limited to remotely pointing at targets. Other relevant types of interaction include virtual navigation, text entry, and direct manipulation of control widgets. However, most techniques for remotely acquiring targets with high precision have studied remote pointing in isolation, focusing on pointing efficiency and ignoring the need to support these other types of interaction. We investigate high-precision pointing techniques capable of acquiring targets as small as 4 millimeters on a 5.5 meters wide display while leaving up to 93 % of a typical tablet device's screen space available for task-specific widgets. We compare these techniques to state-of-the-art distant pointing techniques and show that two of our techniques, a purely relative one and one that uses head orientation, perform as well or better than the best pointing-only input techniques while using a fraction of the interaction resources.

The Second Usable Privacy and Security for Mobile Devices Workshop (U-PriSM 2) was held with MobileHCI'13. The U-PriSM 2 workshop was an opportunity for researchers and practitioners to discuss research challenges and experiences around the usable privacy and security of mobile devices (smart phones and tablets). Security often involves having non-security experts, or even novice users, regularly making important security decisions while their main focus is on other primary tasks. This is especially true for mobile devices where users can quickly and easily install apps, where user interfaces are minimal due to space constraints, and where users are often distracted by their environment.

Mobile devices are endowed with significant sensing capabilities. However, their ability to 'see' their surroundings, during active use, is limited. We present Surround-See, a self-contained smartphone equipped with an omni-directional camera that enables peripheral vision around the device to augment daily mobile tasks. Surround-See provides mobile devices with a field-of-view collinear to the device screen. This capability facilitates novel mobile tasks such as, pointing at objects in the environment to interact with content, operating the mobile device at a physical distance and allowing the device to detect user activity, even when the user is not holding it. We describe Surround-See's architecture, and demonstrate applications that exploit peripheral 'seeing' capabilities during active use of a mobile device. Users confirm the value of embedding peripheral vision capabilities on mobile devices and offer insights for novel usage methods.

In optical see-through displays, light coming from background objects mixes with the light originating from the display, causing what is known as the color blending problem. Color blending negatively affects the usability of such displays as it impacts the legibility and color encodings of digital content. Color correction aims at reducing the impact of color blending by finding an alternative display color which, once mixed with the background, results in the color originally intended.

We propose a novel video visual analytics system for interactive exploration of surveillance video data. Our approach consists of providing analysts with various views of information related to moving objects in a video. To do this we first extract each object's movement path. We visualize each movement by (a) creating a single action shot image (a still image that coalesces multiple frames), (b) plotting its trajectory in a space-time cube and (c) displaying an overall timeline view of all the movements. The action shots provide a still view of the moving object while the path view presents movement properties such as speed and location. We also provide tools for spatial and temporal filtering based on regions of interest. This allows analysts to filter out large amounts of movement activities while the action shot representation summarizes the content of each movement. We incorporated this multi-part visual representation of moving objects in sViSIT, a tool to facilitate browsing through the video content by interactive querying and retrieval of data. Based on our interaction with security personnel who routinely interact with surveillance video data, we identified some of the most common tasks performed. This resulted in designing a user study to measure time-to-completion of the various tasks. These generally required searching for specific events of interest (targets) in videos. Fourteen different tasks were designed and a total of 120 min of surveillance video were recorded (indoor and outdoor locations recording movements of people and vehicles). The time-to-completion of these tasks were compared against a manual fast forward video browsing guided with movement detection. We demonstrate how our system can facilitate lengthy video exploration and significantly reduce browsing time to find events of interest. Reports from expert users identify positive aspects of our approach which we summarize in our recommendations for future video visual analytics systems.

Foot-based gestures have recently received attention as an alternative interaction mechanism in situations where the hands are pre-occupied or unavailable. This paper investigates suitable real-world mappings of foot gestures to invoke commands and interact with virtual workspaces. Our first study identified user preferences for mapping common mobile-device commands to gestures. We distinguish these gestures in terms of discrete and continuous command input. While discrete foot-based input has relatively few parameters to control, continuous input requires careful design considerations on how the user's input can be mapped to a control parameter (e.g. the volume knob of the media player). We investigate this issue further through three user-studies. Our results show that rate-based techniques are significantly faster, more accurate and result if far fewer target crossings compared to displacement-based interaction. We discuss these findings and identify design recommendations.

The human hand can naturally coordinate multiple finger joints, and simultaneously tilt, press and roll a pen to write or draw. For this reason, digital pens are now embedded with auxiliary input sensors to capture these actions. Prior research on auxiliary input channels has mainly investigated them in isolation of one another. In this work, we explore the coordinated use of two auxiliary channels, a class of interaction techniques we refer to as a-coord input. Through two separate experiments, we explore the design space of a-coord input. In the first study we identify if users can successfully coordinate two auxiliary channels. We found a strong degree of coordination between channels. In a second experiment, we evaluate the effectiveness of a-coord input in a task with multiple steps, such as multi-parameter selection and manipulation. We find that a-coord input facilitates coordination even with a complex, aforethought sequential task. Overall our results indicate that users can control at least two auxiliary input channels in conjunction which can facilitate a number of common tasks can on the pen.

Tabletop systems provide a versatile space for collaboration, yet, in many cases, are limited by the inability to differentiate the interactions of simultaneous users. We present See Me, See You, a lightweight approach for discriminating user touches on a vision-based tabletop. We contribute a valuable characterization of finger orientation distributions of tabletop users. We exploit this biometric trait with a machine learning approach to allow the system to predict the correct position of users as they touch the surface. We achieve accuracies as high as 98% in simple situations and above 92% in more challenging conditions, such as two-handed tasks. We show high acceptance from users, who can self-correct prediction errors without significant costs. See Me, See You is a viable solution for providing simple yet effective support for multi-user application features on tabletops.

We present a robot augmented with an ambient display that communicates using a multi-color halo. We use this robot in a public café-style setting where people vote on which colors the robot will display: we ask people to select a color which "best represents their mood". People can vote from a mobile device (e.g., smart phone or laptop) through a web interface. Thus, the robot's display is an abstract aggregate of the current mood of the room. Our research investigates how a robot with an ambient display may integrate into a space. For example, how will the robot alter how people use or perceive the environment, or how people will interact with the robot itself? In this paper we describe our initial prototype, an iRobot Roomba augmented with lights, and highlight the research questions driving our exploration, including initial study design.

Observational studies indicate that most people use one hand to interact with their mobile devices. Interaction on the back-of-devices (BoD) has been proposed to enhance one-handed input for various tasks, including selection and gesturing. However, we do not possess a good understanding of some fundamental issues related to one-handed BoD input. In this paper, we attempt to fill this gap by conducting three studies. The first study explores suitable selection techniques; the second study investigates the performance and suitability of the two main modes of cursor movement: Relative and Absolute; and the last study examines solutions to the problem of reaching the lower part of the device. Our results indicate that for BoD interaction, relative input is more efficient and accurate for cursor positioning and target selection than absolute input. Based on these findings provide guidelines for designing BoD interactions for mobile devices.

Devices with small viewports (e.g., smartphones or GPS) result in interfaces where objects of interest can easily reside outside the view, into off-screen space. Researchers have addressed this challenge and have proposed visual cues to assist users in perceptually locating off-screen objects. However, little attention has been placed on methods for selecting the objects. Current designs of off-screen cues can result in overlaps that can make it difficult to use the cues as handles through which users can select the off-screen objects they represent. In this paper, we present EdgeSplit, a technique that facilitates both the visualization and selection of off-screen objects on small devices. EdgeSplit exploits the space around the device's borders to display proxies of off-screen objects and then partitions the border regions to allow for non-overlapping areas that make selection of objects easier. We present an effective algorithm that provides such partitioning and demonstrate the effectiveness of EdgeSplit for selecting off-screen objects.

Colocated groups using mobile devices do not share all of the benefits of face-to-face collaborators. Close interaction requires application support for awareness features, allowing participants to establish common ground. Following an overview of research on awareness and grounding, the results of an informal user study are presented, which demonstrate how current systems can deter users from engaging in close collaboration. Literature on awareness provides hope for improving this situation, but a naive transfer to mobile distributed user interfaces will not necessarily succeed. From prior art, a concise list of guidelines has been compiled to assist designers in providing awareness information to users of shared mobile workspaces. These guidelines can also serve as heuristics for the evaluation of future systems. An example is provided to demonstrate how these guidelines can be applied to the development of features for providing awareness of current location and browsing history to colocated users of mobile distributed user interfaces.

Co-located collaborative tasks allow teams to leverage the skills of each individual member. While numerous guidelines exist to develop visualizations for individuals working on desktops, very little is known about how groups of individuals interpret and comprehend diverse types of visual constructs on larger displays. To study whether group size impacts the collective understanding of relationships in three-dimensional (3D) spatial structures when using different types of presentation, we carried out three experiments. We compared individual performance at structure understanding tasks to performance of groups containing two or four members. We consider two alternate visualization techniques for extracting 3D structure information: a 3D view with animated rotations and a combination of one static 3D plus three static two-dimensional (2D) projection views. In general our studies suggest that as group size increases, so does accuracy but with a cost in efficiency. Our results also suggest that beyond a threshold limit in group size, performance on certain tasks begins to degrade. Regardless of group size, participants performed better when the display was presented in the animation condition instead of the multiple static views, except when large groups needed to relate the visualization to a physical counterpart. We summarize our results in terms of Steiner's model for explaining the effects of group size and task characteristics on group performance.

Numerous applications such as simulations, air traffic control systems, and video surveillance systems are inherently composed of spatial objects that move in a scene. In many instances, users can benefit from tools that allow them to select these targets in real-time, without having to pause the dynamic display. However, selecting moving objects is considerably more difficult and error prone than selecting stationary targets. In this paper, we evaluate the effectiveness of several techniques that assist in selecting moving targets. We present Comet, a technique that enhances targets based on their speed and direction. We also introduce Target Ghost, which allows users to select a static proxy of the target, while leaving the motion uninterrupted. We found a speed benefit for the Comet in a 1D selection task in comparison to other cursor and target enhancements. For 2D selection, Comet outperformed Bubble cursor but only when Target Ghost was not available. We conclude with guidelines for design.

Manipulating 3D objects on a tabletop is inherently problematic. Tabletops lack a third degree of freedom and thus require novel solutions to support even the simplest 3D manipulations. Our solution is TZee - a passive tangible widget that enables natural interactions with 3D objects by exploiting the lighting properties of diffuse illumination (DI) multi-touch tabletops. TZee is assembled from stacked layers of acrylic glass to extend the tabletop's infrared light slightly above the surface without supplemental power. With TZee, users can intuitively scale, translate and rotate objects in all three dimensions, and also perform more sophisticated gestures, like "slicing" a volumetric object, that have not been possible with existing tabletop interaction schemes. TZee is built with affordable and accessible materials, and one tabletop surface can easily support multiple TZees. Moreover, since TZee is transparent, there are numerous possibilities to augment interactions with feedback, helpful hints, or other visual enhancements. We discuss several important design considerations and demonstrate the value of TZee with several applications.

Although touch-screen laptops are increasing in popularity, users still do not comfortably rely on touch in these environments, as current software interfaces were not designed for being used by the finger. In this paper, we first demonstrate the benefits of using touch as a complementary input modality along with the keyboard and mouse or touchpad in a laptop setting. To alleviate the frustration users experience with touch, we then design two techniques, TouchCuts, a single target expansion technique, and ,i>TouchZoom,/i>, a multiple target expansion technique. Both techniques facilitate the selection of small icons, by detecting the finger proximity above the display surface, and expanding the target as the finger approaches. In a controlled evaluation, we show that our techniques improve performance in comparison to both the computer mouse and a baseline touch-based target acquisition technique. We conclude by discussing other application scenarios that our techniques support.

The limited viewport size of mobile devices requires that users continuously acquire information that lies beyond the edge of the screen. Recent hardware solutions are capable of continually tracking a user's finger around the device. This has created new opportunities for interactive solutions, such as direct off-screen pointing: the ability to directly point at objects that are outside the viewport. We empirically characterize user performance with direct off-screen pointing when assisted by target cues. We predict time and accuracy outcomes for direct off-screen pointing with existing and derived models. We validate the models with good results (R2 ≥ 0.9) and reveal that direct off-screen pointing takes up to four times longer than pointing at visible targets, depending on the desired accuracy tradeoff. Pointing accuracy degrades logarithmically with target distance. We discuss design implications in the context of several real-world applications.

In this paper we describe the use of kick gestures for interaction with mobile devices. Kicking is a well-st udied leg action that can be harnessed in mobile contexts where the hands are busy or too dirty to interact with the phone. In this paper we examine the design space of kicki ng as an interaction technique through two user studies. The first study investigated how well users were able to control the direction of their kicks. Users were able to aim their kicks best when the movement range is divided into segments of at least 24°. In the second study we looked at the velocity of a kick. We found that the users are able to kick with at least two varying velocities. However, they also often undershoot the target velocity. Finally, we propose some specific applications in which kicks can prove beneficial.

Projector phones, handheld game consoles and many other mobile devices increasingly include more than one display, and therefore present a new breed of mobile Multi-Display Environments (MDEs) to users. Existing studies illustrate the effects of visual separation between displays in MDEs and suggest interaction techniques that mitigate these effects. Currently, mobile devices with heterogeneous displays such as projector phones are often designed without reference to visual separation issues; therefore it is critical to establish whether concerns and opportunities raised in the existing MDE literature apply to the emerging category of Mobile MDEs (MMDEs). This paper investigates the effects of visual separation in the context of MMDEs and contrasts these with fixed MDE results, and explores design factors for Mobile MDEs. Our study uses a novel eye-tracking methodology for measuring switches in visual context between displays and identifies that MMDEs offer increased design flexibility over traditional MDEs in terms of visual separation. We discuss these results and identify several design implications.

We consider different hierarchical menu and toolbar-like interface designs from a theoretical perspective and show how a model based on visual search time, pointing time, decision time and expertise development can assist in understanding and predicting interaction performance. Three hierarchical menus designs are modelled -- a traditional pull-down menu, a pie menu and a novel Square Menu with its items arranged in a grid -- and the predictions are validated in an empirical study. The model correctly predicts the relative performance of the designs -- both the eventual dominance of Square Menus compared to traditional and pie designs and a performance crossover as users gain experience. Our work shows the value of modelling in HCI design, provides new insights about performance with different hierarchical menu designs, and demonstrates a new high-performance menu type.

Accelerometers are common on many devices, including those required for text-entry. We investigate how to enter text with devices that are solely enabled with accelerometers. The challenge of text-entry with such devices can be overcome by the careful investigation of the human limitations in gestural movements with accelerometers. Preliminary studies provide insight into two potential text-entry designs that purely use accelerometers for gesture recognition. In two experiments, we evaluate the effectiveness of each of the text-entry designs. The first experiment involves novice users over a 45 minute period while the second investigates the possible performance increases over a four day period. Our results reveal that a matrix-based text-entry system with a small set of simple gestures is the most efficient (5.4wpm) and subjectively preferred by participants.

We introduce LensMouse, a novel device that embeds a touch-screen display -- or tangible 'lens' -- onto a mouse. Users interact with the display of the mouse using direct touch, whilst also performing regular cursor-based mouse interactions. We demonstrate some of the unique capabili-ties of such a device, in particular for interacting with auxil-iary windows, such as toolbars, palettes, pop-ups and dia-log-boxes. By migrating these windows onto LensMouse, challenges such as screen real-estate use and window man-agement can be alleviated. In a controlled experiment, we evaluate the effectiveness of LensMouse in reducing cursor movements for interacting with auxiliary windows. We also consider the concerns involving the view separation that results from introducing such a display-based device. Our results reveal that overall users are more effective with LenseMouse than with auxiliary application windows that are managed either in single or dual-monitor setups. We conclude by presenting other application scenarios that LensMouse could support.

Most studies on tilt based interaction can be classified as point-designs that demonstrate the utility of wrist-tilt as an input medium; tilt parameters are tailored to suit the specific interaction at hand. In this paper, we systematically analyze the design space of wrist-based interactions and focus on the level of control possible with the wrist. In a first study, we investigate the various factors that can influence tilt control, separately along the three axes of wrist movement: flexion/extension, pronation/supination, and ulnar/radial deviation. Results show that users can control comfortably at least 16 levels on the pronation/supination axis and that using a quadratic mapping function for discretization of tilt space significantly improves user performance across all tilt axes. We discuss the findings of our results in the context of several interaction techniques and identify several general design recommendations.

Interaction patterns with handheld mobile devices are constantly evolving. Researchers observed that users prefer to interact with mobile device using one hand. However, only few interaction techniques support this mode of operation. We show that one-handed operations can be enhanced with coordinated interaction using for input the front and back of a mobile device, which we term as Dual-Surface interaction. We present some of the design rationale for introducing coordinated Dual-Surface interactions. We demonstrate that several tasks, including target selection, benefit from Dual-Surface input which allows users to rapidly select small targets in locations that are less accessible when interacting using the thumb with one-handed input. Furthermore, we demonstrate the benefits of virtual enhancements that are possible with behind-the-display relative input to perform complex tasks, such as steering. Our results show that Dual-Surface interactions offer numerous benefits that are not available with input on the front or the back alone.

We introduce ARC-Pad (Absolute+Relative Cursor pad), a novel technique for interacting with large displays using a mobile phone's touchscreen. In ARC-Pad we combine ab-solute and relative cursor positioning. Tapping with ARC-Pad causes the cursor to jump to the corresponding location on the screen, providing rapid movement across large distances. For fine position control, users can also clutch using relative mode. Unlike prior hybrid cursor positioning techniques, ARC-Pad does not require an explicit switch between relative and absolute modes. We compared ARC-Pad with the relative positioning commonly found on touchpads. Users were given a target acquisition task on a large display, and results showed that they were faster with ARC-Pad, without sacrificing accuracy. Users welcomed the benefits associated with ARC-Pad.

Current interactions on direct-touch interactive surfaces are often modeled based on properties of the input channel that are common in traditional graphical user interfaces (GUI) such as x-y coordinate information. Leveraging additional information available on the surfaces could potentially result in richer and novel interactions. In this paper we specifically explore the role of finger orientation. This property is typically ignored in touch-based interactions partly because of the ambiguity in determining it solely from the contact shape. We present a simple algorithm that unambiguously detects the directed finger orientation vector in real-time from contact information only, by considering the dynamics of the finger landing process. Results of an experimental evaluation show that our algorithm is stable and accurate. We then demonstrate how finger orientation can be leveraged to enable novel interactions and to infer higher-level information such as hand occlusion or user position. We present a set of orientation-aware interaction techniques and widgets for direct-touch surfaces. ACM Classification: I.3.6 [Methodology and Techniques]: Interaction techniques.

The development of intelligent assistants has largely benefited from the adoption of decision-theoretic (DT) approaches that enable an agent to reason and account for the uncertain nature of user behaviour in a complex software domain. At the same time, most intelligent assistants fail to consider the numerous factors relevant from a human-computer interaction perspective. While DT approaches offer a sound foundation for designing intelligent agents, these systems need to be equipped with an interaction cost model in order to reason the impact of how (static or adaptive) interaction is perceived by different users. In a DT framework, we formalize four common interaction factors --- information processing, savings, visual occlusion, and bloat. We empirically derive models for bloat and occlusion based on the results of two users experiments. These factors are incorporated in a simulated help assistant where decisions are modeled as a Markov decision process. Our simulation results reveal that our model can easily adapt to a wide range of user types with varying preferences.

Zooming interfaces use animated transitions to smoothly shift the users view between different scales of the workspace. Animated transitions assist in preserving the spatial relationships between views. However, they also increase the overall interaction time. To identify whether zooming interfaces should take advantage of animations, we carried out one experiment that explores the effects of smooth transitions on a spatial task. With metro maps, users were asked to identify the number of metro stops between different subway lines with and without animated zoom-in/out transitions. The results of the experiment show that animated transitions can have significant benefits on user performance - participants in the animation conditions were twice as fast and overall made fewer errors than in the non-animated conditions. In addition, short animations were found to be as effective as long ones, suggesting that some of the costs of animations can be avoided. Users also preferred interacting with animated transitions than without. Our study gives empirical evidence on the benefits of animated transitions in zooming interfaces.

Multi-flick, which consists of repeated flick actions, has received popular media attention as an intuitive and natural document-scrolling technique for stylus based systems. In this paper we put multi-flick to test, by designing several flick-based scrolling techniques. We first map out the de-sign space of multi-flick and identify mapping functions that make multi-flick a natural and intuitive technique for document navigation. In the first experiment we compare several multi-flick variations for navigating lists on three different devices -- a PDA, a tabletPC, and a large table. Our study shows that compound-multi-flick (CMF) is the most preferred technique and it is at least as fast, if not faster than the traditional scrollbar. In a follow-up experiment, we evaluate multi-flick for scrolling text-based documents. Results show that all implementations of multi-flick are as good as the scrollbar for short distances while CMF is the most preferred. We discuss the implications of our findings and present several design guidelines.

To overcome display limitations of small-screen devices, researchers have proposed techniques that point users to objects located off-screen. Arrow-based techniques such as City Lights convey only direction. Halo conveys direction and distance, but is susceptible to clutter resulting from overlapping halos. We present Wedge, a visualization technique that conveys direction and distance, yet avoids overlap and clutter. Wedge represents each off-screen location using an acute isosceles triangle: the tip coincides with the off-screen locations, and the two corners are located on-screen. A wedge conveys location awareness primarily by means of its two legs pointing towards the target. Wedges avoid overlap programmatically by repelling each other, causing them to rotate until overlap is resolved. As a result, wedges can be applied to numbers and configurations of targets that would lead to clutter if visualized using halos. We report on a user study comparing Wedge and Halo for three off-screen tasks. Participants were significantly more accurate when using Wedge than when using Halo.

Studies investigating user control of pressure input have reported time-accuracy trade-offs of, on average, over 30%, when interacting with a large number of pressure levels. To increase the level of control with pressure input, we designed and evaluated four different discretization functions: linear, fisheye, visual fisheye, and clustered. The fisheye discretization dynamically modifies the range of pressure values based on the position of the pressure cursor. Our results show that a fisheye function results in significantly lower error rates and a lower number of crossings than have been reported in the literature. Furthermore, the fisheye function improves control without compromising speed. We discuss the findings of our study and identify several design recommendations for integrating pressure control into common interface tasks.

Selecting items in cascading pull-down menus is a frequent task in most GUIs. These selections involve two major components: steering and selection, with the steering component being the most time-consuming and error-prone. We describe a new technique, called Adaptive Activation-Area Menu (AAMU) that eliminate corner steering. AAMUs contain an enlarged activation area which dynamically resizes itself providing a broader steering path for menu navigation. We also combined AAMUs with Force-field menus, to create Force-AAMUs. We empirically demonstrate that AAMUs and Force-AAMUs outperformed the current default menu. We also compared performances of various other menus including Enlarged activation area menus (EMUs) and Gesture based selection with mouse as an input device. Overall, users show higher satisfaction rates for AAMUs over other menu designs.

Frequent itemset mining plays an essential role in the mining of many different patterns. Most existing frequent itemset mining algorithms return the mined results--namely, frequent itemsets--in the form of textual lists. However, the use of visual representation can enhance the user understanding of the inherent relations in a collection of frequent itemsets. In this paper, we propose an effective visualizer, called WiFIsViz, to display the mined frequent itemsets. WiFIsViz provides users with an overview and details about the itemsets. Moreover, this visualizer is also equipped with several interactive features for effective visualization of the frequent itemsets mined from various real-life applications.

In this paper we investigate the use of a uni-pressure and dual-pressure augmented mouse. With a pressure augmented mouse users can simultaneously control cursor positions as well as multiple levels of discrete selection modes for common desktop application tasks. Two or more independent pressure sensors can be mounted onto several locations on the body of the mouse. To highlight the design potential of a pressure augmented mouse we conducted a multi-part study. In the first part we identified the number of maximum discrete levels controllable with a uni-pressure augmented mouse, the most appropriate locations for installing pressure sensors on the mouse, and the design of new interaction techniques to support selection with pressure-based input. In a follow-up design we introduced an additional sensor and two different types of selection techniques to control a larger number of discrete levels with two pressure sensors. Our results show that users can comfortably control up to 64 modes with a dual-pressure augmented mouse. We discuss the findings of our results in the context of several desktop interaction techniques and identify several design recommendations.

Embodiments are visual representations of people in a groupware system. Embodiments convey awareness information such as presence, location, and movement -- but they provide far less information than what is available from a real body in a face-to-face setting. As a result, it is often difficult to recognize and characterize other people in a groupware system without extensive communication. To address this problem, information-rich embodiments use ideas from multivariate information visualization to maximize the amount of information that is represented about a person. To investigate the feasibility of rich embodiment and their effects on group interaction, we carried out three studies. The first shows that users are able to recall and interpret a large set of variables that are graphically encoded on an embodiment. The second and third studies demonstrated rich embodiments in two groupware systems -- a multiplayer game and a drawing application -- and showed that the enhanced representations do improve recognition and characterization, and that they can enrich interaction in a variety of ways.

Many visualizations use smoothly animated transitions to help the user interact with information structures. These transitions are intended to preserve perceptual constancy during viewpoint transformations. However, animated transitions also have costs -- they increase the transition time, and they can be complicated to implement -- and it is not clear whether the benefits of smooth transitions outweigh the costs. In order to quantify these benefits, we carried out two experiments that explore the effects of smooth transitions. In the first study, subjects were asked to determine whether graph nodes were connected, and navigated the graph either with or without smooth scene transitions. In the second study, participants were asked to identify the overall structure of a tree after navigating the tree through a viewport that either did or did not use smooth transitions for view changes. The results of both experiments show that smooth transitions can have dramatic benefits for user performance -- for example, participants in smooth transition conditions made half the errors of the discrete-movement conditions. In addition, short transitions were found to be as effective as long ones, suggesting that some of the costs of animations can be avoided. These studies give empirical evidence on the benefits of smooth transitions, and provide guidelines about when designers should use them in visualization systems.

The decision making process is usually one of the most critical elements of any face-to-face meeting. Participants in a group meeting follow certain procedures and guidelines for facilitating the decision making process. These include such rules as turn taking, not interrupting the consultation, and keeping suggestions or ideas clear and concise. The ultimate objective of such meetings is to arrive at decisions in a timely manner by having a diversity of opinions from as many participants as possible. We postulate that by visualizing the group dynamics during a face-to-face meeting, administrators might be able to get a better handle on the strengths and weaknesses of the group's consultation process. The end result would be to improve the overall quality of meetings. We present a visualization tool that captures and reveals the ongoing social dynamics during the decision-making process within a face-to-face and real-time meeting. Our system captures the required data through a simple and easy-to-use interface and then visualizes the outcome of the meeting.

Michotte's theory of ampliation suggests that causal relationships are perceived by objects animated under appropriate spatiotemporal conditions. We extend the theory of ampliation and propose that the immediate perception of complex causal relations is also dependent on a set of structural and temporal rules. We designed animated representations, based on Michotte's rules, for showing complex causal relationships or causal semantics. In this paper we describe a set of animations for showing semantics such as causal amplification, causal strength, causal dampening, and causal multiplicity. In a two part study we compared the effectiveness of both the static and animated representations. The first study (N=44) asked participants to recall passages that were previously displayed using both types of representations. Participants were 8% more accurate in recalling causal semantics when they were presented using animations instead of static graphs. In the second study (N=112) we evaluated the intuitiveness of the representations. Our results showed that while users were as accurate with the static graphs as with the animations, they were 9% faster in matching the correct causal statements in the animated condition. Overall our results show that animated diagrams that are designed based on perceptual rules such as those proposed by Michotte have the potential to facilitate comprehension of complex causal relations.

Many systems provide the user with a limited viewport of a larger graphical workspace. In these systems, the user often needs to find and select targets that are in the workspace, but not visible in the current view. Standard methods for navigating to the off-screen targets include scrolling, panning, and zooming; however, these are laborious when users cannot see a target's direction or distance. Techniques such as halos can provide awareness of targets, but actually getting to the target is still slow with standard navigation. To improve off-screen target selection, we developed a new technique called hop, which combines halos with a teleportation mechanism that shows proxies of distant objects. Hop provides both awareness of off-screen targets and fast navigation to the target context. A study showed that users are significantly faster at selecting off-screen targets with hopping than with two-level zooming or grab-and-drag panning, and it is clear that hop will be faster than either halos or proxy-based techniques (like drag-and-pop or vacuum filtering) by themselves. Hop both improves on halo-based navigation and extends the value of proxies to small-screen environments.

We have developed a visual syntax for representing concepts that are contingent on temporal properties (time-dependent semantics). A within-group (N=24) experiment was conducted to identify the representations that conveyed best a given semantic. We then applied our representations to the visualization of algorithms in quantum computing and carried out a second experiment (N=16) on subjects unfamiliar with the semantic concepts that were tested. The results show that our representations are intuitive and facilitate a good level of understanding of the algorithms.

Shading information is extracted by the human visual system during the earliest stages of the object recognition process. While shading can enhance the visibility of structures, it can have a negative impact on the judgment of sizes of elements in a structure. In certain visualization systems the underlying hierarchical structure is not noticeably explicit, such as in space-filling techniques. We hypothesize that in such cases, shading can make the structure more explicit. In This work we report the results of a study comparing two space-filling visualizations: one with and the other without shading. Our results show that on structure-based tasks, users performed better with the tool when shading information was included than without. However we did not obtain statistically significant results to suggest that shading information degraded users' performance on tasks requiring comparison of local features such as file sizes. A subjective evaluation shows that users preferred interacting with the system when shading was available.

Language theorists argue that the reason why spoken language is acquired so rapidly is that we have an innate predisposition for understanding linguistic structures. Theories of perception also hold that there may be deeply seated mechanisms for decomposing visual objects and analyzing them into both component parts and the structural interrelationships of those parts. We propose the theory that diagrams that activate the mechanisms for structural object perception should be similarly easy to learn. This builds on previous work in which we have developed diagramming principles based on the theory of structural object perception. We call these geon diagrams. We have previously shown that such diagrams are easy to remember and to analyze. To evaluate our hypothesis that geon diagrams should also be easy to understand we carried out an empirical study to evaluate the learnability of geon diagram semantics in comparison with the well-established UML convention. The results support our theory of learnability. Both "novices" and "experts" found the geon diagram syntax easier to apply in a diagram-to-textual description matching task than the equivalent UML syntax.

This work reports on ongoing work related to the representation of hierarchical structures using multiple synthetic voices. We manipulated three synthetic voice parameters, average pitch, pitch range and speech rate, to represent nodes in a hierarchy. We created hierarchies containing 10 nodes and three levels deep. A within-subjects design (N=12) was conducted to compare the effect of multiple synthetic voices to single synthetic voices for locating the positions of items in a hierarchy. Subjects were trained with the set of rules we used for constructing the multiple synthetic voices. In a node-finding task, participants identified the position of a previously listened-to node. Our results show that subjects recalled the nodes' positions in the hierarchy significantly better when the hierarchies were equipped with multiple synthetic voices than without.

The class of diagrams known collectively as node-link diagrams are used extensively for many applications, including planning, communications networks, and computer software. The defining features of these diagrams are nodes, represented by a circle or rectangle connected by links usually represented by some form of line or arrow. We investigate the proposition that drawing three-dimensional shaded elements instead of using simple lines and outlines will result in diagrams that are easier to interpret. A set of guidelines for such diagrams is derived from perception theory and these collectively define the concept of the geon diagram. We also introduce a new substructure identification task for evaluating diagrams and use it to test the effectiveness of geon diagrams. The results from five experiments are reported. In the first three experiments geon diagrams are compared to Unified Modeling Language (UML) diagrams. The results show that substructures can be identified in geon diagrams with approximately half the errors and significantly faster. The results also show that geon diagrams can be recalled much more reliably than structurally equivalent UML diagrams. In the final two experiments geon diagrams are compared with diagrams having the same outline but not constructed with shaded solids. This is designed to specifically test the importance of using 3D shaded primitives. The results also show that substructures can be identified much more accurately with shaded components than with 2D outline equivalents and remembered more reliably. Implications for the design of diagrams are discussed.

Most diagrams, particularly those used in software engineering, are line drawings consisting of nodes drawn as rectangles or circles, and edges drawn as lines linking them. In the present paper we review some of the literature on human perception to develop guidelines for effective diagram drawing. Particular attention is paid to structural object recognition theory. According to this theory as objects are perceived they are decomposed into 3D set of primitives called geons, together with the skeleton structure connecting them. We present a set of guidelines for drawing variations on node-link diagrams using geon-like primitives, and provide some examples. Results from three experiments are reported that evaluate 3D geon diagrams in comparison with 2D UML (Unified Modeling Language) diagrams. The first experiment measures the time and accuracy for a subject to recognize a sub-structure of a diagram represented either using geon primitives or UML primitives. The second and ...