People with Blind Visual Impairments (BVI) face unique challenges when sharing images, as these may accidentally contain sensitive or inappropriate content. In many instances, they are unaware of the potential risks associated with sharing such content, which can compromise their privacy and interpersonal relationships. To address this issue, we investigated image filtering techniques that could help BVI users manage sensitive content before sharing with various audiences, including family, friends, or strangers. We conducted a study with 20 BVI participants, evaluating different filters applied to images varying in sensitivity, such as personal moments or embarrassing shots. Results indicated that pixelation was the least preferred method, while preferences for other filters varied depending on image type and sharing context. Additionally, participants reported greater comfort when sharing filtered versus unfiltered images across audiences. Based on the results, we offer a set of design guidelines to enhance the image-sharing experience for BVI individuals.

A central goal of Augmented Reality (AR)-based Situated Visualization (SV) is to seamlessly integrate digital information into its relevant physical context. While existing frameworks describe numerous design dimensions, the field lacks a rigorous model to evaluate this integration. To address this, we introduce SCORE, a framework for quantifying diegesis - a narratology concept describing the extent to which an element belongs to its narrative world. Grounded in a systematic analysis of 50 contemporary SV works, SCORE defines five dimensions of diegesis in SV: Spatial proximity, Concreteness, cOherence, Referential context, and Environmental context. In addition to qualitative comparison, our framework also provides a quantitative measure of diegesis, enabling SCORE to distinguish AR-based SVs that prior models have treated as theoretically equivalent. We validate the framework by demonstrating a consistent correlation between higher scores and positive usability outcomes. Based on these findings, we offer insights for SV designers.

Large curved displays are ideal for viewing 360° content, such as 3D maps, but typically restrict users to a 180° viewport, leaving information off-screen. Since users naturally direct their heads toward regions on-screen before interacting, head movements offer a promising alternative for workspace manipulation to bring off-screen content into view. We explore rate control functions (linear, sigmoid, polynomial) and zone control functions (continuous, friction, interrupted, additive) to translate head rotations into workspace control, enabling users to access off-screen content. Polynomial rate control emerges as the best choice, achieving the fastest trial times and highest subjective ratings. Using a map navigation task, our second study demonstrates that users perform better with the polynomial head-based technique than with the industry-standard controller-based methods, click-and-drag and joystick-push, for 360° workspace navigation. Based on these findings, we provide guidelines to inform the design of future 360° workspace navigation techniques for large curved displays.

Teleportation in Virtual Reality (VR) allows users to instantly move between locations, typically by aiming at a destination with a parabolic ray emitted from a controller and confirming the action with a button press. Recent advances in hand-tracking within Head-Mounted Displays (HMDs) have enabled researchers to explore hand-based teleportation, but these techniques often suffer from reduced pointing accuracy and increased effort. To address these challenges, we introduce NinjaPort, a novel, hand-based teleportation technique that allows users to control four virtual copies of their dominant hand with rays, each positioned at a different distance. Users can quickly select a desired teleportation distance by pinching a specific finger on their non-dominant hand, where each finger (e.g., index to pinky) maps to one of the four virtual hands. In our first user study, we explored design parameters to determine the optimal spatial arrangement of the virtual hands. Based on the findings, we refined NinjaPort and conducted a second study comparing it against a commercially available hand-based teleportation method and a research-based alternative. Our findings show that NinjaPort enables more accurate and faster teleportation, with less hand rotation. We conclude with design guidelines for creating effective and user-friendly hand-based VR teleportation techniques.

To interact with Augmented Reality (AR) content while walking, the user interfaces (UIs) need to move along with the user without distracting their field of view. This paper investigates on-hand reference frames for AR interaction while walking. First, we conduct a user study evaluating six on-hand reference frames. Results show that the Pinch Grip With Offset (PGWO), which anchors UIs to the pinch grip while floating at a distance, outperforms other on-hand reference frames regarding speed, accuracy, workload, and user preference. Next, we conduct a follow-up study to compare PGWO’s performance with head and torso reference frames, commonly used in previous studies, to see whether PGWO’s benefits hold up against well-established reference frames. Results revealed better performance and higher user preference for PGWO than both reference frames. Finally, we present design recommendations for developing future AR systems that are more efficient and user-friendly for on-the-go interaction.

Hand Proximate User Interfaces (HPUI) on Head Mounted Displays (HMD) leverage hand tracking to anchor content on the hand and interact with it using thumb-to-finger interactions. Similar to many other interaction techniques on HMDs, HPUI realizes these interactions by combining simple geometry in game engines. This, in turn, leads to accidental triggers, akin to the "fat-finger problem" on touch screens. To address this, we explore and provide insight into how the thumb’s surface interacts when using HPUI by approximating the thumb’s surface with a large number of raycasts. We observe that different regions of the thumb are used when interacting with different parts of the hand. The results also highlight the need to consider the temporal component. We then propose approaches to improving the precision of thumb-to-finger interactions on HPUI and show that these improve target selection accuracy with denser target layouts.

Researchers explore various on-body locations for anchoring user interfaces in Augmented Reality (AR), primarily to leverage onbody haptic feedback and enhance usability. However, most studies are conducted in controlled laboratory settings, limiting their applicability to real-world use. Consequently, there remains a lack of research on identifying suitable on-body AR UI placements across diverse social and mobility contexts. To address this gap, we conduct a user study investigating user preferences for anchoring UIs on six on-body locations-palm, back of the palm, inner and outer forearm, and right and left lap-across different mobility conditions (standing, walking, sitting) and social contexts (private, semiprivate, and public settings). Results show that users prefer anchoring AR interfaces on the outer forearm and palm across all conditions. However, participants noted the limited interaction space on hand surfaces. To address this, a follow-up study evaluated six UI layouts anchored to the palm and forearm, comparing body-aligned vs. vertically oriented placements and small vs. enlarged interfaces. Results show that vertically oriented, enlarged layouts yield superior performance, offering context-sensitive guidance for designing AR interfaces that balance comfort, preference, and usability.

Teleportation in Virtual Reality (VR) is a locomotion technique that allows users to navigate between locations within a virtual environment instantly. Traditionally, VR teleportation is performed using physical controllers, where users control a teleportation pointer — represented by a straight line or parabola — and activate the teleportation to the target destination by pressing a button. Recent advances in hand and eye-tracking capabilities in Head-Mounted Displays (HMDs) enable designers to leverage hand and eye-based interactions to enhance the immersion and naturalness of controller-free VR usage. However, there has been limited research on comparing different controller-free methods for VR teleportation across various elevations. To address this gap, we conducted a user study exploring three controller-free pointing techniques (gaze, hand, and head), four confirmation modalities (finger pinch, eye-blink, dwell, and voice), and two types of teleportation pointers (linear and parabolic) for VR teleportation across various elevations. Our results show that head-based pointing was faster and more accurate than other techniques, with head and gaze achieving higher throughput than hand-based methods. For confirmation, finger pinch yielded the best performance in terms of task completion time and throughput, followed by dwell, voice, and eye-blink; dwell was the most accurate. The linear pointer outperformed the parabolic pointer in some contexts. Based on these findings, we propose design guidelines to enhance controller-free VR teleportation using various input modalities.

Designing a comfortable, familiar, and efficient one-handed text entry method for Head-Mounted Displays (HMDs) remains a significant challenge. Existing midair typing systems induce fatigue, while novel techniques often demand extensive training or sacrifice input efficiency. Consequently, we introduce ThumbSwype , a novel thumb-to-finger text entry technique that adapts smartphone swipe typing for HMDs. Users see the traditional QWERTY keyboard overlaid on their index, middle, and ring fingers, allowing them to perform swipe gestures with their thumb to type words. In an evaluation study (N=16) , participants achieved a mean of 14.52 words per minute (WPM), which is 63.8% of their smartphone swipe-typing performance, with a peak average of 20.2 WPM. We compare ThumbSwype’s performance with related work, and discuss directions for future improvement.

Head-mounted displays (HMDs) enable users to navigate the Reality-Virtuality Continuum, facilitating transitions between the Real world, Augmented Reality, Augmented Virtuality, and the Virtual world. Traditional transition methods use double taps on HMDs or buttons on handheld controllers to transition between the worlds. However, this can often disrupt hands-free interaction and hinder the overall immersion. Although prior work explored transitioning within a reality, little is known about solutions facilitating transitioning across multiple worlds. In this paper, we investigate index finger-based solutions for transitioning between multiple realities. We designed and fabricated finger-worn button configurations of 2 × 2, 2 × 1, and 4 × 1, and compared them with finger-worn solutions such as Joystick, Rotary wheel, and Slider. The results showed that the 2 × 2 button configuration is the most effective technique, minimizing trial time and ensuring user comfort. Overall, this research enhances VR user experiences by improving interaction techniques for fluid switching between realities in the Reality-Virtuality Continuum.

Augmented Reality User Interfaces commonly stay floated in midair unless explicitly moved by users. This often results in suboptimal performance due to the absence of haptic feedback and arm fatigue. We explore the transition of AR Interfaces to leverage onbody and physical surfaces, e.g., the arm and desk, in addition to mid-air. We begin with a user study to assess the potential of these surfaces for transitioning virtual UIs. Study results indicate a strong user preference for transitioning interfaces from mid-air to physical surfaces when they are available. We further explore three UI transition mechanisms: manual, semi-automatic, and automatic, each with varying levels of automation and user control. Results from a user study reveal that semi-automatic transition is the most preferred method, as it offers a good balance between automation and user control. We conclude with design guidelines for transitioning AR UIs across mid-air, on-body, and physical surfaces.

Teleportation, a widely-used locomotion technique in Virtual Reality (VR), allows instantaneous movement within VR environments. Enhanced hand tracking in modern VR headsets has popularized hands-only teleportation methods, which eliminate the need for physical controllers. However, these techniques have not fully explored the potential of bi-manual input, where each hand plays a distinct role in teleportation: one controls the teleportation point and the other confirms selections. Additionally, the influence of users’ posture, whether sitting or standing, on these techniques remains unexplored. Furthermore, previous teleportation evaluations lacked assessments based on established human motor models such as Fitts’ Law. To address these gaps, we conducted a user study $( \mathrm{N}=20)$ to evaluate bi-manual pointing performance in VR teleportation tasks, considering both sitting and standing postures. We proposed a variation of the Fitts’ Law model to accurately assess users’ teleportation performance. We designed and evaluated various bi-manual teleportation techniques, comparing them to uni-manual and dwell-based techniques. Results showed that bi-manual techniques, particularly when the dominant hand is used for pointing and the non-dominant hand for selection, enable faster teleportation compared to other methods. Furthermore, bi-manual and dwell techniques proved significantly more accurate than uni-manual teleportation. Moreover, our proposed Fitts’ Law variation more accurately predicted users’ teleportation performance compared to existing models. Finally, we developed a set of guidelines for designers to enhance VR teleportation experiences and optimize user interactions.

Touchscreen is the primary solution to interact with public devices such as Automated Teller Machines (ATMs). However, the touch modality raises health concerns since users have to touch the screens, and therefore risking the spread of contagious diseases. We design PalmSpace, an alternate input technique leveraging users’ hand palms to interact with public devices. With Palmspace, UI elements are mapped onto the users’ palms and can be accessed by touching various locations directly on the palm. We conduct a series of user studies to evaluate several design options, such as interface layout, item size, preferred item location, and suitable feedback for items. Based on the results, we design PalmSpace and compare its performance with mid-air input. We show that PalmSpace is a potential solution to interact with public devices without using their touchscreen. We conclude with design guidelines for using the palm as an alternative input space for touchscreen devices.

Large curved displays are becoming increasingly popular due to their ability to provide users with a wider field of view and a more immersive experience compared to flat displays. Current interaction techniques for large curved displays often assume a user is positioned at the display's centre, crucially failing to accommodate general use conditions where the user may move during use. In this work, we investigated how user position impacts pointing interaction on large curved displays and evaluated cursor enhancement techniques to provide faster and more accurate performance across positions. To this effect, we conducted two user studies. First, we evaluated the effects of user position on pointing performance on a large semi-circular display (3m-tall, 3270R curvature) through a 2D Fitts' Law selection task. Our results indicate that as users move away from the display, their pointing speed significantly increases (at least by 9%), but accuracy decreases (by at least 6%). Additionally, we observed participants were slower when pointing from laterally offset positions. Secondly, we explored which pointing techniques providing motor- and visual-space enhancements best afford effective pointing performance across user positions. Across a total of six techniques tested, we found that a combination of acceleration and distance-based adjustments with cursor enlargement significantly improves target selection speed and accuracy across different user positions. Results further show techniques with visual-space enhancements (e.g., cursor enlargement) are significantly faster and more accurate than their non-visually-enhanced counterparts. Based on our results we provide design recommendations for implementing cursor enhancement techniques for large curved displays.

Smartphone overuse around family and friends has been shown to be increasing over the past years and often leads to limited one-to-one interaction between co-located individuals. Smartphone-based virtual agents have been shown to be effective for behavior intervention and mediation, such as promoting physical activity. Little is known about leveraging smartphone-based agents to play a role in communication and facilitate conversation between co-located individuals. In this paper, we explore strengthening conversations between co-located couples by introducing a smartphone-based agent that acts as a conversation facilitator between them. We contrast the results with a text-based alternative. Our findings suggest that virtual agents serve as a valuable social entity mediating support in couples' communication and relationship dynamics. Through this, we suggest design considerations for this context that leverage the unique qualities of virtual agents.

The increasing interest in thermal haptic feedback devices, particularly for virtual reality (VR) applications, highlights the need for more immersive user experiences. However, replicating precise thermal sensations on the fingers remains challenging due to the complexity of finger joints and movements. In this paper, we introduce ThermoGrasp, a novel thermal display designed to enhance VR experiences by providing realistic thermal feedback during precision object grasping. ThermoGrasp is a modular wearable device that targets controlled thermal feedback on the distal phalanges. The implications of designing its VR application were assessed through two experimental studies. The first study focused on the device's ability to accurately convey thermal sensations across different fingers during various precision grasps. The second study investigated the overall haptic experience in VR, examining the impact of thermal feedback on user immersion and realism during interactions with objects of varying temperatures. Participants' subjective responses were analyzed based on factors such as autotelicity, expressiveness, immersion, realism, and harmony. The findings indicate that precise, localized thermal feedback significantly enhances the VR experience, offering a marked improvement over traditional haptic feedback methods.

Mobile devices have the potential to facilitate remote tasks through Augmented Reality (AR) solutions by integrating digital information into the real world. Although prior studies have explored Mobile Augmented Reality (MAR) for co-located collaboration, none have investigated the impact of various viewing attributes that can influence remote task performance, such as target object viewing angles, synchronization styles, or having a secondary small screen showing other users current view in the MAR environment. In this paper, we explore five techniques considering these attributes, specifically designed for two modes of remote tasks: collaborative and competitive. We conducted a user study employing various combinations of those attributes for both tasks. In both instances, results indicate users' optimal performance and preference for the technique that allows asynchronous viewing of object manipulations on the small screen. Overall, this paper contributes novel techniques for remote tasks in MAR, addressing aspects such as viewing angle and synchronization in object manipulation alongside secondary small-screen interfaces. Additionally, it presents the results of a user study evaluating the effectiveness, usability, and user preference of these techniques in remote settings and offers a set of recommendations for designing and implementing MAR solutions to enhance remote activities.

With the recent Virtual Reality (VR) Head-Mounted Displays (HMDs), users can seamlessly transition between the virtual and real worlds with techniques such as passthrough. These techniques leverage on-device cameras to capture the real world and show users a view of their physical surroundings while wearing the HMDs. However, they often require users to hold a controller or frequently tap on the HMD, limiting the potential for hands-free interaction and thereby hindering a truly immersive and natural VR experience. To address this limitation, we designed FingerButton, a finger-worn push button device that enables seamless transitions between real and virtual environments. We conducted two studies, where the first one explored a set of hand gestures for transitioning between two environments that are commonly used for “mode switching” within realities. In the second study, we compared FingerButton with the best two-hand gesture identified in the first study and other commercially available solutions (e.g., double tap) for a between-reality selection task. The results show that the physical finger button is faster and user-preferred than other techniques for the transition tasks. Overall, this research contributes to understanding and improving the interaction techniques for fluid switching between the real and virtual worlds, thereby enhancing VR user experiences.

The small display size of the smartwatches makes it difficult to display large amounts of information on the device. Prior work explored leveraging a second device (e.g., Head-mounted displays) to extend the space where users can access large information space with virtual displays anchored on their wrists. Though researchers showed that having an additional virtual screen increased information bandwidth, little is known about the effect of virtual display sizes on users’ performance. In this paper, we examined the impact of display sizes on spatial memory, workload, and user experience to better understand the prospects of virtually-augmented displays for smartwatches. Results from a user study revealed that a 4.8 inches display size can be the “sweet spot” for the virtually-augmented displays to ensure improved spatial memory performance and better user experience with less workload. Finally, we provided a set of design guidelines focusing to display size, spatial memory, user experience, and workload for designing virtually augmented user interfaces for smartwatches.

Conditionally automated vehicles provide unique opportunities for drivers to engage in non-driving-related tasks (NDRTs); however, drivers must remain prepared to respond to take-over requests. This paper explores design challenges and potential solutions for supporting reading as an NDRT in SAE Level 3 vehicles. Specifically, we assess two prominent text-presentation techniques: vertical scrolling text presentation (VSTP) and rapid serial visual presentation (RSVP), exploring both in conjunction with their integration with auditory speech displays (ASD). A driving simulation study involving N = 32 participants revealed that RSVP surpassed VSTP in regaining situational awareness, as indicated by lower average braking actuation, and was also preferred by participants. The integration of ASDs with both techniques reduced perceived cognitive workload and improved the user experience, albeit with compromised lateral control. Our findings can help advance the design of human-centered interfaces for reading in conditionally automated vehicles.

Large curved displays have emerged as a powerful platform for collaboration, data visualization, and entertainment. These displays provide highly immersive experiences, a wider field of view, and higher satisfaction levels. Yet, large curved displays are not commonly available due to their high costs. With the recent advancement of Head Mounted Displays (HMDs), large curved displays can be simulated in Virtual Reality (VR) with minimal cost and space requirements. However, to consider the virtual display as an alternative to the physical display, it is necessary to uncover user performance differences (e.g., pointing speed and accuracy) between these two platforms. In this paper, we explored users’ pointing performance on both physical and virtual large curved displays. Specifically, with two studies, we investigate users’ performance between the two platforms for standard pointing factors such as target width, target amplitude as well as users’ position relative to the screen. Results from user studies reveal no significant difference in pointing performance between the two platforms when users are located at the same position relative to the screen. In addition, we observe users’ pointing performance improves when they are located at the center of a semi-circular display compared to off-centered positions. We conclude by outlining design implications for pointing on large curved virtual displays. These findings show that large curved virtual displays are a viable alternative to physical displays for pointing tasks.

User interactions with smartwatches are limited due to one-finger usage on small touch screens. However, smartwatch interaction capabilities can be extended beyond the screen into the around-device space by leveraging multi-finger interactions. In this paper, we explore suitable finger postures and gestures in mid-air and on the back of the palm to extend interaction capabilities on the unmodified commodity smartwatch while ensuring their learnability through discoverability. We conduct a design study to find a set of finger postures and gestures suitable for interacting with off-the-shelf smartwatches. An application is then designed to detect the posture-gesture set with on-device dual cameras while examining their classification accuracy. To facilitate learnability through discoverability of the gesture-posture set, we explore ‘context-aware-hints’ that shows a sub-set of gesture-posture and their associated actions based on the usage context. Results from a user study show that the guidance helps users to discover and learn them quickly and accurately.

Virtual Reality (VR) headsets equipped with multiple cameras enable hands-only teleportation techniques without requiring any physical controller. Hands-only teleportation is an effective alternative to controllers for navigation tasks in virtual reality - allowing users to move from one point to another instantaneously. However, the current implementation of hands-only techniques does not consider users' physical attributes (e.g., arm's reach). Thus, a hands-only teleportation technique can lead to different user experiences based on physical attributes. We propose PAWS, a personalized arm and wrist-based teleportation technique that incorporates users' physical attributes for improved teleportation experiences. We first evaluate different degrees of teleportation personalization with no-, partial, and full personalization. We find that full personalization offers faster locomotion - but at the cost of degraded performances with distant targets due to increased sensitivity. We hence further explore different combinations of mapping functions (e.g., sigmoid, quadratic) to personalize motor movements and find that asymmetric functions result in improved performance. Overall, our results show that PAWS helps users to navigate quickly in virtual environments.

Large curved displays inside Virtual Reality environments are becoming popular for visualizing high-resolution content during analytical tasks, gaming or entertainment. Prior research showed that such displays provide a wide field of view and offer users a high level of immersion. However, little is known about users' performance (e.g., pointing speed and accuracy) on them. We explore users' pointing performance on large virtual curved displays. We investigate standard pointing factors (e.g., target width and amplitude) in combination with relevant curve-related factors, namely display curvature and both linear and angular measures. Our results show that the less curved the display, the higher the performance, i.e., faster movement time. This result holds for pointing tasks controlled via their visual properties (linear widths and amplitudes) or their motor properties (angular widths and amplitudes). Additionally, display curvatures significantly affect the error rate for both linear and angular conditions. Furthermore, we observe that curved displays perform better or similar to flat displays based on throughput analysis. Finally, we discuss our results and provide suggestions regarding pointing tasks on large curved displays in VR.

Teleportation, a widely used locomotion technique in Virtual Reality (VR), is used to move users through a virtual environment. Until recently, handheld controllers have been used for teleportation, where users use controllers to point to a location and perform an action (e.g., button press) to be instantly moved to the targeted location. Recent advancements in VR hand tracking enable users to move through and interact with the virtual world without controllers. This opens the opportunity for compelling alternatives to explore hand tracking-based teleportation techniques for more natural, intuitive and immersive interactions. Prior work mostly explores using arm movement for teleportation as an alternative to using the controller. In this paper, we design and evaluate WriArm, a VR locomotion technique that leverages both wrist and arm movement for VR teleportation. We first conduct a design study to find suitable hand gesture sets that can be mapped to teleportation activities such as activation, pointing, confirmation and cancellation for WriArm and arm-based techniques. Based on the results, we conduct a study comparing users’ performance while navigating tasks with the two techniques and three gesture sets. Results show that WriArm improves navigation efficiency by allowing users to navigate the environment quickly. We conclude with design guidelines for arm and wrist-based teleportation in VR.

Silent speech input converts non-acoustic features like tongue and lip movements into text. It has been demonstrated as a promising input method on mobile devices and has been explored for a variety of audiences and contexts where the acoustic signal is unavailable (e.g., people with speech disorders) or unreliable (e.g., noisy environment). Though the method shows promise, very little is known about peoples’ perceptions regarding using it. In this work, first, we conduct two user studies to explore users’ attitudes towards the method with a particular focus on social acceptance and error tolerance. Results show that people perceive silent speech as more socially acceptable than speech input and are willing to tolerate more errors with it to uphold privacy and security. We then conduct a third study to identify a suitable method for providing real-time feedback on silent speech input. Results show users find an abstract feedback method effective and significantly more private and secure than a commonly used video feedback method.

Advancements in eye-tracking technology has compelled researchers to explore potential eye-based interactions with diverse devices. Though many commercial devices are now equipped with eye-tracking solutions (e.g., HTC VIVE Pro), little is known about users social acceptance and preference of eye-based interaction techniques, especially with smartphones. We report on three studies to explore users’ social acceptance and preferences regarding different head- and eye-based inputs with smartphones. Study results show that eye movements are more socially acceptable than other head- and eye-based techniques due to its subtle nature. Based on these findings, we further examine users preferences regarding saccade and pursuit eye movements. Results reveal users’ preference for saccade compared to pursuit eye movements. In a third study exploring delimiting actions to discriminate between intentional and unintentional eye-inputs, Dwell is shown as the preferred delimiter, both in public and private spaces. We conclude with design guidelines for eye-based interactions on smartphones.

Text entry is a common activity in virtual reality (VR) systems. There is a limited number of available hands-free techniques, which allow users to carry out text entry when users’ hands are busy such as holding items or hand-based devices are not available. The most used hands-free text entry technique is DwellType, where a user selects a letter by dwelling over it for a specific period. However, its performance is limited due to the fixed dwell time for each character selection. In this paper, we explore two other hands-free text entry mechanisms in VR: BlinkType and NeckType, which leverage users’ eye blinks and neck’s forward and backward movements to select letters. With a user study, we compare the performance of the two techniques with DwellType. Results show that users can achieve an average text entry rate of 13.47, 11.18 and 11.65 words per minute with BlinkType, NeckType, and DwellType, respectively. Users’ subjective feedback shows BlinkType as the preferred technique for text entry in VR.

Intensive exploration and navigation of hierarchical lists on smartphones can be tedious and time-consuming as it often requires users to frequently switch between multiple views. To overcome this limitation, we present PinchList, a novel interaction design that leverages pinch gestures to support seamless exploration of multi-level list items in hierarchical views. With PinchList, sub-lists are accessed with a pinch-out gesture whereas a pinch-in gesture navigates back to the previous level. Additionally, pinch and flick gestures are used to navigate lists consisting of more than two levels. We conduct a user study to refine the design parameters of PinchList such as a suitable item size, and quantitatively evaluate the target acquisition performance using pinch-in/out gestures in both scrolling and non-scrolling conditions. In a second study, we compare the performance of PinchList in a hierarchal navigation task with two commonly used touch interfaces for list browsing: pagination and expand-and-collapse interfaces. The results reveal that PinchList is significantly faster than other two interfaces in accessing items located in hierarchical list views. Finally, we demonstrate that PinchList enables a host of novel applications in list-based interaction?

The information on a non-prescription (over-the-counter) medicine label helps patients to make informed decisions when purchasing a medicine. Since non-prescription medicines can be purchased without consulting a healthcare professional, there is a growing concern that people do not put enough emphasis on reading medicine label information, resulting in drug misuse with possible health consequences. In this paper, we investigate patients' use of medicine labels with the goal of developing a smartphone application to guide them toward reading the important information (e.g., warnings, dosage) on the labels. We first conducted two studies examining (i) users' rating on the information that they commonly read and (ii) healthcare professionals' rating on the information that patients should read before purchasing a non-prescription medicine. Our results revealed that patients put less emphasis on reading many information such as dosage, warnings and precautions, that the healthcare professions highly recommend the patients to read. Inspired by the findings, we designed a smartphone application to make users aware of the important information on non-prescription medicines. Along the way, we conduct a study examining different information presentation techniques to show medicine labels on smartphones, where our results show that icons and texts are more accurate and preferred techniques by users. In a further study where users explore medicines with our smartphone application, we observed a significant increase (mean 27%) in the general users' rating on the information categories that were recommended by healthcare professionals. This suggests that the users were guided to read important information by the smartphone application.

The use of crowdsourcing platforms for data collection in HCI research is attractive in their ability to provide rapid access to large and diverse participant samples. As a result, several researchers have conducted studies investigating the similarities and differences between data collected through crowdsourcing and more traditional, laboratory-style data collection. We add to this body of research by examining the feasibility of conducting social acceptability studies via crowdsourcing. Social acceptability can be a key determinant for the early adoption of emerging technologies, and as such, we focus our investigation on social acceptability for Head-Worn Display (HWD) input modalities. Our results indicate that data collected via a crowdsourced experiment and a laboratory-style setting did not differ at a statistically significant level. These results provide initial support for crowdsourcing platforms as viable options for conducting social acceptability research.

Selecting an item of interest on smartwatches can be tedious and time-consuming as it involves a series of swipe and tap actions. We present PageFlip, a novel method that combines into a single action multiple touch operations such as command invocation and value selection for efficient interaction on smartwatches. PageFlip operates with a page flip gesture that starts by dragging the UI from a corner of the device. We first design PageFlip by examining its key design factors such as corners, drag directions and drag distances. We next compare PageFlip to a functionally equivalent radial menu and a standard swipe and tap method. Results reveal that PageFlip improves efficiency for both discrete and continuous selection tasks. Finally, we demonstrate novel smartwatch interaction opportunities and a set of applications that can benefit from PageFlip.

Many smartwatches have a digital crown as a complementary input modality to the touchscreen. While the crown eliminates issues of touchscreen occlusion and imprecision, i.e., the 'fat-finger' problem, the crown input is limited as it only supports bi-directional rotations along a single dimension. We present TiltCrown, a crown prototype that increases a smartwatch's input bandwidth through an isometric joystick. TiltCrown supports angular tilt, rotations and button press events for touchless smartwatch interaction. In a user study we explore how accurate and how fast users can perform item selection tasks using TiltCrown.

The popularity of head-worn displays (HWD) technologies such as Virtual Reality (VR) and Augmented Reality (AR) headsets is growing rapidly. To predict their commercial success, it is essential to understand the acceptability of these new technologies, along with new methods to interact with them. In this vein, the evaluation of social acceptability of interactions with these technologies has received significant attention, particularly from the performer's (i.e., user's) viewpoint. However, little work has considered social acceptability concerns from observers' (i.e., spectators') perspective. Although HWDs are designed to be personal devices, interacting with their interfaces are often quite noticeable, making them an ideal platform to contrast performer and observer perspectives on social acceptability. Through two studies, this paper contrasts performers' and observers' perspectives of social acceptability interactions with HWDs under different social contexts. Results indicate similarities as well as differences, in acceptability, and advocate for the importance of including both perspectives when exploring social acceptability of emerging technologies. We provide guidelines for understanding social acceptability specifically from the observers' perspective, thus complementing our current practices used for understanding the acceptability of interacting with these devices.

In recent years, around device input has emerged as a complement to standard touch input, albeit in limited tasks and contexts, such as for item selection or map navigation. We push the boundaries for around device interactions to facilitate an entire smartphone application: browsing through large information lists to make a decision. To this end, we present AirPanes, a novel technique that allows two-handed in-air interactions, conjointly with touch input to perform analytic tasks, such as making a purchase decision. AirPanes resolves the inefficiencies of having to switch between multiple views or panes in common smartphone applications. We explore the design factors that make AirPanes efficient. In a controlled study, we find that AirPanes is on average 50% more efficient that standard touch input for an analytic task. We offer recommendations for implementing AirPanes in a broad range of applications.

We present SoundCraft, a smartwatch prototype embedded with a microphone array, that localizes angularly, in azimuth and elevation, acoustic signatures: non-vocal acoustics that are produced using our hands. Acoustic signatures are common in our daily lives, such as when snapping or rubbing our fingers, tapping on objects or even when using an auxiliary object to generate the sound. We demonstrate that we can capture and leverage the spatial location of such naturally occurring acoustics using our prototype. We describe our algorithm, which we adopt from the MUltiple SIgnal Classification (MUSIC) technique [31], that enables robust localization and classification of the acoustics when the microphones are required to be placed at close proximity. SoundCraft enables a rich set of spatial interaction techniques, including quick access to smartwatch content, rapid command invocation, in-situ sketching, and also multi-user around device interaction. Via a series of user studies, we validate SoundCraft's localization and classification capabilities in non-noisy and noisy environments.

Motivated by a rise in the variety and number of mobile devices that users carry, we investigate scenarios when operating these devices in a spatially interlinked manner can lead to interfaces that generate new advantages. Our exploration is focused on the design of SAMMI, a spatially-aware multi-device interface to assist with analytic map navigation tasks, where, in addition to browsing the workspace, the user has to make a decision based on the content embedded in the map. We focus primarily on the design space for spatially interlinking a smartphone with a smartwatch. As both smart devices are spatially tracked, the user can browse information by moving either device in the workspace. We identify several design factors for SAMMI and through a first study we explore how best to combine these for efficient map navigation. In a second study we compare SAMMI to the common Flick-&-Pinch gestures for an analytic map navigation task. Our results reveal that SAMMI is an efficient spatial navigation interface, and by means of an additional spatially tracked display, can facilitate quick information retrieval and comparisons. We finally demonstrate other potential use cases for SAMMI that extend beyond map navigation to facilitate interaction with spatial workspaces.

Several research groups have demonstrated advantages of extending a mobile device's input vocabulary with in-air gestures. Such gestures show promise but are not yet being integrated onto commercial devices. One reason for this might be the uncertainty about users' perceptions regarding the social acceptance of such around-device gestures. In three studies, performed in public settings, we explore users' and spectators' attitudes about using around-device gestures in public. The results show that people are concerned about others' reactions. They are also sensitive and selective regarding where and in front of whom they would feel comfortable using around-device gestures. However, acceptance and comfort are strongly linked to gesture characteristics, such as, gesture size, duration and in-air position. Based on our findings we present recommendations for around-device input designers and suggest new approaches for evaluating the social acceptability of novel input methods.

Exploring information content on mobile devices can be tedious and time consuming. We present Around-Device Binning, or AD-Binning, a novel mobile user interface that allows users to off-load mobile content in the space around the device. We informed our implementation of AD-Binning by exploring various design factors, such as the minimum around-device target size, suitable item selection methods, and techniques for placing content in off-screen space. In a task requiring exploration, we find that AD-Binning improves browsing efficiency by avoiding the minute selection and flicking mechanisms needed for on-screen interaction. We conclude with design guidelines for off screen content storage and browsing.

Mobile devices are endowed with significant sensing capabilities. However, their ability to 'see' their surroundings, during active use, is limited. We present Surround-See, a self-contained smartphone equipped with an omni-directional camera that enables peripheral vision around the device to augment daily mobile tasks. Surround-See provides mobile devices with a field-of-view collinear to the device screen. This capability facilitates novel mobile tasks such as, pointing at objects in the environment to interact with content, operating the mobile device at a physical distance and allowing the device to detect user activity, even when the user is not holding it. We describe Surround-See's architecture, and demonstrate applications that exploit peripheral 'seeing' capabilities during active use of a mobile device. Users confirm the value of embedding peripheral vision capabilities on mobile devices and offer insights for novel usage methods.

The human hand can naturally coordinate multiple finger joints, and simultaneously tilt, press and roll a pen to write or draw. For this reason, digital pens are now embedded with auxiliary input sensors to capture these actions. Prior research on auxiliary input channels has mainly investigated them in isolation of one another. In this work, we explore the coordinated use of two auxiliary channels, a class of interaction techniques we refer to as a-coord input. Through two separate experiments, we explore the design space of a-coord input. In the first study we identify if users can successfully coordinate two auxiliary channels. We found a strong degree of coordination between channels. In a second experiment, we evaluate the effectiveness of a-coord input in a task with multiple steps, such as multi-parameter selection and manipulation. We find that a-coord input facilitates coordination even with a complex, aforethought sequential task. Overall our results indicate that users can control at least two auxiliary input channels in conjunction which can facilitate a number of common tasks can on the pen.

Observational studies indicate that most people use one hand to interact with their mobile devices. Interaction on the back-of-devices (BoD) has been proposed to enhance one-handed input for various tasks, including selection and gesturing. However, we do not possess a good understanding of some fundamental issues related to one-handed BoD input. In this paper, we attempt to fill this gap by conducting three studies. The first study explores suitable selection techniques; the second study investigates the performance and suitability of the two main modes of cursor movement: Relative and Absolute; and the last study examines solutions to the problem of reaching the lower part of the device. Our results indicate that for BoD interaction, relative input is more efficient and accurate for cursor positioning and target selection than absolute input. Based on these findings provide guidelines for designing BoD interactions for mobile devices.

Devices with small viewports (e.g., smartphones or GPS) result in interfaces where objects of interest can easily reside outside the view, into off-screen space. Researchers have addressed this challenge and have proposed visual cues to assist users in perceptually locating off-screen objects. However, little attention has been placed on methods for selecting the objects. Current designs of off-screen cues can result in overlaps that can make it difficult to use the cues as handles through which users can select the off-screen objects they represent. In this paper, we present EdgeSplit, a technique that facilitates both the visualization and selection of off-screen objects on small devices. EdgeSplit exploits the space around the device's borders to display proxies of off-screen objects and then partitions the border regions to allow for non-overlapping areas that make selection of objects easier. We present an effective algorithm that provides such partitioning and demonstrate the effectiveness of EdgeSplit for selecting off-screen objects.

Numerous applications such as simulations, air traffic control systems, and video surveillance systems are inherently composed of spatial objects that move in a scene. In many instances, users can benefit from tools that allow them to select these targets in real-time, without having to pause the dynamic display. However, selecting moving objects is considerably more difficult and error prone than selecting stationary targets. In this paper, we evaluate the effectiveness of several techniques that assist in selecting moving targets. We present Comet, a technique that enhances targets based on their speed and direction. We also introduce Target Ghost, which allows users to select a static proxy of the target, while leaving the motion uninterrupted. We found a speed benefit for the Comet in a 1D selection task in comparison to other cursor and target enhancements. For 2D selection, Comet outperformed Bubble cursor but only when Target Ghost was not available. We conclude with guidelines for design.