Traditional methods of viewing medical images (e.g., the PACS system), typically require surgeons to interact with a computer through physical devices such as keyboards and mice, which may violate the sterile principles of the operating room. To address this issue, this study introduces an augmented reality (AR) technology paired with a user-defined gesture set that aligns with surgeons' mental models, allowing surgeons to access and manipulate the PACS system on AR glasses while maintaining sterile conditions. This study represents a beneficial attempt at participatory design for professional users (surgeons) in a constrained space (sterile interaction zone), offering significant theoretical implications and practical value for sterile interaction of other smart medical devices and future Spatial Computing applications.

In the era of Big Data, augmented reality (AR)-based 3D visualization technology is gradually becoming an essential tool of effective information dissemination. However, 3D data visualization inevitably faces a series of challenges, particularly when visualizing large datasets within limited physical spaces. This often results in high spatial density, characterized by smaller target objects and occlusion of data objects, which greatly complicate object selection and subsequent interaction. To address these issues, this paper proposes a multimodal progressive target selection technique-StarPicker, which integrates the SpotLight metaphor and Clock metaphor. By leveraging wrist rotation and gesture interactions, StarPicker facilitates a coarse-to-fine semantic disambiguation and precise selection of target objects. To validate the effectiveness of StarPicker, a comparative user study was conducted against the state-of-the-art techniques (GridWall and FlowerCone) under high-density (up to 360 objects) and small-target (1cm) conditions. Experimental results demonstrate that StarPicker significantly outperforms the baselines in terms of target selection accuracy and user satisfaction, while achieving comparable or better completion times, especially in denser scenes. This work offers a novel approach to target selection and interaction technology in the field of AR-based 3D Big Data visualization.

Dragging is a fundamental interaction technique in human-computer interaction. Prior research on dragging has generally been concerned with 2D objects on 2D interactive surfaces, with limited attention to 3D interactive spaces. We performed three experiments to systematically investigate dragging using raycasting in virtual reality (VR), a representative 3D interactive environment. The first experiment examined dragging 2D targets on 2D surfaces placed in VR. The second experiment investigated dragging 3D targets positioned at identical depths in VR. Both experiments examined the effects of target depth, angular target amplitude, and angular target on dragging performance. The third experiment investigated dragging 3D targets with varying depths, focusing on the effects of target layout, target amplitude, and target width. Findings revealed the different impacts of the above factors on dragging interaction. We also proposed a new 3D Fitts' law model tailored for dragging using raycasting in VR, which fits well with our data. Finally, we discuss our results and provide design recommendations for dragging (i.e., translational positioning) tasks using raycasting under the tested experimental conditions in VR.

Recognizing the potential of freehand gestures for interacting with virtual objects in virtual environments, our research introduces a user-defined freehand gesture set of typical referents in virtual reality (VR), focusing on a specific scenario: VR museums. We conducted a comprehensive elicitation study with two experiments to define and refine the gesture set. Meanwhile, we demonstrated an enhanced real-time Wizard of Oz approach that facilitated users’ understanding of referents in VR and their gesture design. Our findings revealed significant improvements in gesture consistency and user agreement through two experiments, with an average agreement score of firstchoice and second-choice advancing from 0.211 and 0.160 to 0.412 and 0.284, respectively. By offering a consistent user-centered gesture set, this work contributes to guiding museum curators toward creating more immersive user experiences in VR museums. The gestures can also be extended to other VR applications that necessitate travel, selection and manipulation, and system control tasks.

Target selection is one of essential operation made available by interaction techniques in virtual reality (VR) environments. However, effectively positioning or selecting occluded objects is under-investigated in VR, especially in the context of high-density or a high-dimensional data visualization with VR. In this paper, we propose ClockRay, an occluded-object selection technique that can maximize the intrinsic human wrist rotation skills through the integration of emerging ray selection techniques in VR environments. We describe the design space of the ClockRay technique and then evaluate its performance in a series of user studies. Drawing on the experimental results, we discuss the benefits of ClockRay compared to two popular ray selection techniques - RayCursor and RayCasting. Our findings can inform the design of VR-based interactive visualization systems for high-density data.

With the rapid development of natural human-computer interaction technologies, gesture-based interfaces have become popular. Although gesture interaction has received extensive attention from both academia and industry, most existing studies focus on hand gesture input, leaving foot-gesture-based interfaces underexplored, especially in scenarios where the user’s hands are occupied for other interaction tasks such as washing the hair in smart shower rooms. In such scenarios, users often have to perform interactive tasks (e.g., controlling water volume) with their eyes closed when water and shampoo liquid flow along with their head to eyes area. One possible way to address this problem is to use eyes-free (rather than eyes-engaged), foot-gesture-based interactive techniques that allow users to interact with the smart shower system without visual involvement. Through our online survey, 71.60% of the participants (58/81) have the requirements of using foot-gesture-based eyes-free interactions during showers. To this end, we conducted a three-phase study to explore foot-gesture-based interaction to achieve eyes-free interaction in smart shower rooms. We first derived a set of user-defined foot gestures for eyes-free interaction in smart shower rooms. Then, we proposed a taxonomy for foot gesture interaction. Our findings indicated that end-users preferred single-foot (76.1%), atomic (73.3%), deictic (65.0%), and dynamic (76.1%) foot gestures, which markedly differs from the results reported by previous studies on user-defined hand gestures. In addition, most of the user-defined dynamic foot gestures involve atomic movements perpendicular to the ground (40.1%) or parallel to the ground (27.7%). We finally distilled a set of concrete guidelines for foot gesture interfaces based on observing end-users’ mental model and behaviors when interacting with foot gestures. Our research can inform the design and development of foot-gesture-based interaction techniques for applications such as smart homes, intelligent vehicles, VR games, and accessibility design.

In the design of gesture-based user interfaces, traditional gesture elicitation studies suffer from the legacy bias problem. In this paper, we conducted an exploratory study about the practical effects of frame-based design for gestural interaction with immersive VR shopping applications. In this study, we derived gestures via the traditional guessability and the framed guessability approaches. Experimental results indicated that priming participants with a frame, or a scenario, could significantly reduce the impact of legacy bias, and result in superior gesture vocabulary. However, no evidence was found that the priming technique would generate more gesture types, which may lead to lower agreement scores due to the reduction of legacy bias. Based on our findings, we propose some concrete design guidelines for gesture-based interaction. We highlight the implications of this work for the design of all gesture-based applications.

In recent years, virtual reality (VR) technologies have been applied to the field of journalism, where the concept of immersive VR news has been proposed. However, despite the fanfare, strong response, and sensational effect caused by its advent, immersive VR news remains a novel journalism paradigm that faces new challenges in its production process. Currently, there is a lack of a unified design framework, and, since most studies in this area have focused on non-interactive VR news, the understanding of the effects of more interactive VR technologies on the news consumer remains inadequate. In this study, we propose a more practical design framework for immersive VR news products. Following this framework, we designed a VR news application and conducted user evaluation in terms of media effects and user experience. Based on the experimental findings, which demonstrated that non-interactive VR news products resulted in a distracting user experience and less immersion, while interactive VR news offered improved media effects and user experience, we then derived concrete design guidelines for immersive VR news design. Finally, we highlight that this study provides a theoretical and practical reference framework for the further study of VR news.

Existing mobile phones use the front screen for gesture interactions, but they often suffer from usability problems such as content occlusion and spurious triggering caused by fat finger errors. Recent advances in new sensor technologies have made it possible for modern mobile phones to recognize gesture inputs on the rear screen. Although inputs from the backside of a mobile phone have demonstrated their usefulness for simple interaction tasks such as target acquisition, little is known regarding the best practices in gesture design for complex concurrent tasks on dual-screen mobile devices. Therefore, we conducted a three-stage study to explore user-defined multi-finger gestures for concurrent game tasks on a dual-screen mobile device. Experimental results indicate the existence of a consensus between participants regarding gesture choices for specified game tasks. Based on this consensus, we developed a gesture taxonomy and generated a set of user-defined gestures for dual-screen mobile devices. A subsequent benchmark test validated the popularity of the user-defined gestures with both ordinary and professional game players.

Gesture elicitation studies, which are a popular technology for collecting requirements and expectations by involving real users in gesture design processes, often suffer from gesture disagreement and legacy bias and may not generate optimal gestures for a target system in practice. This paper reports a research project on user-defined gestures for interacting with immersive VR shopping applications. The main contribution of this work is the proposal of a more practical method for deriving more reliable gestures than traditional gesture elicitation studies. We applied this method to a VR shopping application and obtained empirical evidence for the benefits of deriving two gestures in the a priori stage and selecting the top-two gestures in the a posteriori stage of traditional elicitation studies for each referent. We hope that this research can help lay a theoretical foundation for freehand-gesture-based user interface design and be generalised to all freehand-gesture-based applications.

Accurately understanding a user’s intention is often essential to the success of any interactive system. An information retrieval system, for example, should address the vocabulary problem (Furnas et al., 1987) to accommodate different query terms users may choose. A system that supports natural user interaction (e.g., full-body game and immersive virtual reality) must recognize gestures that are chosen by users for an action. This article reports an experimental study on the gesture choice for tasks in three application domains. We found that the chance for users to produce the same gesture for a given task is below 0.355 on average, and offering a set of gesture candidates can improve the agreement score. We discuss the characteristics of those tasks that exhibit the gesture disagreement problem and those tasks that do not. Based on our findings, we propose some design guidelines for free-hand gesture-based interfaces.