Gesture interaction is standard in modern extended reality head-mounted displays, e.g., Microsoft HoloLens 2 and Apple Vision Pro. These gestures come from multiple sources, such as elicitation studies, the application designer, or an expert. However, it is unclear if the source of the gesture affects the memorability of gestures (i.e., level of recall). In a user study, 35 participants used the HoloLens 2 to test three gestures set from different sources: 1) expert-defined, whereby we asked four experts to define gestures; 2) user-defined, whereby each participant defined their gestures; and 3) elicitation-defined, whereby we used gestures proposed by previous work. Our results showed that expert-defined gestures (72.73% recall) were more likely to be remembered than elicitation-defined (59.87% recall) gestures. Further, user-defined gestures (88.75% recall) were the most memorable. These data suggest that the source of gestures has a substantial impact on their memorability.

As augmented reality (AR) technology and hardware become more mature and affordable, researchers have been exploring more intuitive and discoverable interaction techniques for immersive environments. This paper investigates multimodal interaction for 3D object manipulation in a multi-object AR environment. To identify the user-defined gestures, we conducted an elicitation study involving 24 participants and 22 referents using an augmented reality headset. It yielded 528 proposals and generated a winning gesture set with 25 gestures after binning and ranking all gesture proposals. We found that for the same task, the same gesture was preferred for both one and two-object manipulation, although both hands were used in the two-object scenario. We present the gestures and speech results, and the differences compared to similar studies in a single object AR environment. The study also explored the association between speech expressions and gesture stroke during object manipulation, which could improve the recognizer efficiency in augmented reality headsets.

Elicitation studies have become a popular method of participatory design. While traditionally used to examine unimodal gesture interactions, elicitation has started being used with other novel interaction modalities. Unfortunately, there has been no work that examines the impact of referent display on elicited interaction proposals. To address that concern this work provides a detailed comparison between two elicitation studies that were similar in design apart from the way that participants were prompted for interaction proposals (i.e., the referents). Based on this comparison the impact of referent display on speech and gesture interaction proposals are each discussed. The interaction proposals between these elicitation studies were not identical. Gesture proposals were the least impacted by referent display, showing high proposal similarity between the two works. Speech proposals were highly biased by text referents with proposals directly mirroring text-based referents an average of 69.36% of the time. In short, the way that referents are presented during elicitation studies can impact the resulting interaction proposals; however, the level of impact found is dependent on the modality of input elicited.

• All time information was reported as milliseconds but given as 100ths of a second.• Additionally, the time differences between gesture initiation and speech initiation were reported as being 60% of their actual value. This resulted in those time differences being reported as 6 times less than they were (e.g., 130ms should be 780ms).

This study explored interactions between users and a 3D virtual world using a multi-touch display and sought to identify the advantages of this form of interaction. An experiment was conducted with 28 subjects, who were classified into two groups: experienced gamers and casual gamers. The experiment involved a pseudo-universe, where users were required to find five objects using the multi-touch display, and five objects using a game controller (GamePad). Overall, multi-touch was slightly faster than the GamePad in the search for objects. Experienced users performed significantly better when using the GamePad, compared to casual gamers. When casual gamers were analyzed separately, their performance was better (at a statistically significant level) with the multi-touch display, compared to the GamePad. Finally, users had a significantly higher error rate when switching from the GamePad to the keyboard, than when switching from the multi-touch to the keyboard.

This research establishes a better understanding of the syntax choices in speech interactions and of how speech, gesture, and multimodal gesture and speech interactions are produced by users in unconstrained object manipulation environments using augmented reality. The work presents a multimodal elicitation study conducted with 24 participants. The canonical referents for translation, rotation, and scale were used along with some abstract referents (create, destroy, and select). In this study time windows for gesture and speech multimodal interactions are developed using the start and stop times of gestures and speech as well as the stoke times for gestures. While gestures commonly precede speech by 81 ms we find that the stroke of the gesture is commonly within 10 ms of the start of speech. Indicating that the information content of a gesture and its co-occurring speech are well aligned to each other. Lastly, the trends across the most common proposals for each modality are examined. Showing that the disagreement between proposals is often caused by a variation of hand posture or syntax. Allowing us to present aliasing recommendations to increase the percentage of users' natural interactions captured by future multimodal interactive systems.

The primary objective of this research is to understand how users manipulate virtual objects in augmented reality using multimodal interaction (gesture and speech) and unimodal interaction (gesture). Through this understanding, natural-feeling interactions can be designed for this technology. These findings are derived from an elicitation study employing Wizard of Oz design aimed at developing user-defined multimodal interaction sets for building tasks in 3D environments using optical see-through augmented reality headsets. The modalities tested are gesture and speech combined, gesture only, and speech only. The study was conducted with 24 participants. The canonical referents for translation, rotation, and scale were used along with some abstract referents (create, destroy, and select). A consensus set of gestures for interactions is provided. Findings include the types of gestures performed, the timing between co-occurring gestures and speech (130 milliseconds), perceived workload by modality (using NASA TLX), and design guidelines arising from this study. Multimodal interaction, in particular gesture and speech interactions for augmented reality headsets, are essential as this technology becomes the future of interactive computing. It is possible that in the near future, augmented reality glasses will become pervasive.

We present a whole-body gesture elicitation study using Head Mounted Displays, including a legacy bias reduction technique. The motivation for this study was to understand the type of gesture agreement rates for selection and manipulation interactions and to improve the user experience for whole-body interactions. We found that regardless of the production technique used to remove legacy bias, legacy bias was still found in some of the produced gestures.

We present a whole-body gesture elicitation study using Head Mounted Displays, including a legacy bias reduction. The motivation for this study was to understand the type of gesture agreement rates for selection and manipulation interactions and to improve the user experience for whole-body interactions. We looked at 23 participants and 20 distinct referents (with multiple gestures per referent). We found that regardless of the production technique used to remove legacy bias, legacy bias was still found in some of the produced gestures. In some instances, gestures were derived from previous interactions but were still appropriate for the environment presented. This study provides a rich set of information and useful recommendations for future designers and/or developers.

CubeVR incorporates the traditional studio-based learning in architecture with virtual reality (VR) and in doing so it enables features that are not available or difficult to achieve in the real-world. CubeVR allows students and instructors to move from the existing studio to a virtual environment, allowing for a more impactful learning experience. This paper presents our current state of the CubeVR prototype, informal impressions of users, and the use of different affordances in the virtual world.

Understanding gaming expertise is important in user studies. We present a study comprised of 60 participants playing a First Person Shooter Game (Counter-Strike: Global Offensive). This study provides results related to a keyboard model used to determine an objective measurement of gamers' skill. We also show that there is no correlation between frequency questionnaires and user skill.