Numerous studies have investigated the effects of system fidelity as a whole on one’s total sense of presence in virtual reality (VR). The Fidelity-based Presence Scale (FPS), a recently introduced presence questionnaire, provides a method for investigating the effects of different system fidelities (interaction, scenario, and display) on different aspects of one’s sense of presence. In this paper, we present one of the first studies to investigate those effects for a locomotion task by conducting a 2 × 2 × 2 within-subjects experiment that reveals insight on how the components of system fidelity affect sense of presence. Like recent research, our results indicate that interaction fidelity and display fidelity significantly affect one’s interaction presence and display presence, respectively. However, unlike prior work, we did not find that changes in scenario fidelity significantly affected one’s scenario presence. We discuss other results and the possible implications of this research.

Personal computers and handheld devices provide keyboard shortcuts and swipe gestures to enable users to efficiently switch between applications, whereas today’s virtual reality (VR) systems do not. In this work, we present an exploratory study on user interface aspects to support efficient switching between worlds in VR. We created eight interfaces that afford previewing and selecting from the available virtual worlds, including methods using portals and worlds-in-miniature (WiMs). To evaluate these methods, we conducted a controlled within-subjects empirical experiment (N=22) where participants frequently transitioned between six different environments to complete an object collection task. Our quantitative and qualitative results show that WiMs supported rapid acquisition of high-level spatial information while searching and were deemed most efficient by participants while portals provided fast pre-orientation. Finally, we present insights into the applicability, usability, and effectiveness of the VR world switching methods we explored, and provide recommendations for their application and future context/world switching techniques and interfaces.

As the field of intelligent interfaces is evolving, there is also a growing need for feedback mechanisms that are both expressive for participants and also contain reliable and useful information for the researcher conducting the study for survey data collection. Our study involves exploring two different kinds of survey methods: a standardized slider scale for web-based surveys and a free-form text input with a Large Language Model (LLM) acting as a backbone. The experiment includes 36 participants completing a 4×4 sliding-tile game at two different levels (easy and hard) with difficulty standardized via Manhattan-distance targets. The response mode order was counterbalanced across two sequences. This task aimed to evaluate the accuracy and quality of participant responses through different survey methodologies. Our key findings are that the LLM survey results are equivalent to the ones reported through the Web-Based slider scale questionnaire method. Our contribution is an intelligent framework that allows text-based reflections within an adaptive survey interface, helping both participants to express their experiences naturally and also researchers to gain valuable information about their system.

Within the virtual reality (VR) research community, there have been several efforts to develop questionnaires with the aim of better understanding the sense of presence. Despite having numerous surveys, the community does not have a questionnaire that informs which components of a VR application contributed to the sense of presence. Furthermore, previous literature notes the absence of consensus on which questionnaire or questions should be used. Therefore, we conducted a Delphi study, engaging presence experts to establish a consensus on the most important presence questions and their respective verbiage. We then conducted a validation study with an exploratory factor analysis (EFA). The efforts between our two studies led to the creation of the Fidelity-based Presence Scale (FPS). With our consensus-driven approach and fidelity-based factoring, we hope the FPS will enable better communication within the research community and yield important future results regarding the relationship between VR system fidelity and presence.

Augmented reality (AR) and virtual reality (VR) applications commonly employ interaction cues that denote to the user what interaction to take. In this paper, we present a Method for Analyzing the Grammar of Incomplete Cues (MAGIC), which provides an approach for evaluating the design of interaction cues based on the completeness or incompleteness of the functional grammar that they convey through perceptual stimuli. To demonstrate the importance of complete cues, we also present a user study investigating the effects of complete and incomplete cues on which interactions participants choose. The results indicate that incomplete cues do not afford sufficient information, so users make assumptions about the intended interactions. Furthermore, the results indicate that users are more likely to choose intended interactions when the cues are complete. Hence, we present MAGIC as a potentially useful tool for helping interaction designers avoid usability issues with incomplete interaction cues.

In this paper, we review the existing benchmarks for continuous gesture recognition, e.g., the online analysis of hand movements over time to detect and recognize meaningful gestures from a specific dictionary. Focusing on human–computer interaction scenarios, we classify these benchmarks based on input data types, gesture dictionaries, and evaluation metrics. Specific metrics for the continuous recognition task are crucial for understanding how effectively gestures are spotted in real time within input streams. We also discuss the most effective detection and classification methods proposed for these benchmarks. Our findings indicate that the number and quality of publicly available datasets remain limited, and evaluation methodologies for continuous recognition are not yet standardized. These issues highlight the need for new benchmarks that reflect real-world usage conditions and can support the development of best practices in gesture-based interface design.

Communication in collaboration, especially synchronous, remote communication, is crucial to the success of task-specific goals. Insufficient or excessive forms of communication may lead to detrimental effects on task performance while increasing mental fatigue. However, identifying which combinations of communication modalities provide the most efficient transfer of information in collaborative settings will greatly improve collaboration. To investigate this, we developed a remote, synchronous, asymmetric VR collaborative assembly task application, where users play the role of either mentor or mentee, and were exposed to different combinations of three communication modalities: voice, gestures, and gaze. Through task-based experiments with 25 pairs of participants (50 individuals), we evaluated quantitative and qualitative data and found that gaze did not differ significantly from multiple combinations of communication modalities. Our qualitative results indicate that mentees experienced more difficulty and frustration in completing tasks than mentors, with both types of users preferring all three modalities to be present.

An underexplored interaction metaphor in virtual reality (VR) is throwing, with a considerable challenge in achieving accurate and natural results. We conducted an empirical investigation of participants’ performance in a VR throwing task, measuring their accuracy and preferences across Point of Release (PoR) mechanics (manual and automatic) with various input device categories (hand-held, on-body, external) and throwable object types. Participants were tasked with throwing a baseball, a bowling ball, and a football toward targets using 5 input configurations (2 manual and 3 automatic PoR). Results from 30 participants indicate that the overall highest accuracy was achieved with an automatic PoR configuration (on-body tracker). The post-study and VR survey results indicate that the majority of participants preferred a manual PoR configuration (hand-held VR controller-derived) for the throwing direction, throwing speed, and as being the closest to real-life throwing. Our findings are useful for VR researchers and developers who want to implement throwing as a technique in their applications.

Object selection has been explored extensively in the VR research literature. However, the research is typically conducted in constrained experimental setups. It remains unclear whether the designed selection techniques fit the prevalent practical uses and whether the experimental tasks represent important challenges in real applications. To identify and help bridge these gaps, we surveyed current consumer VR applications, containing 206 popular VR game and 3D modeling applications. We extracted 1300+ selection scenarios based on video analyses of these applications and derived a taxonomy to understand common patterns on where and how selections occur. Our findings reveal significant gaps in selection tasks and techniques between research and consumer applications. We also present an interactive visualization tool to help researchers explore the VR object selection scenarios. Finally, we discuss how our work can help researchers and developers evaluate techniques in meaningful tasks and drive the design of techniques.

Augmented Reality (AR) tools have demonstrated considerable promise to enhance creative architectural design and support the retrofitting problem-solving processes through on-site daylighting visualization. AR’s capacity to integrate embodied motion enhances the non-expert’s understanding of the spatial characteristics and design ramifications within the built environment for complex facade design. Motion provides insights and increases the accessibility of retrofitting, encouraging more energy-efficient rework as opposed to complete building reconstruction. This study investigates the decision-making outcomes and cognitive-physical load implications of integrating a Building Information Modeling-driven AR system into the retrofitting design process and how movement is best leveraged to understand daylighting impacts. We conducted a study with 128 non-expert participants, who were asked to choose a window facade retrofit to improve an interior space. We analyze the effects of head movement, head rotations, and eye movements to understand how embodied motion improves overall objective performance across several daylighting and energy design metrics. We found no significant difference in the overall decision-making outcome between those who used an AR tool or a conventional desktop approach and that greater eye movement in AR was related to non-experts better balancing the complicated impacts facades have on daylight, aesthetics, and energy. This study indicates future expansion of AR retrofitting tools should encourage more eye movement.

Gesture recognition is a tool to enable novel interactions with different techniques and applications, like Mixed Reality and Virtual Reality environments. With all the recent advancements in gesture recognition from skeletal data, it is still unclear how well state-of-the-art techniques perform in a scenario using precise motions with two hands. This paper presents the results of the SHREC 2024 contest organized to evaluate methods for their recognition of highly similar hand motions using the skeletal spatial coordinate data of both hands. The task is the recognition of 7 motion classes given their spatial coordinates in a frame-by-frame motion. The skeletal data has been captured using a Vicon system and pre-processed into a coordinate system using Blender and Vicon Shogun Post. We created a small, novel dataset with a high variety of durations in frames. This paper shows the results of the contest, showing the techniques created by the 5 research groups on this challenging task and comparing them to our baseline method.

We conducted a co-design study with 26 users in dyadic groups who were assigned to the opposing roles of "Streamers'' and "Viewers'' to design interactive, wearable telepresence prototypes for interpersonal use. The goal was to elicit values, identify value tensions, and resolve these tensions in the design of a system that allows Streamers to share their live experiences with a remote Viewer. Leveraging the lens of value-sensitive design (VSD), we found that these different stakeholders prioritized different values in design, but possibly due to our prompt to design for someone they cared about/knew, often accounted for one another's needs in their solutions to arrive at designs that were affordable, unobtrusive, and socially acceptable for the Streamer, while giving the Viewer a sense of autonomy. Our work highlights the strengths of co-design when eliciting important human values in the design of sociotechnical systems for wearable telepresence, reconciling value tensions, and conceptualizing novel hardware- and software-based solutions for enhancing the interpersonal telepresence experience of both viewers and streamers. A key insight from our study is that no single system will meet all users' needs; therefore, we should move towards building customizable toolkits to account for differing values and needs.

We developed and deployed an interpersonal telepresence prototype aimed at providing a positive one-to-one interaction between a Streamer and a Viewer. Our prototype uses four distributed, wearable cameras hidden from the public eye. It was designed to reduce the risk of Streamer self-consciousness while providing the Viewer with a greater sense of autonomy. We deployed our prototype with sixteen participants in dyads, who worked together to complete a scavenger hunt, and compared it the baseline of Skype. We found how our prototype better supported Streamer social well-being and physical comfort, and it also better supported Viewer autonomy. However, almost all participants desired a change to the design of the prototype, hinting that we need to provide better customization for future iterations of interpersonal telepresence devices.

Virtual Reality (VR) systems are widely used, and it is essential to know if spatial perception in virtual environments (VEs) is similar to reality. Research indicates that users tend to underestimate distances in VR. Prior work suggests that actual distance judgments in VR may not always match the users self-reported preference of where they think they most accurately estimated distances. However, no explicit investigation evaluated whether user preferences match actual performance in a spatial judgment task. We used blind walking to explore potential dissimilarities between actual distance estimates and user-selected preferences of visual complexities, VE conditions, and targets. Our findings show a gap between user preferences and actual performance when visual complexities were varied, which has implications for better visual perception understanding, VR applications design, and research in spatial perception, indicating the need to calibrate and align user preferences and true spatial perception abilities in VR.

Chronic lower back pain due to improper lifting techniques poses a major workplace safety hazard. The major risk factors for improper loading posture (ILP) include overloading, and improper loading of the lumbar muscles, ligaments, and vertebrae due to repetitive mechanical stresses exerted upon them. The current intervention technology relies on the users’ intent and willingness to self-correct ILP through alert-based feedback or involves wearing bulky lift assist devices to prevent ILP. We address these issues with a physiological feedback system that utilizes IMU sensors for ILP detection and Electrical Muscle Stimulation (EMS) for automatic dynamic ILP correction for restoring ideal lifting angles for torso inclination and knee bend. In a user study involving 36 participants, our automatic approach delivered significantly faster correction and outperformed alternative feedback mechanisms (Audio and Vibro-tactile) and was perceived to be interesting, comfortable and a potential commercial product.

Recurrent neural networks (RNN) require large training datasets from which they learn new class models. This limitation prohibits their use in custom gesture applications where only one or two end user samples are given per gesture class. One common way to enhance sparse datasets is to use data augmentation to synthesize new samples. Although there are numerous known techniques, they are often treated as standalone approaches when in reality they are often complementary. We show that by intelligently chaining augmentation techniques together that simulate different gesture production variability types, such as those affecting the temporal and spatial qualities of a gesture, we can significantly increase RNN accuracy without sacrificing training time. Through experimentation on four public stroke-based 2D gesture datasets, we show that RNNs trained with our data augmentation chaining technique achieves state-of-the-art recognition accuracy in both writer-dependent and writer-independent test scenarios.

We present a taxonomy of human-centered collaborative XR tasks. XR technologies have extended into the realm of collaboration, improving the quality and accessibility of teamwork. However, after a comprehensive assessment of the literature on the interaction between XR technologies and collaboration, no comprehensive method that emphasizes task actions and properties exists to classify collaborative tasks. Thus, our suggested taxonomy represents a classification system for collaborative tasks. After conducting a thorough literature review across different research venues, we conducted several exhaustive classification and review cycles for over 800 papers collected, which resulted in 148 papers retained to create the taxonomy. We dissected the actions and properties that the collaborative endeavors and tasks of these papers encompass as well as the types of categorizations and relations these papers illustrate. We expand on the design choices and usage of our taxonomy, followed by its limitations and future work. We built this taxonomy in order to reduce ambiguities and confusion regarding the design and comprehension of human-based collaborative tasks that use XR technology, which could prove useful in aiding the development and understanding of these tasks. Our taxonomy reveals a framework for understanding how collaborative tasks are designed and a systematic way of classifying different methods by which people can collaborate and interact in environments that involve XR, while still promoting efficient communication, teamwork, goal achievement and productivity.

Users tend to underestimate distances in virtual reality (VR), and several efforts have been directed toward finding the causes and developing tools that mitigate this phenomenon. One hypothesis that stands out in the field of spatial perception is the rest frame hypothesis (RFH), which states that visual frames of reference (RFs), defined as fixed reference points of view in a virtual environment (VE), contribute to minimizing sensory mismatch. RFs have been shown to promote better eye-gaze stability and focus, reduce VR sickness, and improve visual search, along with other benefits. However, their effect on distance perception in VEs has not been evaluated. In this paper, we use a blind walking task to explore the effect of three head-centric RFs (mesh mask, nose, and hat) on egocentric distance estimation. We found that at near and mid-field distances, certain RFs can improve the user’s distance estimation accuracy and reduce distance underestimation. These findings mean that the addition of head-centric RFs, a simple avatar augmentation method, can lead to meaningful improvements in distance judgments, user experience, and task performance in VR.

Standard selection techniques such as ray casting fail when virtual objects are partially or fully occluded. In this paper, we present two novel approaches that combine cone-casting, world-in-miniature, and grasping metaphors to disocclude objects in the representation local to the user. Through a within-subject study where we compared 4 selection techniques across 3 levels of object occlusion, we found that our techniques outperformed an alternative one that also focuses on maintaining the spatial relationships between objects. We discuss application scenarios and future research directions for these types of selection techniques.

Augmented Reality (AR) tools have shown significant potential in providing on-site visualization of Building Information Modeling (BIM) data and models for supporting construction evaluation, inspection, and guidance. Retrofitting existing buildings, however, remains a challenging task requiring more innovative solutions to successfully integrate AR and BIM. This study aims to investigate the impact of AR+BIM technology on the retrofitting training process and assess the potential for future on-site usage. We conducted a study with 64 non-expert participants, who were asked to perform a common retrofitting procedure of an electrical outlet installation using either an AR+BIM system or a standard printed blueprint documentation set. Our findings indicate that AR+BIM reduced task time significantly and improved performance consistency across participants, while also decreasing the physical and cognitive demands of the training. This study provides a foundation for augmenting future retrofitting construction research that can extend the use of [Formula: see text] technology, thus facilitating more efficient retrofitting of existing buildings. A video presentation of this article and all supplemental materials are available at https://github.com/DesignLabUCF/SENSEable_RetrofittingTraining.

We performed a mixed-design study with 56 participants to compare the effect of horizontal FOV (hfov) and vertical FOV (vfov) on egocentric distance perception in four different realistic virtual environments (VEs). We also compared VE attributes of indoor/outdoor and cluttered/uncluttered. The participants blind-walked towards four different targets at 3m, 4m, 5m, and 6m distance while wearing a backpack computer and a wide FOV head-mounted display (HMD). The combinations of 165°, 110° and 45° hfovs, and 110° and 35° vfovs was simulated in the same HMD. The results indicated more accurate distance judgement with larger hfov with no significant effect of vfov. More accurate distance judgement in indoor VEs compared to outdoor VEs was observed. Also, participants judged distances more accurately in cluttered environments versus uncluttered environments. These results highlight that the environment is important in distance-critical VR applications and wider hfov should be considered for an improved distance judgment.

Gesture recognition systems using nearest neighbor pattern matching are able to distinguish gesture from non-gesture actions by rejecting input whose recognition scores are poor. However, in the context of gesture customization, where training data is sparse, learning a tight rejection threshold that maximizes accuracy in the presence of continuous high activity (HA) data is a challenging problem. To this end, we present the Voight-Kampff Machine (VKM), a novel approach for rejection threshold selection. VKM uses new synthetic data techniques to select an initial threshold that the system thereafter adjusts based on the training set size and expected gesture production variability. We pair VKM with a state-of-the-art custom gesture segmenter and recognizer to evaluate our system across several HA datasets, where gestures are interleaved with non-gesture actions. Compared to alternative rejection threshold selection techniques, we show that our approach is the only one that consistently achieves high performance.

Training simulations in virtual reality (VR) have become a focal point of both research and development due to allowing users to familiarize themselves with procedures and tasks without needing physical objects to interact with or needing to be physically present. However, the increasing popularity of VR training paradigms raises the question: Are VR-based training assessments accurate? Many VR training programs, particularly those focused on inspection tasks, employ simple pass or fail assessments. However, these types of assessments do not necessarily reflect the user’s knowledge.

In recent years, pass-through cameras have resurfaced as inclusions for virtual reality (VR) hardware. With modern cameras that now have increased resolution and frame rate, Video See-Through (VST) Head-Mounted Displays (HMD) can be used to provide an Augmented Reality (AR) experience. However, because users see their surroundings through video capture and HMD lenses, there is question surrounding how people perceive their environment with these devices. We conducted a user study with 26 participants to help understand if distance perception is altered when viewing surroundings with a VST HMD. Although previous work shows that distance estimation in VR with an HTC Vive is comparable to that in the real world, our results show that the inclusion of a ZED Mini pass-through camera causes a significant difference between normal, unrestricted viewing and that through a VST HMD.

Synthetic data generation to improve classification performance (data augmentation) is a well-studied problem. Recently, generative adversarial networks (GAN) have shown superior image data augmentation performance, but their suitability in gesture synthesis has received inadequate attention. Further, GANs prohibitively require simultaneous generator and discriminator network training. We tackle both issues in this work. We first discuss a novel, device-agnostic GAN model for gesture synthesis called DeepGAN. Thereafter, we formulate DeepNAG by introducing a new differentiable loss function based on dynamic time warping and the average Hausdorff distance, which allows us to train DeepGAN’s generator without requiring a discriminator. Through evaluations, we compare the utility of DeepGAN and DeepNAG against two alternative techniques for training five recognizers using data augmentation over six datasets. We further investigate the perceived quality of synthesized samples via an Amazon Mechanical Turk user study based on the HYPE∞ benchmark. We find that DeepNAG outperforms DeepGAN in accuracy, training time (up to 17 × faster), and realism, thereby opening the door to a new line of research in generator network design and training for gesture synthesis. Our source code is available at https://www.deepnag.com.

It is important to learn from the past as we endeavor into this uncharted territory of mobile, human-to-human, one-to-one telepresence for interpersonal use. With the ever-increasing access to live-streaming cameras, we are now at the cusp of being able to create novel, immersive, and interpersonal telepresence activities that have the potential to change how humans interact with one another on a daily basis. Due to its novelty, there are likely socio-technical gaps between the needs of users and the technical specifications of the prototypes that are currently being designed to support the complex social interactions of human-to-human telepresence. Therefore, in this paper, we use a socio-technical lens to conduct a systematic literature review of 52 peer-reviewed articles of early work in this space. Overall, we found that while progress has been made to address the social needs of those involved in one-to-one telepresence scenarios, there are discontinuities within the existing literature that need to be addressed, particularly with the way we attempt to measure and quantify human-centered outcomes with unvalidated instruments. We also found that the social needs of on-site users have been neglected, as in many articles the user was merely treated as a surrogate, or reported feeling socially awkward or unsafe, due to the conspicuous nature of the body-worn technology in public environments. These findings are prevalent, even as researchers consider adding to this body-worn burden in an attempt to improve the receiving users' sense of immersion and presence. To preserve the beneficial nature of telepresence interaction while ensuring that all users' needs are met, researchers should endeavor to further understand the dynamics of the relationship between all parties in the remote environment. Our paper creates a future research agenda that emphasizes the importance of ensuring that all parties involved feel comfortable in their role during interpersonal telepresence interactions.

We present Machete, a straightforward segmenter one can use to isolate custom gestures in continuous input. Machete uses traditional continuous dynamic programming with a novel dissimilarity measure to align incoming data with gesture class templates in real time. Advantages of Machete over alternative techniques is that our segmenter is computationally efficient, accurate, device-agnostic, and works with a single training sample. We demonstrate Machete’s effectiveness through an extensive evaluation using four new high-activity datasets that combine puppeteering, direct manipulation, and gestures. We find that Machete outperforms three alternative techniques in segmentation accuracy and latency, making Machete the most performant segmenter. We further show that when combined with a custom gesture recognizer, Machete is the only option that achieves both high recognition accuracy and low latency in a video game application.

With the rise of immersive virtual reality and telepresence, it is important to understand the factors that contribute to creating an optimal user experience. In particular, there are divergent recommendations for setting camera heights in virtual contexts that facilitate telepresence. Therefore, we conducted a 2x2 mixed-design experiment with 93 college students asking them to select their preferred camera height when varying camera placement (overhead, chest) and the presence of human avatars (present, not present). We found that while camera placement did not have a significant effect of preferred camera height, the presence of avatars (increased height preference) and gender (women preferred lower heights) were significant. Our results provide evidence that factors within a virtual environment and individual differences influence users’ preferences of camera height. Thus, systems designed for immersive virtual reality and telepresence should customize camera height based on these factors.

Those who design gesture recognizers and user interfaces often use data collection applications that enable users to comfortably produce gesture training samples. In contrast, games present unique contexts that impact cognitive load and have the potential to elicit rapid gesticulations as players react to dynamic conditions, which can result in high gesture form variability. However, the extent to which these gestures differ is presently unknown. To this end, we developed two games with unique mechanics, Follow the Leader (FTL) and Sleepy Town, as well as a standard data collection application. We collected gesture samples from 18 participants across all conditions for gestures of varying complexity, and through an analysis using relative, global, and distribution coverage measures, we confirm significant differences between conditions. We discuss the implications of our findings, and show that our FTL design is closer to being an ecologically valid data collection protocol with low implementation complexity.

Graphical menus have been extensively used in desktop applications and widely adopted and integrated into virtual environments (VEs). However, while desktop menus are well evaluated and established, adopted 2D menus in VEs are still lacking a thorough evaluation. In this paper, we present the results of a comprehensive study on body-referenced graphical menus in a virtual environment. We compare menu placements (spatial, arm, hand, and waist) in conjunction with various shapes (linear and radial) and selection techniques (ray-casting with a controller device, head, and eye gaze). We examine task completion time, error rates, number of target re-entries, and user preference for each condition and provide design recommendations for spatial, arm, hand, and waist graphical menus. Our results indicate that the spatial, hand, and waist menus are significantly faster than the arm menus, and the eye gaze selection technique is more prone to errors and has a significantly higher number of target re-entries than the other selection techniques. Additionally, we found that a significantly higher number of participants ranked the spatial graphical menus as their favorite menu placement and the arm menu as their least favorite one.

In this paper we present AffordIt!, a tool for adding affordances to the component parts of a virtual object. Following 3D scene reconstruction and segmentation procedures, users find themselves with complete virtual objects, but no intrinsic behaviors have been assigned, forcing them to use unfamiliar Desktop-based 3D editing tools. AffordIt! offers an intuitive solution that allows a user to select a region of interest for the mesh cutter tool, assign an intrinsic behavior and view an animation preview of their work. To evaluate the usability and workload of AffordIt! we ran an exploratory study to gather feedback. In the study we utilize two mesh cutter shapes that select a region of interest and two movement behaviors that a user then assigns to a common household object. The results show high usability with low workload ratings, demonstrating the feasibility of AffordIt! as a valuable 3D authoring tool. Based on these initial results we also present a road-map of future work that will improve the tool in future iterations.

Depth perception of objects can greatly affect a user's experience of an augmented reality (AR) application. Many AR applications require depth matching of real and virtual objects and have the possibility to be influenced by depth cues. Color and luminance are depth cues that have been traditionally studied in two-dimensional (2D) objects. However, there is little research investigating how the properties of three-dimensional (3D) virtual objects interact with color and luminance to affect depth perception, despite the substantial use of 3D objects in visual applications. In this paper, we present the results of a paired comparison experiment that investigates the effects of object shape, fidelity, color, and luminance on depth perception of 3D objects in handheld mobile AR. The results of our study indicate that bright colors are perceived as nearer than dark colors for a high-fidelity, simple 3D object, regardless of hue. Additionally, bright red is perceived as nearer than any other color. These effects were not observed for a low-fidelity version of the simple object or for a more-complex 3D object. High-fidelity objects had more perceptual differences than low-fidelity objects, indicating that fidelity interacts with color and luminance to affect depth perception. These findings reveal how the properties of 3D models influence the effects of color and luminance on depth perception in handheld mobile AR and can help developers select colors for their applications.

Human gaze awareness is important for social and collaborative interactions. Recent technological advances in augmented reality (AR) displays and sensors provide us with the means to extend collaborative spaces with real-time dynamic AR indicators of one's gaze, for example via three-dimensional cursors or rays emanating from a partner's head. However, such gaze cues are only as useful as the quality of the underlying gaze estimation and the accuracy of the display mechanism. Depending on the type of the visualization, and the characteristics of the errors, AR gaze cues could either enhance or interfere with collaborations. In this paper, we present two human-subject studies in which we investigate the influence of angular and depth errors, target distance, and the type of gaze visualization on participants' performance and subjective evaluation during a collaborative task with a virtual human partner, where participants identified targets within a dynamically walking crowd. First, our results show that there is a significant difference in performance for the two gaze visualizations ray and cursor in conditions with simulated angular and depth errors: the ray visualization provided significantly faster response times and fewer errors compared to the cursor visualization. Second, our results show that under optimal conditions, among four different gaze visualization methods, a ray without depth information provides the worst performance and is rated lowest, while a combination of a ray and cursor with depth information is rated highest. We discuss the subjective and objective performance thresholds and provide guidelines for practitioners in this field.

Our work investigates body-worn 360° camera placements for telepresence, to balance height and clarity of view. We conducted a user study in a Virtual Reality (VR) simulation, using a 3x3 within-subjects experimental design varying placement and height, with 26 participants. We found that shoulder mounted cameras were significantly less preferable than our other conditions due to the occlusions caused by the wearer’s head. Our results did not show a significant effect of camera height within a range of +/- 12 inches from the user’s natural height. As such, in the context of body-worn 360° cameras, there is leeway for camera height, whereas strategic bodily placements are more important. Based on these results, we provide design recommendations for content creators using wearable cameras for immersive telepresence.

This work presents a study that explores the differences between authoring Scenario-Based Training (SBT) simulation content using Augmented Reality (AR) and a Desktop interface. Through an iterative design process two interface conditions were developed and then evaluated qualitatively and quantitatively. Our conceptual model is a graph based visualization that is presented to help designers understand the scenario learning artifacts and relationships. Our major contribution relies on the comparison made between the two authoring tools (AR, Desktop) with the same capabilities. Results show that no significant difference was found in time taken to complete tasks nor on the perceived usability of the systems. However, as expected the Desktop interface was perceived as more efficient. Based on these findings, insights on future directions for building AR immersive authoring tools are provided.

This work presents a comparison of two applications (Augmented Reality (AR) and desktop) to author Scenario-Based Training (SBT) simulations. Through an iterative design process two interface conditions are developed and then evaluated qualitatively and quantitatively. A graph based authoring visualization help designers understand the scenario learning artifacts and relationships. No significant difference was found on time taken to complete tasks nor on the perceived usability of the systems. However, Desktop was perceived as more efficient, corroborated by the significantly higher number of mistakes made in AR. Findings are presented towards building better AR immersive authoring tools.

While we are in the midst of a renaissance of interest in augmented reality (AR), there remain a small number of application domains which have seen significant development. Education is a domain that often drives innovation with emerging technologies. One particular subject which benefits from additional visualization capabilities is physics. In this paper, we present the results of a series of interviews with secondary school teachers about their experience with AR and the features which would be most beneficial to them from a pedagogical perspective. To gather meaningful information, a prototype application was developed and presented to the teachers. Based on the feedback collected from the teachers, we present a set of design recommendations for AR physics education tools, as well as other useful collects comments.

Accurate tracking of a user in a marker-less environment can be difficult, even more so when agile head or hand movements are expected. When relying on feature detection as part of a SLAM algorithm the issue arises that a large rotational delta causes previously tracked features to become lost. One approach to overcome this problem is with multiple sensors increasing the horizontal field of view. In this paper, we perform a systematic evaluation of tracking accuracy by recording several agile movements and providing different camera configurations to evaluate against. We begin with four sensors in a square configuration and test the resulting output from a chosen SLAM algorithm. We then systematically remove a camera from the feed covering all permutations to determine the level of accuracy and tracking loss. We cover some of the lessons learned in this preliminary experiment and how it may guide researchers in tracking extremely agile movements.

Past research has shown that humans exhibit certain eye-head responses to the appearance of visual stimuli, and these natural reactions change during different activities. Our work builds upon these past observations by offering new insight to how humans behave in Virtual Reality (VR) compared to Physical Reality (PR). Using eye- and head- tracking technology, and by conducting a study on two groups of users - participants in VR or PR - we identify how often these natural responses are observed in both environments. We find that users statistically move their heads more often when viewing stimuli in VR than in PR, and VR users also move their heads more in the presence of text. We open a discussion for identifying the HWD factors that cause this difference, as this may not only affect predictive models using eye movements as features, but also VR user experience overall.

Mobile applications ("apps") developed to promote online safety for children are underutilized and rely heavily on parental control features that monitor and restrict their child's mobile activities. This asymmetry in parental surveillance initiates an interesting research question -- how do children themselves feel about such parental control apps? We conducted a qualitative analysis of 736 reviews of 37 mobile online safety apps from Google Play that were publicly posted and written by children (ages 8-19). Our results indicate that child ratings were significantly lower than that of parents with 76% of the child reviews giving apps a single star. Children felt that the apps were overly restrictive and invasive of their personal privacy, negatively impacting their relationships with their parents. We relate these findings with HCI literature on mobile online safety, including broader literature around privacy and surveillance, and outline design opportunities for online safety apps.

We introduce an interactive system for extracting the geometries of generalized cylinders and cuboids from single-or multiple-view point clouds. Our proposed method is intuitive and only requires the object's silhouettes to be traced by the user. Leveraging the user's perceptual understanding of what an object looks like, our proposed method is capable of extracting accurate models, even in the presence of occlusion, clutter or incomplete point cloud data, while preserving the original object's details and scale. We demonstrate the merits of our proposed method through a set of experiments on a public RGB-D dataset. We extracted 16 objects from the dataset using at most two views of each object. Our extracted models represent a high degree of visual similarity to the original objects. Further, we achieved a mean normalized Hausdorff distance of 5.66% when comparing our extracted models with the dataset's ground truths.

Previous research has shown that sketch-based input is efficient and preferable in the context of algebraic equation solving. However, research has not been conducted to evaluate whether this holds true when involving geometry input to facilitate quantitative problem-solving. We developed a bimodal (graphing geometric shapes and writing algebraic expressions) user interface, in order to conduct a within-subject, controlled experiment with 24 college students and varied two types of geometry input: 1) sketch-based input and 2) structured input. The sketch-based input was significantly faster than the structured input, but there were no significant differences based on perception and cognition. However, after a post-hoc analysis, we found a significant interaction effect on perception between prior knowledge and geometry input. Novice students preferred the sketch-based input, but advanced students preferred the structured input. Our study implies that natural sketch-based input may be less preferable than structured input for geometry-based interfaces toward math problem-solving.

Despite decades of research, there is yet no general rapid prototyping recognizer for dynamic gestures that can be trained with few samples, work with continuous data, and achieve high accuracy that is also modality-agnostic. To begin to solve this problem, we describe a small suite of accessible techniques that we collectively refer to as the Jackknife gesture recognizer. Our dynamic time warping based approach for both segmented and continuous data is designed to be a robust, go-to method for gesture recognition across a variety of modalities using only limited training samples. We evaluate pen and touch, Wii Remote, Kinect, Leap Motion, and sound-sensed gesture datasets as well as conduct tests with continuous data. Across all scenarios we show that our approach is able to achieve high accuracy, suggesting that Jackknife is a capable recognizer and good first choice for many endeavors.

We introduce Code Park, a novel tool for visualizing codebases in a 3D game-like environment. Code Park aims to improve a programmer's understanding of an existing codebase in a manner that is both engaging and intuitive, appealing to novice users such as students. It achieves these goals by laying out the codebase in a 3D park-like environment. Each class in the codebase is represented as a 3D room-like structure. Constituent parts of the class (variable, member functions, etc.) are laid out on the walls, resembling a syntax-aware "wallpaper". The users can interact with the codebase using an overview, and a first-person viewer mode. We conducted two user studies to evaluate Code Park's usability and suitability for organizing an existing project. Our results indicate that Code Park is easy to get familiar with and significantly helps in code understanding compared to a traditional IDE. Further, the users unanimously believed that Code Park was a fun tool to work with.

Most modern stereoscopic 3D applications use fixed stereoscopic 3D parameters (separation and convergence) to render the scene on a 3D display. But, keeping these parameters fixed during usage does not always provide the best experience since it can reduce the amount of depth perception possible in some applications which have large variability in object distances. We developed two stereoscopic rendering techniques which actively vary the stereo parameters based on the scene content. Our first algorithm calculates a low resolution depth map of the scene and chooses ideal stereo parameters based on that depth map. Our second algorithm uses eye tracking data to get the gaze direction of the user and chooses ideal stereo parameters based on the distance of the gazed object. We evaluated our techniques in an experiment that uses three depth judgment tasks: depth ranking, relative depth judgment and path tracing. Our results indicate that variable stereo parameters provide enhanced depth discrimination compared to static parameters and were preferred by our participants over the traditional fixed parameter approach. We discuss our findings and possible implications on the design of future stereoscopic 3D applications.

We present AnalyticalInk, a novel math learning environment prototype that uses a semantic graph as the knowledge representation of algebraic and geometric word problems. The system solves math problems by reasoning upon the semantic graph and automatically generates conceptual and procedural scaffoldings in sequence. We further introduces a step-wise tutoring framework, which can check students' input steps and provide the adaptive scaffolding feedback. Based on the knowledge representation, AnalyticalInk highlights keywords that allow users to further drag them onto the workspace to gather insight into the problem's initial conditions. The system simulates a pen-and-paper environment to let users input both in algebraic and geometric workspaces. We conducted an usability evaluation to measure the effectiveness of AnalyticalInk. We found that keyword highlighting and dragging is useful and effective toward math problem solving. Answer checking in the tutoring component is useful. In general, our prototype shows the promise in helping users to understand geometrical concepts and master algebraic procedures under the problem solving.

We explore the benefits of intelligent prototype selection for $-family recognizers. Currently, the state of the art is to randomly select a subset of prototypes from a dataset without any processing. This results in reduced computation time for the recognizer, but also increases error rates. We propose applying optimization algorithms, specifically random mutation hill climb and a genetic algorithm, to search for reduced sets of prototypes that minimize recognition error. After an evaluation, we found that error rates could be reduced compared to random selection and rapidly approached the baseline accuracies for a number of different $-family recognizers.

Training gesture recognizers with synthetic data generated from real gestures is a well known and powerful technique that can significantly improve recognition accuracy. In this paper we introduce a novel technique called gesture path stochastic resampling (GPSR) that is computationally efficient, has minimal coding overhead, and yet despite its simplicity is able to achieve higher accuracy than competitive, state-of-the-art approaches. GPSR generates synthetic samples by lengthening and shortening gesture subpaths within a given sample to produce realistic variations of the input via a process of nonuniform resampling. As such, GPSR is an appropriate rapid prototyping technique where ease of use, understandability, and efficiency are key. Further, through an extensive evaluation, we show that accuracy significantly improves when gesture recognizers are trained with GPSR synthetic samples. In some cases, mean recognition errors are reduced by more than 70%, and in most cases, GPSR outperforms two other evaluated state-of-the-art methods.

Math boxes is a recently introduced pen-based user interface for simplifying the task of hand writing difficult mathematical expressions. Visible bounding boxes around subexpressions are automatically generated as the system detects relevant spatial relationships between symbols including superscripts, subscripts, and fractions. Subexpressions contained in a math box can then be extended by adding new terms directly into its given bounds. When new characters are accepted, box boundaries are dynamically resized and neighboring terms are translated to make room for the larger box. Feedback on structural recognition is given via the boxes themselves. In this work, we extend the math boxes interface to include support for subexpression modifications via a new set of pen-based interactions. Specifically, techniques to expand and rearrange terms in a given expression are introduced. To evaluate the usefulness of our proposed methods, we first conducted a user study in which participants wrote a variety of equations ranging in complexity from a simple polynomial to the more difficult expected value of the logistic distribution. The math boxes interface is compared against the commonly used offset typeset (small) method, where recognized expressions are typeset in a system font near the user’s unmodified ink. In this initial study, we find that the fluidness of the offset method is preferred for simple expressions but that, as difficulty increases, our math boxes method is overwhelmingly preferred. We then conducted a second user study that focused only on modifying various mathematical expressions. In general, participants worked faster with the math boxes interface, and most new techniques were well received. On the basis of the two user studies, we discuss the implications of the math boxes interface and identify areas where improvements are possible.

We present the results of a comprehensive video game study which explores how the gaming experience is effected when several 3D user interface technologies are used simultaneously. We custom designed an air-combat game integrating several 3DUI technologies (stereoscopic 3D, head tracking, and finger-count gestures) and studied the combined effect of these technologies on the gaming experience. Our game design was based on existing design principles for optimizing the usage of these technologies in isolation. Additionally, to enhance depth perception and minimize visual discomfort, the game dynamically optimizes stereoscopic 3D parameters (convergence and separation) based on the user's look direction. We conducted a within subjects experiment where we examined performance data and self-reported data on users perception of the game. Our results indicate that participants performed significantly better when all the 3DUI technologies (stereoscopic 3D, head-tracking and finger-count gestures) were available simultaneously with head tracking as a dominant factor. We explore the individual contribution of each of these technologies to the overall gaming experience and discuss the reasons behind our findings.

We present math boxes, a novel pen-based user interface for simplifying the task of hand writing difficult mathematical expressions. Visible bounding boxes around certain subexpressions are automatically generated as the system detects specific relationships including superscripts, subscripts, and fractions. Subexpressions contained in a box can then be extended by adding new terms directly into its given bounds. Upon accepting new characters, box boundaries are dynamically resized and neighboring terms are translated to make room for the larger box. Feedback on structural recognition is given via the boxes themselves. We also provide feedback on character recognition by morphing the user's individual characters into a cleaner version stored in our ink database.

Sketch recognition has the potential to be an important input method for computers in the coming years, particularly for STEM (science, technology, engineering, and math) education. However, designing and building an accurate and sophisticated sketch recognition system is a time-consuming and daunting task. Since sketch recognition mistakes are still common, it is important to understand how users perceive and tolerate recognition errors and other user interface elements with these imperfect systems. In order to solve this problem, we developed a Wizard of Oz sketch recognition tool, the WOZ Recognizer, that supports controlled recognition accuracy, multiple recognition modes, and multiple sketching domains for performing controlled experiments. We present the design of the WOZ Recognizer and our process for representing recognition domains using graphs and symbol alphabets. In addition, we discuss how sketches are altered, how to control the WOZ Recognizer, and how users interact with it. Finally, we present an expert user case study that examines the WOZ Recognizer’s usability.

We present a systematic exploration of how to utilize video game context (e.g., player and environmental state) to modify and augment existing 3D gesture recognizers to improve accuracy for large gesture sets. Specifically, our work develops and evaluates three strategies for incorporating context into 3D gesture recognizers. These strategies include modifying the well-known Rubine linear classifier to handle unsegmented input streams and per-frame retraining using contextual information (CA-Linear); a GPU implementation of dynamic time warping (DTW) that reduces the overhead of traditional DTW by utilizing context to evaluate only relevant time sequences inside of a multithreaded kernel (CA-DTW); and a multiclass SVM with per-class probability estimation that is combined with a contextually based prior probability distribution (CA-SVM). We evaluate each strategy using a Kinect-based third-person perspective VE game prototype that combines parkour-style navigation with hand-to-hand combat. Using a simple gesture collection application to collect a set of 57 gestures and the game prototype that implements 37 of these gestures, we conduct three experiments. In the first experiment, we evaluate the effectiveness of several established classifiers on our gesture set and demonstrate state-of-the-art results using our proposed method. In our second experiment, we generate 500 random scenarios having between 5 and 19 of the 57 gestures in context. We show that the contextually aware classifiers CA-Linear, CA-DTW, and CA-SVM significantly outperform their non--contextually aware counterparts by 37.74%, 36.04%, and 20.81%, respectively. On the basis of the results of the second experiment, we derive upper-bound expectations for in-game performance for the three CA classifiers: 96.61%, 86.79%, and 96.86%, respectively. Finally, our third experiment is an in-game evaluation of the three CA classifiers with and without context. Our results show that through the use of context, we are able to achieve an average in-game recognition accuracy of 89.67% with CA-Linear compared to 65.10% without context, 79.04% for CA-DTW compared to 58.1% without context, and 90.85% with CA-SVM compared to 75.2% without context.

Counting using one's fingers is a potentially intuitive way to enumerate a list of items and lends itself naturally to gesture-based menu systems. In this paper, we present the results of the first comprehensive study on Finger-Count menus to investigate its usefulness as a viable option for 3D menu selection tasks. Our study compares 3D gesture-based finger counting (Finger Count menus) with two gesture-based menu selection techniques (Hand-n-Hold, Thumbs-Up), derived from existing motion-controlled video game menu selection strategies, as well as 3D Marking menus. We examined selection time, selection accuracy and user preference for all techniques. We also examined the impact of different spatial layouts for menu items and different menu depths. Our results indicate that Finger-Count menus are significantly faster than the other menu techniques we tested and are the most liked by participants. Additionally, we found that while Finger-Count menus and 3D Marking menus have similar selection accuracy, Finger-Count menus are almost twice as fast compared to 3D Marking menus.

We explore the potential of stereoscopic 3D (S3D) vision in offering distinct gameplay using an S3D-specific game called Deepress3D. Our game utilizes established S3D design principles for optimizing GUI design, visual comfort and game mechanics which rely on depth perception in time-pressured spatial conflicts. The game collects detailed S3D player metrics and allows players to choose between different, evenly matched strategies. We conducted a between subjects study comparing S3D and monoscopic versions of Deepress3D that examined player behavior and performance and measured user-reported data on presence, simulator sickness, and game experience. Confirming previous results, stereo users reported higher spatial presence. More importantly, for the first time, our game metrics indicate that S3D vision can measurably change player behavior depending on actual game content and level design, without necessarily affecting performance or emotional experience. These findings indicate the potential for optimizing applications for stereo users distinguishing them as a distinct group in HCI research.

We describe a novel theoretical framework for modeling structured drawings which contain one or more patterns of repetition in their constituent elements. We then present PatternSketch, a sketch-based drawing tool built using our framework to allow quick construction of structured drawings. PatternSketch can recognize and beautify drawings containing line segments, polylines, arcs, and circles. Users can employ a series of gestures to identify repetitive elements and create new elements based on automatically inferred patterns. PatternSketch leverages the programming-by-example (PBE) paradigm, enabling it to infer non-trivial patterns from a few examples. We show that PatternSketch, with its sketch-based user interface and a unique pattern inference algorithm, enables efficient and natural construction of structured drawings.

We explore the capabilities of head tracking combined with head mounted displays (HMD) as an input modality for robot navigation. We use a Parrot AR Drone to test five techniques which include metaphors for plane-like banking control, car-like turning control and virtual reality-inspired translation and rotation schemes which we compare with a more traditional game controller interface. We conducted a user study to observe the effectiveness of each of the interfaces we developed in navigating through a number of archways in an indoor course. We examine a number of qualitative and quantitative metrics to determine performance and preference among each metaphor. Our results show an appreciation for head rotation based controls over other head gesture techniques, with the classic controller being preferred overall. We discuss possible shortcomings with head tracked HMDs as a primary input method as well as propose improved metaphors that alleviate some of these drawbacks.

We present a modeling approach to develop an agent that assists users discreetly in teleoperation when avateering a robot via an inexpensive motion sensor. Avateering is a powerful metaphor, and can be an effective teleoperating strategy. Avateering a Humanoid Robot (HR) with no wearable device encumberment, such as using the popular Kinect/NUI motion sensor, is also desirable and very promising. However, this control scheme makes it difficult for the slave robot to make contact and interact with objects with high accuracy due to factors such as viewpoint, individually-unique and unilateral human-side control imprecision, and lack of informative tactile feedback. Our research explores the addition of an assistive agent that arbitrates user input without disrupting the overall experience and expectation. Additionally, our agent assists with maintaining a higher level of accuracy for interaction tasks, in our case, a grasping and lifting scenario. Using theWebots robot simulator, we implemented 4 assistive agents to augment the user in avateering the Darwin-OP robot. The agent iterations are described, and results of a user study are presented. We discuss user perception towards the avateering metaphor when enhanced by the agent and also when unassisted, including perceived easiness of the task, responsiveness of the robot, and accuracy.

We present CSTutor, a sketch-based interface designed to help students understand data structures, specifically Linked Lists, Binary Search Trees, AVL Trees, and Heaps. CSTutor creates an environment that seamlessly combines a user’s sketched diagram and code. In each of these data structure modes, the user can naturally sketch a data structure on the canvas just as they would on a white board. CSTutor analyzes the user’s diagrams in real time, and automatically generates code in a separate code view to reflect any changes the user has made. Additionally, the code can also be edited and any new code changes will animate the data structure drawn on the canvas. The connection between the data structure drawn on the canvas and the code implementation is intended to bridge the gap between the conceptual diagram of a data structure and the actual implementation. We also present the results of two semester-long studies using CSTutor in a CS1 course. The results indicate that students preferred CSTutor and were more engaged using it than a standard whiteboard lecture; however, results were mixed in quiz and exam performance.

ABSTRACT Code Bubbles is an attempt to redefine the user interface for an integrated programming environment. As it represents a whole new user interface, implementing it as a plug‐in is inherently difficult. We get around this difficulty by combining two different plug‐in architectures, a standard one based on registrations and callbacks and a message‐based one that puts the plug‐in at arm's length and defines a narrower two‐way interface. This paper describes both how we have implemented Code Bubbles as a plug‐in to Eclipse and how Code Bubbles itself is implemented as a set of plug‐ins representing the different aspects of the environment, using both traditional and message‐based plug‐in architectures as appropriate. It also shows how the resultant architecture is flexible enough to support collaboration, different back ends, and a cloud‐based environment. Copyright © 2013 John Wiley & Sons, Ltd.

Creating graphical user interfaces (GUI) for stereoscopic 3D (S3D) games is a difficult choice between visual comfort and effect. We present a S3D Game GUI Design Space and a list of S3D-specific attributes that emphasizes integrating visually comfortable interfaces into the game world, story and S3D view. To showcase our approach, we created two GUI concepts and evaluated them with 32 users. Our results show quality improvements for a combination of bottom position and visual attachment for a menu. In a referencing interface, placing the reference near to the target depth significantly improved perceived quality, game integration, and increased presence. These results confirm the need to create S3D GUIs with perceptual constraints in mind, demonstrating the potential to extend the user experience. Additionally, our design space offers a formal and flexible way to create new effects in S3D GUIs.

We present a study exploring upper body 3D spatial interaction metaphors for control and communication with Unmanned Aerial Vehicles (UAV) such as the Parrot AR Drone. We discuss the design and implementation of five interaction techniques using the Microsoft Kinect, based on metaphors inspired by UAVs, to support a variety of flying operations a UAV can perform. Techniques include a first-person interaction metaphor where a user takes a pose like a winged aircraft, a game controller metaphor, where a user's hands mimic the control movements of console joysticks, "proxy" manipulation, where the user imagines manipulating the UAV as if it were in their grasp, and a pointing metaphor in which the user assumes the identity of a monarch and commands the UAV as such. We examine qualitative metrics such as perceived intuition, usability and satisfaction, among others. Our results indicate that novice users appreciate certain 3D spatial techniques over the smartphone application bundled with the AR Drone. We also discuss the trade-offs in the technique design metrics based on results from our study.

Selection in 3D virtual environments can vary wildly depending on the context of the selection. Scene attributes such as object velocity, scene density, and user's cursor velocity can impact the user's ability to accurately select an object. Many 3D selection techniques have been explored, and are usually optimal for a specific set of conditions. As a result, software developers must compromise by choosing a single technique that works well on average, but is lacking in at least one scenario. We present a preliminary study that explores the feasibility of new autoselection algorithms that automatically determines the most appropriate selection technique in real-time, thus leveraging the performance benefits of each technique. We evaluated the techniques across three levels of scene density and three levels of object velocity.

Presents the table of contents of the proceedings of the 2013 IEEE Virtual Reality Conference as published in this issue of the IEEE Transactions on Visualization and Computer Graphics .

The apers in this special issue were presented at the 2013 IEEE Virtual Reality Conference held March 16-20, 2013, in Orlando, Florida.

We present QuickDraw, a prototype sketch-based drawing tool, that facilitates drawing of precise geometry diagrams that are often drawn by students and academics in several scientific disciplines. Quickdraw can recognize sketched diagrams containing components such as line segments and circles, infer geometric constraints relating recognized components, and use this information to beautify the sketched diagram. Beautification is based on a novel algorithm that iteratively computes various sub-components of the components using an extensible set of deductive rules. We conducted a user study comparing QuickDraw with four state-of-the-art diagramming tools: Microsoft PowerPoint, Cabri II Plus, Geometry Expressions and Geometer's SketchPad. Our study demonstrates a strong interest among participants for the use of sketch-based software for drawing geometric diagrams. We also found that QuickDraw enables users to draw precise diagrams faster than the majority of existing tools in some cases, while having them make fewer corrections.

Recent advances in digital game technology are making stereoscopic games more popular. Stereoscopic 3D graphics promise a better gaming experience but this potential has not yet been proven empirically. In this paper, we present a comprehensive study that evaluates player experience of three stereoscopic games in comparison with their monoscopic counterparts. We examined 60 participants, each playing one of the three games, using three self-reporting questionnaires and one psychophysiological instrument. Our main results are (1) stereoscopy in games increased experienced immersion, spatial presence, and simulator sickness; (2) the effects strongly differed across the three games and for both genders, indicating more affect on male users and with games involving depth animations; (3) results related to attention and cognitive involvement indicate more direct and less thoughtful interactions with stereoscopic games, pointing towards a more natural experience through stereoscopy.

We present a study that investigates user performance benefits of playing video games using 3D motion controllers in 3D stereoscopic vision in comparison to monoscopic viewing. Using the PlayStation 3 game console coupled with the PlayStation Move Controller, we explored five different games that combine 3D stereo and 3D spatial interaction. For each game, quantitative and qualitative measures were taken to determine if users performed better and learned faster in the experimental group (3D stereo display) than in the control group (2D display). A game expertise pre-questionnaire was used to classify participants into beginners and expert game player categories to analyze a possible impact on performance differences. The results show two cases where the 3D stereo display did help participants perform significantly better than with a 2D display. For the first time, we can report a positive effect on gaming performance based on stereoscopic vision, although reserved to isolated tasks and depending on game expertise. We discuss the reasons behind these findings and provide recommendations for game designers who want to make use of 3D stereoscopic vision and 3D motion control to enhance game experiences.

Our original work on the Code Bubbles environment demonstrated that a working-set based framework for software development showed promise. We have spent the past several years extending the underlying concepts into a fully-functional system. In our demonstration, we will show the current Code Bubbles environment for Java, how it works, how it can be used, and why we prefer it over more traditional programming environments. We will also show how we have extended the framework to enhance software development tasks such as complex debugging, testing, and collaboration. This paper describes the features we will demonstrate.

We present PhysicsBook, a prototype system that enables users to solve physics problems using a sketch-based interface and then animates any diagram used in solving the problem to show that the solution is correct. PhysicsBook recognizes the diagrams in the solution and infers relationships among diagram components through the recognition of mathematics and annotations such as arrows and dotted lines. For animation, PhysicsBook uses a customized physics engine that provides entry points for hand-written mathematics and diagrams. We discuss the design of PhysicsBook, including details of algorithms for sketch recognition, inference of user intent and creation of animations based on the mathematics written by a user. Specifically, we describe how the physics engine uses domain knowledge to perform data transformations in instances where it cannot use a given equation directly. This enables PhysicsBook to deal with domains of problems that are not directly related to classical mechanics. We provide examples of scenarios of how PhysicsBook could be used as part of an intelligent tutoring system and discuss the strengths and weaknesses of our current prototype. Lastly, we present the findings of a preliminary usability study with five participants.

We present LogicPad, a pen-based application for boolean algebra visualization that lets users manipulate boolean function representations through handwritten symbol and gesture recognition coupled with a drag-and-drop interface. We discuss LogicPad's user interface and the general algorithm used for verifying the equivalence of three different boolean function representations: boolean expressions, truth tables, and logic gate diagrams. We also conducted a short, informal user study evaluating LogicPad's user interface, visualization techniques, and overall performance. Results show that visualizations were generally well-liked and verification results matched user expectations.

We present CSTutor, a sketch-based interface designed to help students understand data structures. It currently supports Linked Lists, Binary Search Trees, AVL Trees, and Heaps, and creates an environment in which a user's sketched diagram and code are combined seamlessly. In each of the data structure modes, the user can naturally sketch a data structure on the canvas just as they would on the white board. CSTutor analyzes the user's diagrams in real time, and automatically generates code in a separate code view to reflect any changes the user has made. Additionally, the code can also be edited and any new code changes animate the data structure drawn on the canvas. The connection between the data structure drawn on the canvas and the code implementation is intended to bridge the gap between the conceptual diagram of a data structure and the actual implementation. We also present the results of a perceived usefulness survey. The results of the study indicate that the majority of students would find CSTutor helpful for learning data structures.

3D object selection is more demanding when, 1) objects densly surround the target object, 2) the target object is significantly occluded, and 3) when the target object is dynamically changing location. Most 3D selection techniques and guidelines were developed and tested on static or mostly sparse environments. In contrast, games tend to incorporate densly packed and dynamic objects as part of their typical interaction. With the increasing popularity of 3D selection in games using hand gestures or motion controllers, our current understanding of 3D selection needs revision. We present a study that compared four different selection techniques under five different scenarios based on varying object density and motion dynamics. We utilized two existing techniques, Raycasting and SQUAD, and developed two variations of them, Zoom and Expand, using iterative design. Our results indicate that while Raycasting and SQUAD both have weaknesses in terms of speed and accuracy in dense and dynamic environments, by making small modifications to them (i.e., flavoring), we can achieve significant performance increases.

We present a prototype system for interactive construction and modification of 3D physical models using building blocks. Our system uses a depth sensing camera and a novel algorithm for acquiring and tracking the physical models. The algorithm, Lattice-First, is based on the fact that building block structures can be arranged in a 3D point lattice where the smallest block unit is a basis in which to derive all the pieces of the model. The algorithm also makes it possible for users to interact naturally with the physical model as it is acquired, using their bare hands to add and remove pieces. We present the details of our algorithm, along with examples of the models we can acquire using the interactive system. We also show the results of an experiment where participants modify a block structure in the absence of visual feedback. Finally, we discuss two proof-of-concept applications: a collaborative guided assembly system where one user is interactively guided to build a structure based on another user's design, and a game where the player must build a structure that matches an on-screen silhouette.

We present a study that investigates user performance benefits of 3D stereo in modern video games. Based on an analysis of several video games that are best suited for use with commercial 3D stereo drivers and vision systems, we chose five modern titles focusing on racing, first and third person shooter, and sports game genres. For each game, quantitative and qualitative measures were taken to determine if users performed better and learned faster in the experimental group (3D stereo display) than in the control group (2D display). A game experience pre-questionnaire was used to classify participants into beginner, intermediate, and advanced gameplay categories to ensure prior game experience did not bias the experiment. Our results indicate that although participants preferred playing in 3D stereo for the games we tested, it does not provide any significant advantage in overall user performance. In addition, users' learning rates were comparable in the 3D stereo display and 2D display cases.

We present a user study that begins to explore aspects of players experience in first person games that use 3D gestures for interaction. Our study uses Wizard of Wii, a video game prototype that has players perform a set of 25 distinct gestures over the course of four different game quests using the Nintendo Wii Remote. Our results indicate that players' ability to recall gestures improved with repeated play and they believed themselves to be performing better in the game. However, while the recognition accuracy improved significantly with repeated play, players were unable to perceive the difference.

Developers spend significant time reading and navigating code fragments spread across multiple locations. The file-based nature of contemporary IDEs makes it prohibitively difficult to create and maintain a simultaneous view of such fragments. We propose a novel user interface metaphor for code understanding based on collections of lightweight, editable fragments called bubbles, which form concurrently visible working sets. We present the results of a qualitative usability evaluation, and the results of a quantitative study which indicates Code Bubbles significantly improved code understanding time, while reducing navigation interactions over a widely-used IDE, for two controlled tasks.

We present a Wizard-of-Oz study exploring full body video game interaction. Using the commercial video game Mirror's Edge, players are presented with several different tasks such as running, jumping, and climbing. Following our protocol, participants were given complete freedom in choosing the motions and gestures to compete these tasks. Our experiment results show a mix of natural and constrained gestures adapted to space and field of view restrictions. We present guidelines for future full body interfaces.

Today's integrated development environments (IDEs) are hampered by their dependence on files and file-based editing. We propose a novel user interface that is based on collections of lightweight editable fragments, called bubbles, which when grouped together form concurrently visible working sets. In this paper we describe the design of a prototype IDE user interface for Java based on working sets. A quantitative evaluation shows that developers could expect to view a sizeable number of functions concurrently with relatively few UI operations. A qualitative user evaluation with 23 professional developers indicates a high level of excitement, interest, and potential benefits and uses.

Today's integrated development environments (IDEs) are hampered by their dependence on files and file-based editing. We propose a novel user interface that is based on collections of lightweight editable fragments, called bubbles, which when grouped together form concurrently visible working sets. We describe the design of a prototype IDE user interface for Java based on working sets.

We present a new approach for creating dynamic illustrations to assist in the understanding of concepts in physics and mathematics using pen-based interaction. Our approach builds upon mathematical sketching by combining the ability to make associations between handwritten mathematics and free-form drawings with an underlying physics engine. This combination lets users create animations without having to directly specify object behavior with position functions through time, yet still supports writing the mathematics needed to formulate a problem. This functionality significantly expands the capabilities of mathematical sketching to support a wider variety of dynamic illustrations. We describe our approach to creating this mathematical sketching/physics engine fusion and discuss how it provides a foundation for using mathematical sketching in intelligent tutoring systems.

We present the development and evaluation of the Virtual Experience Test (VET). The VET is a survey instrument used to measure holistic virtual environment experiences based upon the five dimensions of experiential design: sensory, cognitive, affective, active, and relational. Experiential Design (ED) is a holistic approach to enhance presence in virtual environments that goes beyond existing presence theory (i.e. a focus on the sensory aspects of VE experiences) to include affective and cognitive factors. To evaluate the VET, 62 participants played the commercial video game Mirror's Edge. After gameplay both the VET and the ITC-Sense of Presence Inventory (ITC-SOPI) were administered. A principal component analysis was performed on the VET and it was determined that the actual question clustering coincided with the proposed dimensions of experiential design. Furthermore, scores from the VET were shown to have a significant relationship with presence scores on the ITC-SOPI. The results of this research produced a validated measure of holistic experience that could be used to evaluate virtual environments. Furthermore, our experiment indicates that virtual environments utilizing holistic designs can result in significantly higher presence.

We present a novel, real-time, markerless vision-based tracking system, employing a rigid orthogonal configuration of two pairs of opposing cameras. Our system uses optical flow over sparse features to overcome the limitation of vision-based systems that require markers or a pre-loaded model of the physical environment. We show how opposing cameras enable cancellation of common components of optical flow leading to an efficient tracking algorithm. Experiments comparing our device with an electromagnetic tracker show that its average tracking accuracy is 80% over 185 frames, and it is able to track large range motions even in outdoor settings.

We present a systematic study on the recognition of 3D gestures using spatially convenient input devices. Specifically, we examine the linear acceleration-sensing Nintendo Wii Remote coupled with the angular velocity-sensing Nintendo Wii MotionPlus. For the study, we created a 3D gesture database, collecting data on 25 distinct gestures totalling 8500 gestures samples. Our experiment explores how the number of gestures and the amount of gestures samples used to train two commonly used machine learning algorithms, a linear and AdaBoost classifier, affect overall recognition accuracy. We examined these gesture recognition algorithms with user dependent and user independent training approaches and explored the affect of using the Wii Remote with and without the Wii MotionPlus attachment. Our results show that in the user dependent case, both the Ad-aBoost and linear classification algorithms can recognize up to 25 gestures at over 90% accuracy, with 15 training samples per gesture, and up to 20 gestures at over 90% accuracy, with only five training samples per gesture. In particular, all 25 gestures could be recognized at over 99% accuracy with the linear classifier using 15 training samples per gesture, with the Wii Remote coupled with the Wii MotionPlus. In addition, both algorithms can recognize up to nine gestures at over 90% accuracy using a user independent training database with 100 samples per gesture. The Wii MotionPlus attachment played a significant role in improving accuracy in both the user dependent and independent cases.

GestureBar is a novel, approachable UI for learning gestural interactions that enables a walk-up-and-use experience which is in the same class as standard menu and toolbar interfaces. GestureBar leverages the familiar, clean look of a common toolbar, but in place of executing commands, richly discloses how to execute commands with gestures, through animated images, detail tips and an out-of-document practice area. GestureBar's simple design is also general enough for use with any recognition technique and for integration with standard, non-gestural UI components. We evaluate GestureBar in a formal experiment showing that users can perform complex, ecologically valid tasks in a purely gestural system without training, introduction, or prior gesture experience when using GestureBar, discovering and learning a high percentage of the gestures needed to perform the tasks optimally, and significantly outperforming a state of the art crib sheet. The relative contribution of the major design elements of GestureBar is also explored. A second experiment shows that GestureBar is preferred to a basic crib sheet and two enhanced crib sheet variations.

In recent years the popularity of music and rhythm-based games has experienced tremendous growth. However almost all of these games require custom hardware to be used as input devices, and these devices control only one or two similar instruments. In this paper we describe One Man Band, a prototype video game for musical expression that uses novel 3D spatial interaction techniques using accelerometer-based motion controllers. One Man Band provides users with 3D gestural interfaces to control both the timing and sound of the music played, with both single and collaborative player modes. We further investigate the ability to detect different musical gestures without explicit selection of mode, giving the user the ability to seamlessly transition between instrument types with a single input device. A formal user study is then presented comparing the musical interface of One Man Band to that of Nintendo's Wii Music. Our results indicate that users generally preferred the interface of One Man Band over that of Wii Music. We also found that users desire to express their own ideas and have explicit control of the melodies created in music-based video games.

Existing work on menu techniques has shown linear menus to be less efficient and reliable for menuing tasks when compared to radial menus. With the rise in popularity of 3D spatial interaction in console gaming, such as the Nintendo Wii, it is important to determine whether the existing findings still hold true when using a 3D pointing device such as the Wii Controller. Linear menus were compared with two other menu techniques: radial menus and rotary menus. Effectiveness was measured through task completion time and the number of task errors. A subjective measure was also taken to determine participant preferences. Participants performed faster and made fewer errors when using the radial menu technique. Radial menus were also preferred by participants. These results indicate that radial menus are an effective menu technique when used with a 3D pointing device. This appears to agree with previous work regarding radial menus and indicates that the usage of radial menus in gaming applications should be investigated further.

In recent years the popularity of music and rhythm-based games has experienced tremendous growth. However, almost all of these games require custom hardware to be used as input devices, and these devices control only one or two similar instruments. In this paper we describe One Man Band, a prototype video game for musical expression that uses novel 3D spatial interaction techniques using accelerometer-based motion controllers. One Man Band provides users with 3D gestural interfaces to control both the timing and sound of the music played, with both single and collaborative player modes. We further investigate the ability to detect different musical gestures without explicit selection of mode, giving the user the ability to seamlessly transition between instrument types with a single input device.

3D User Interface (3DUI) design and development requires practitioners (designers and developers) to represent their ideas in representations designed for machine execution rather than natural representations, hampering development of effective 3DUIs. As such, Concept-Oriented Design (COD) was created as a theory of software development for both natural and executable design and development. Instantiated in the toolkit Chasm, Chasm is a natural, tiered, executable User Interface Description Language (UIDL) for 3DUIs resulting in improved understandability, as well as reduced complexity and reuse. Chasm's utility is shown through evaluations by domain experts, case studies of long-term use, and an analysis of spaces .

We present an online system for recognizing handwritten mathematical matrices in the context of an interactive computational tool called MathPaper. Automatic segmentation and recognition of multiple expressions are supported based on a spacing algorithm that leverages recognized symbol identities, sizes, and relative locations of individual symbols. Matrices with ellipses can be recognized and instantiated with non-ellipsis elements. Both well- and non-well-formed matrices can also be recognized. Matrix elements can be any general mathematical expressions including imbedded matrices. Our recognizer also addresses the poor column alignment problem of handwritten matrices, and allows for slight horizontal overlaps between elements in neighboring columns and different rows.

We present a study exploring the effect of positional offset between the user's interaction and display frame-of-reference in a surround-screen virtual environment (SSVE). In our experiment, users were asked to match a target color using a 3D color widget under three different display-interaction offset conditions: no offset (i.e., collocation), a three inch offset, and a two foot offset. Our results suggest that collocation of the display and interaction frames-of-reference may degrade accuracy in widget-based tasks and that collocation does not necessarily lead the user to spend more time on the task

We present a practical technique for using a writer-independent recognition engine to improve the accuracy and speed while reducing the training requirements of a writer-dependent symbol recognizer. Our writer-dependent recognizer uses a set of binary classifiers based on the AdaBoost learning algorithm, one for each possible pairwise symbol comparison. Each classifier consists of a set of weak learners, one of which is based on a writer-independent handwriting recognizer. During online recognition, we also use the n-best list of the writer-independent recognizer to prune the set of possible symbols and thus reduce the number of required binary classifications. In this paper, we describe the geometric and statistical features used in our recognizer and our all-pairs classification algorithm. We also present the results of experiments that quantify the effect incorporating a writer-independent recognition engine into a writer-dependent recognizer has on accuracy, speed, and user training time.

We present mathematical sketching , a novel, pen-based, modeless gestural interaction paradigm for mathematics problem solving. Mathematical sketching derives from the familiar pencil-and-paper process of drawing supporting diagrams to facilitate the formulation of mathematical expressions; however, with a mathematical sketch, users can also leverage their physical intuition by watching their hand-drawn diagrams animate in response to continuous or discrete parameter changes in their written formulas. Diagram animation is driven by implicit associations that are inferred, either automatically or with gestural guidance, from mathematical expressions, diagram labels, and drawing elements. The modeless nature of mathematical sketching enables users to switch freely between modifying diagrams or expressions and viewing animations. Mathematical sketching can also support computational tools for graphing, manipulating and solving equations; initial feedback from a small user group of our mathematical sketching prototype application, MathPad 2 , suggests that it has the potential to be a powerful tool for mathematical problem solving and visualization.

We present an experiment comparing double exponential smoothing and Kalman filter-based predictive tracking algorithms with derivative free measurement models. Our results show that the double exponential smoothers run approximately 135 times faster with equivalent prediction performance. The paper briefly describes the algorithms used in the experiment and discusses the results.

Presents a novel class of virtual reality input devices that combine pop-through buttons with 6-DOF trackers. Compared to similar devices that use conventional buttons, pop-through devices double the number of potential discrete interaction modes, since each button has two activation states corresponding to light and firm pressure. This additional state per button provides a foundation to address a range of shortcomings with conventional virtual environment (VE) input devices that includes reducing the physical dexterity required to perform interactions, reducing the cognitive complexity of some compound tasks and enabling the design of less obtrusive devices without sacrificing expressive power. Specifically, we present two novel input devices: the FingerSleeve was designed to be minimally obtrusive physically, whereas the TriggerGun was designed to be physically similar to, yet more functional than a conventional hand-held trigger device. Further, we present a set of novel navigation and interaction techniques that leverage the capabilities of our pop-through button devices to improve interaction quality and provide insight into harnessing the potential of pop-through buttons for other tasks. Finally, we discuss how we incorporated one of our devices into a real application.

Article Share on CavePainting: a fully immersive 3D artistic medium and interactive experience Authors: Daniel F. Keefe Brown Univ., Providence, RI Brown Univ., Providence, RIView Profile , Daniel Acevedo Feliz Brown Univ., Providence, RI Brown Univ., Providence, RIView Profile , Tomer Moscovich Brown Univ., Providence, RI Brown Univ., Providence, RIView Profile , David H. Laidlaw Brown Univ., Providence, RI Brown Univ., Providence, RIView Profile , Joseph J. LaViola Brown Univ., Providence, RI Brown Univ., Providence, RIView Profile Authors Info & Claims I3D '01: Proceedings of the 2001 symposium on Interactive 3D graphicsMarch 2001Pages 85–93https://doi.org/10.1145/364338.364370Published:01 March 2001Publication History 144citation2,008DownloadsMetricsTotal Citations144Total Downloads2,008Last 12 Months140Last 6 weeks22 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

This paper presents a set of interaction techniques for hands-free multi-scale navigation through virtual environments. We believe that hands-free navigation, unlike the majority of navigation techniques based on hand motions, has the greatest potential for maximizing the interactivity of virtual environments since navigation modes are offloaded from modal hand gestures to more direct motions of the feet and torso. Not only are the users' hands freed to perform tasks such as modeling, notetaking and object manipulation, but we also believe that foot and torso movements may inherently be more natural for some navigation tasks. The particular interactions that we developed include a leaning technique for moving small and medium distances, a foot-gesture controlled Step WIM that acts as a floor map for moving larger distances, and a viewing technique that enables a user to view a full 360 degrees in only a three-walled semi-immersive environment by subtly amplifying the mapping between th...