Diminished Reality (DR) modifies or removes elements of the perceived environment. We explored DR for attention support through diary studies, interviews, and speculative co-design with 15 university students with ADHD. Participants described a persistent gap between intention and attention, and the use of externalization strategies that encounter fundamental limits in physical reality. In co-design, rather than reasoning from visual operations, participants articulated which properties of stimuli they needed to perceive versus filter, revealing cases where visual diminishment does not reliably map to attentional diminishment. We contribute rich descriptive data on the attentional challenges and DR preferences of students with ADHD, and propose Attentional Diminishment, a sensitizing concept that reframes DR for attention support from visual outcomes to attentional effects.

Throughput is a widely used performance metric, combining speed and accuracy into a single measure, while reducing the effect of subjective speed–accuracy trade-offs. Despite its wide application in 2D steering tasks, its direct extension to 3D presents unique challenges since 3D trajectories exhibit higher variability, and perceptual–motor factors undermine existing formulations. Consequently, throughput has not been systematically adopted for evaluating steering in 3D virtual environments. In this paper, using a controlled virtual reality user study with a ring-and-wire task, we introduce and validate a novel throughput formulation for 3D steering based on the bivariate standard deviation of the trajectory for the effective width calculation. Our results show that this formulation provides smoother throughput values across subjective speed–accuracy differences and improves model fit compared to traditional approaches. This work advances our theoretical understanding of the Steering law in 3D contexts, provides researchers and practitioners with a robust evaluation method, and establishes a foundation for future studies of complex 3D trajectory interactions.

This paper presents an empirical evaluation of mouse-based interaction in seated Virtual Reality (VR) scenarios as an alternative to traditional VR controllers. We introduce Scaled Mouse, an interaction technique that shows the position of the real mouse in VR and maps standard mouse actions to 3D object manipulation using velocity-based scaling. We present two user studies assessing this method. In the first study, we compared a VR controller against our mouse-based method for manipulation tasks. Results show that the mouse consistently achieved higher rotation accuracy in 3D rotation and docking tasks than the controller, while maintaining comparable task completion times. Building on prior scaled interaction techniques, our second study investigated mouse interaction in a productivity-oriented scenario where users could combine 3D manipulation with a 2D adjustment panel. Scaled Mouse interaction was significantly faster and more accurate than the controller for 3D docking tasks and was rated significantly higher in usability and workload measures. Together, these findings establish the mouse as a viable and effective input device for seated VR contexts, offering both familiarity and accuracy advantages over controllers, particularly for professional and productivity-focused applications.

Constant-and Dual-Threshold Dwell keyboards

Transcription task Figure 1: Three participants taking part in the study: (a) completing the personality questionnaire, (b) performing the composition task, and (c) performing the transcription task.

Target selection is one of the most fundamental tasks in VR interaction systems. Prediction heuristics can provide users with a smoother interaction experience in this process. Our work aims to predict the ray landing pose for hand-based raycasting selection in Virtual Reality (VR) using a Long Short-Term Memory (LSTM)-based neural network with time-series data input of speed and distance over time from three different pose channels: hand, Head-Mounted Display (HMD), and eye. We first conducted a study to collect motion data from these three input channels and analyzed these movement behaviors. Additionally, we evaluated which combination of input modalities yields the optimal result. A second study validates raycasting across a continuous range of distances, angles, and target sizes. On average, our technique’s predictions were within 4.6° of the true landing Pose when 50% of the way through the movement. We compared our LSTM neural network model to a kinematic information model and further validated its generalizability in two ways: by training the model on one user’s data and testing on other users (cross-user) and by training on a group of users and testing on entirely new users (unseen users). Compared to the baseline and a previous kinematic method, our model increased prediction accuracy by a factor of 3.5 and 1.9, re spectively, when 40% of the way through the movement.

Steering involves continuous movement along constrained paths, well-studied in 2D. The extensions to 3D using the Ring-and-Wire and Ball-and-Tunnel tasks were often treated as interchangeable in previous work. In this paper, we directly compare these two tasks through a within-subjects user study (n = 18) with varying 3D path orientations. The results show that Ring-and-Wire significantly outperformed Ball-and-Tunnel, with 17.17% lower task time, 21.65% higher throughput, and 21.52% faster average speed. Participants also preferred Ring-and-Wire and reported lower workload. Visual ambiguity, especially near the tunnel’s rear surface, complicated spatial perception in the Ball-and-Tunnel task. We thus recommend that future studies choose 3D steering tasks carefully for experiments, as the two tasks are not interchangeable.

Any investigation of learning unfamiliar text entry systems is affected by the need to train participants on multiple new components simultaneously, such as novel interaction techniques and layouts. The Guided Evaluation Method (GEM) addresses this challenge by bypassing the need to learn layout-specific skills for text entry. However, a gap remains as the GEM's performance has not been assessed in situations where users are unfamiliar with the interaction technique involved, here eye-gaze-based dwell. To address this, we trained participants on only the eye-gaze-based interaction technique over eight days with QWERTY and then evaluated their performance on the OPTI layout with the GEM. Results showed that the unfamiliar OPTI layout outperformed QWERTY, with QWERTY's speed aligning with previous findings, suggesting that interaction technique expertise significantly impacts performance outcomes. Importantly, we also identified that for scenarios where the familiarity with the involved interaction technique(s) is the same, the GEM analyzes the performance of keyboard layouts effectively and quickly identifies the best option.

Recent advances in eye-tracking technology have positioned gaze as an efficient and intuitive input method for Virtual Reality (VR), offering a natural and immersive user experience. As a result, gaze input is now leveraged for fundamental interaction tasks such as selection, manipulation, crossing, and steering. Although several studies have modeled user steering performance across various path characteristics and input methods, our understanding of gaze-based steering in VR remains limited. This gap persists because the unique qualities of eye movements-involving rapid, continuous motions-and the variability in eye-tracking make findings from other input modalities nontransferable to a gaze-based context, underscoring the need for a dedicated investigation into gaze-based steering behaviors and performance. To bridge this gap, we present two user studies to explore and model gaze-based steering. In the first one, user behavior data are collected across various path characteristics and eye-tracking conditions. Based on this data, we propose four refined models that extend the classic Steering Law to predict users' movement time in gaze-based steering tasks, explicitly incorporating the impact of tracking quality. The best-performing model achieves an adjusted R2 of 0.956, corresponding to a 16% improvement in movement time prediction. This model also yields a substantial reduction in AIC (from 1550 to 1132) and BIC (from 1555 to 1142), highlighting improved model quality and better balance between goodness of fit and model complexity. Finally, data from a second study with varied settings, such as a different eye-tracking sampling rate, illustrate the strong robustness and predictability of our models. Finally, we present scenarios and applications that demonstrate how our models can be used to design enhanced gaze-based interactions in VR systems.

Navigating multiscale virtual environments necessitates an interaction method to travel across different levels of scale (LoS). Prior research has studied various techniques that enable users to seamlessly adjust their scale to navigate between different LoS based on specific user contexts. We introduce a scroll-based scale control method optimized for exocentric navigation, targeted at scenarios where speed and accuracy in continuous scaling are crucial. We pinpoint the challenges of scale control in settings with multiple LoS and evaluate how distinct designs of scaling techniques influence navigation performance and usability. Through a user study, we investigated two pivotal elements of a scaling technique: the input method and the scaling center. Our findings indicate that our scroll-based input method significantly reduces task completion time and error rate and enhances efficiency compared to the most frequently used bi-manual method. Moreover, we found that the choice of scaling center affects the ease of use of the scaling method, especially when paired with specific input methods.

This paper reviews the literature on the visualization of individual mobility data, with a focus on thematic integration. It emphasizes the importance of visualization in understanding mobility patterns within a population and how it helps mobility experts to address domain-specific questions. We analyze 38 papers published between 2010 and 2024 in GIS and VIS venues that describe visualizations of multidimensional data related to individual movements in urban environments, concentrating on individual mobility rather than traffic data. Our primary aim is to report advances in interactive visualization for individual mobility analysis, particularly regarding the representation of thematic information about people’s motivations for mobility. Our findings indicate that the thematic dimension is only partially represented in the literature, despite its critical significance in transportation. This gap often stems from the challenge of identifying data sources that inherently provide this information, necessitating visualization designers and developers to navigate multiple, heterogeneous data sources. We identify the strengths and limitations of existing visualizations and suggest potential research directions for the field. • Analysis of 38 visual analytics solutions designed to aid the exploration of individual mobility data across spatial, temporal, and thematic dimensions. • An analysis of how visualization tools incorporate the theme dimension and which thematic properties are addressed. • Analysis of how the subject is approached within GIS and VIS communities.

Emergency response to large-scale disasters is often supported with multimedia from social media. However, while these features are common in everyday video calls, the complex needs of 911 and other systems make it difficult to directly incorporate these features. We assess an ME911 (Multimedia-Enabled 911) app to understand how the design will need to deviate from common norms and how callers will respond to those non-standard choices. We expand the role of 911 call taker control over emergency situations to the calling interface while incorporating key features like map-based location finding. Participants’ experiences in mock emergencies show the non-standard design helps callers in the unfamiliar setting of emergency calling yet it also causes confusion and delays. We find the need for emergency-specific deviations from design norms is supported by participant feedback. We discuss how broader system changes will support callers to use these non-standard designs during emergencies.

In Fitts’ law studies to investigate pointing, throughput is used to characterize the performance of input devices and users, which is claimed to be independent of task difficulty or the user’s subjective speed-accuracy bias. While throughput has been recognized as a useful metric for target-pointing tasks, the corresponding formulation for path-steering tasks and its evaluation have not been thoroughly examined in the past. In this paper, we conducted three experiments using linear, circular, and sine-wave path shapes to propose and investigate a novel formulation for the effective parameters and the throughput of steering tasks. Our results show that the effective width substantially improves the fit to data with mixed speed-accuracy biases for all task shapes. Effective width also smoothed out the throughput across all biases, while the usefulness of the effective amplitude depended on the task shape. Our study thus advances the understanding of user performance in trajectory-based tasks.

Visual guides help to align strokes and raise accuracy in Virtual Reality (VR) sketching tools. Automatic guides that appear at relevant sketching areas are convenient to have for a seamless sketching with a guide. We explore guides that exploit eye-tracking to render them adaptive to the user’s visual attention. EyeGuide and EyeConGuide cause visual grid fragments to appear spatially close to the user’s intended sketches, based on the information of the user’s eye-gaze direction and the 3D position of the hand. Here we evaluated the techniques in two user studies across simple and complex sketching objectives in VR. The results show that gaze-based guides have a positive effect on sketching accuracy, perceived usability and preference over manual activation in the tested tasks. Our research contributes to integrating gaze-contingent techniques for assistive guides and presents important insights into multimodal design applications in VR.

In current graphical user interfaces, there exists a (typically unavoidable) end-to-end latency from each pointing-device movement to its corresponding cursor response on the screen, which is known to affect user performance in target selection, e.g., in terms of movement time (MT). Previous work also reported that a long latency increases MTs in path-steering tasks, but the quantitative relationship between latency and MT had not been previously investigated for path-steering. In this work, we derive models to predict MTs for path-steering and evaluate them with five tasks: goal crossing as a preliminary task for model derivation, linear-path steering, circular-path steering, narrowing-path steering, and steering with target pointing. The results show that the proposed models yielded an adjusted R2 > 0.94, with lower AICs and smaller cross-validation RMSEs than the baseline models, enabling more accurate prediction of MTs.

In Virtual Reality (VR) applications, interacting with distant objects relies heavily on mid-air object manipulation. Yet, the inherent distance between the user and the object often restricts movement precision. This paper introduces the Signed Distance Field (SDF) method for mid-air object manipulation and combines it with the ray casting interaction technique to investigate its effect on user performance and user experience. To increase movement accuracy, we leverage the speed-accuracy trade-off to dynamically adjust object manipulation speed based on the SDF algorithm’s output. Our study with 18 participants examines the effects of SDF across three different tasks with different complexity. Our results showed that ray casting with SDF reduces the number of errors in complex tasks without slowing down the participants and improves the user experience. We hope that our proposed assistive system, designed for tasks and applications, can be used as an interaction technique to enable more accurate manipulation of distant objects in fields like surgical planning, architecture, and games.

Designing adaptive documents that are visually appealing across various devices and for diverse viewers is a challenging task. This is due to the wide variety of devices and different viewer requirements and preferences. Alterations to a document’s content, style, or layout often necessitate numerous adjustments, potentially leading to a complete layout redesign. We introduce FlexDoc, a framework for creating and consuming documents that seamlessly adapt to different devices, author, and viewer preferences and interactions. It eliminates the need to manually create multiple document layouts, as FlexDoc enables authors to define desired document properties using templates and employs both discrete and continuous optimization in a novel comprehensive optimization process, which leverages automatic text summarization and image carving techniques to adapt both layout and content during consumption dynamically. Further, we demonstrate FlexDoc in real-world scenarios.

Previous studies on Eye-Hand Coordination Training (EHCT) focused on the comparison of user motor performance across different hardware with cross-sectional studies. In this paper, we compare user motor performance with an EHCT setup in Augmented Reality (AR), Virtual Reality (VR), and on a 2D touchscreen display in a longitudinal study. Through a ten-day user study, we thoroughly analyzed the motor performance of twenty participants with five task instructions focusing on speed, error rate, accuracy, precision, and none. As a novel evaluation criterion, we also analyzed the participants’ performance in terms of effective throughput. The results showed that each task instruction has a different effect on one or more psychomotor characteristics of the trainee, which highlights the importance of personalized training programs. Regarding different display technologies, the majority of participants could see more improvement in VR than in 2D or AR. We also identified that effective throughput is a good candidate for monitoring overall motor performance progress in EHCT systems.

Current visualization research has identified the potential of more immersive settings for data exploration, leveraging VR and AR technologies. To explore how a traditional visualization system could be adapted into an immersive framework, and how it could benefit from this, we decided to revisit a landmark paper presented ten years ago at IEEE VIS. TaxiVis, by Ferreira et al., enabled interactive spatio-temporal querying of a large dataset of taxi trips in New York City. Here, we reimagine how TaxiVis’ functionalities could be implemented and extended in a 3D immersive environment. Among the unique features we identify as being enabled by the Immersive TaxiVis prototype are alternative uses of the additional visual dimension, a fully visual 3D spatio-temporal query framework, and the opportunity to explore the data at different scales and frames of reference. By revisiting the case studies from the original paper, we demonstrate workflows that can benefit from this immersive perspective. Through reporting on our experience, and on the vision and reasoning behind our design decisions, we hope to contribute to the debate on how conventional and immersive visualization paradigms can complement each other and on how the exploration of urban datasets can be facilitated in the coming years.

To understand current practice and explore the potential for more comprehensive evaluations of 3D immersive sketching, drawing, and painting, we present a survey of evaluation methodologies used in existing 3D sketching research, a breakdown and discussion of important phases (sub-tasks) in the 3D sketching process, and a framework that suggests how these factors can inform evaluation strategies in future 3D sketching research. Existing evaluations identified in the survey are organized and discussed within three high-level categories: 1) evaluating the 3D sketching activity, 2) evaluating 3D sketching tools, and 3) evaluating 3D sketching artifacts. The new framework suggests targeting evaluations to one or more of these categories and identifying relevant user populations. In addition, building upon the discussion of the different phases of the 3D sketching process, the framework suggests to evaluate relevant sketching tasks, which may range from low-level perception and hand movements to high-level conceptual design. Finally, we discuss limitations and challenges that arise when evaluating 3D sketching, including a lack of standardization of evaluation methods and multiple, potentially conflicting, ways to evaluate the same task and user interface usability; we also identify opportunities for more holistic evaluations. We hope the results can contribute to accelerating research in this domain and, ultimately, broad adoption of immersive sketching systems.

Recent work in immersive analytics suggests benefits for systems that support work across both 2D and 3D data visualizations, i.e., cross-virtuality analytics systems. Here, we introduce HybridAxes, an immersive visual analytics system that enables users to conduct their analysis either in 2D on desktop monitors or in 3D within an immersive AR environment - while enabling them to seamlessly switch and transfer their graphs between modes. Our user study results show that the cross-virtuality sub-systems in HybridAxes complement each other well in helping the users in their data-understanding journey. We show that users preferred using the AR component for exploring the data, while they used the desktop to work on more detail-intensive tasks. Despite encountering some minor challenges in switching between the two virtuality modes, users consistently rated the whole system as highly engaging, user-friendly, and helpful in streamlining their analytics processes. Finally, we present suggestions for designers of cross-virtuality visual analytics systems and identify avenues for future work.

The spread of COVID-19 has encouraged the practice of using video conferencing for family doctor appointments. Existing applications and off-the-shelf devices face challenges in dealing with capturing the correct view of patients' bodies and supporting ease of use. We created Dr.’s Eye, a video conferencing prototype to support varying types of body exams in home settings. With our prototype, we conducted a study with participants using mock appointments to understand the simultaneous use of the camera and display and to get insights into the issues that might arise in real doctor appointments. Results show the benefits of providing more flexibility with a decoupled camera and display, and privacy protection by limiting the camera view. Yet, challenges remain in maneuvering two devices, presenting feedback for the camera view, coordinating camera work between the participant and the examiner, and reluctance towards showing private body regions. This inspires future research on how to design a video system for doctor appointments.

When working in Virtual Reality (VR), the user’s performance is affected by how the user holds the input device (e.g., controller), typically using either a precision or a power grip. Previous work examined these grip styles for 3D pointing at targets at different depths in peripersonal space and found that participants had a lower error rate with the precision grip but identified no difference in movement speed, throughput, or interaction with target depth. Yet, this previous experiment was potentially affected by tracking differences between devices. This paper reports an experiment that partially replicates and extends the previous study by evaluating the effect of grip style on the 3D selection of nearby targets with the same device. Furthermore, our experiment re-investigates the effect of the vergence-accommodation conflict (VAC) present in current stereo displays on 3D pointing in peripersonal space. Our results show that grip style significantly affects user performance. We hope that our results are useful for researchers and designers when creating virtual environments.

State-of-the-art Virtual Reality (VR) and Augmented Reality (AR) headsets rely on singlefocal stereo displays. For objects away from the focal plane, such displays create a vergence-accommodation conflict (VAC), potentially degrading user interaction performance. In this paper, we study how the VAC affects pointing at targets within arm's reach with virtual hand and raycasting interaction in current stereo display systems. We use a previously proposed experimental methodology that extends the ISO 9241–411:2015 multi-directional selection task to enable fair comparisons between selecting targets in different display conditions. We conducted a user study with eighteen participants and the results indicate that participants were faster and had higher throughput in the constant VAC condition with the virtual hand. We hope that our results enable designers to choose more efficient interaction methods in virtual environments.

Viewpoint transitions have been shown to improve users' spatial orientation and help them build a cognitive map when they are navigating an unfamiliar virtual environment. Previous work has investigated transitions in single-scale virtual environments, focusing on trajectories and continuity. We extend this work with an in-depth investigation of transition techniques in multiscale virtual environments (MVEs). We identify challenges in navigating MVEs with nested structures and assess how different transition techniques affect spatial understanding and usability. Through two user studies, we investigated transition trajectories, interactive control of transition movement, and speed modulation in a nested MVE. We show that some types of viewpoint transitions enhance users' spatial awareness and confidence in their spatial orientation and reduce the need to revisit a target point of interest multiple times.

The vergence-accommodation conflict (VAC) limits user performance in current Virtual Reality (VR) systems. In this paper, we investigate the effects of the VAC in a single-focal VR system using three experimental conditions: with no VAC, with a constant VAC, and with a varying VAC. Previous work in this area had yielded conflicting results, so we decided to re-investigate this issue. Eighteen participants performed an ISO 9241:411 task in a study that closely replicates previous work, except that the angle of the task space was rotated 20 degrees downward, to make the task less fatiguing to perform, which addresses a potential confound in previous work. We found that the varying VAC condition had worse performance than the other conditions, which indicates that the contrasting results in previous work were very likely due to biomechanical factors. We hope that our work contributes to the understanding of the influence of the VAC in VR systems and potential strategies for improving user experience and performance in immersive virtual environments.

Previous work hypothesized that for Virtual Reality (VR) and Augmented Reality (AR) displays a mismatch between disparities and optical focus cues, known as the vergence and accommodation conflict (VAC), affects depth perception and thus limits user performance in 3D selection tasks within arm’s reach (peri-personal space). To investigate this question, we built a multifocal stereo display, which can eliminate the influence of the VAC for pointing within the investigated distances. In a user study, participants performed a virtual hand 3D selection task with targets arranged laterally or along the line of sight, with and without a change in visual depth, in display conditions with and without the VAC. Our results show that the VAC influences 3D selection performance in common VR and AR stereo displays and that multifocal displays have a positive effect on 3D selection performance with a virtual hand.

Augmented/Virtual reality and video-based media play a vital role in the digital learning revolution to train novices in spatial tasks. However, creating content for these different media requires expertise in several fields. We present EditAR, a unified authoring, and editing environment to create content for AR, VR, and video based on a single demonstration. EditAR captures the user’s interaction within an environment and creates a digital twin, enabling users without programming backgrounds to develop content. We conducted formative interviews with both subject and media experts to design the system. The prototype was developed and reviewed by experts. We also performed a user study comparing traditional video creation with 2D video creation from 3D recordings, via a 3D editor, which uses freehand interaction for in-headset editing. Users took 5 times less time to record instructions and preferred EditAR, along with giving significantly higher usability scores.

Previous work has shown that the mismatch between disparity and optical focus cues, i.e., the vergence and accommodation conflict (VAC), affects virtual hand selection in immersive systems. To investigate if the VAC also affects distal pointing with ray casting, we ran a user study with an ISO 9241:411 multidirectional selection task where participants selected 3D targets with three different VAC conditions, no VAC, i.e., targets placed roughly at 75 cm, which matches the focal plane of the VR headset, constant VAC, i.e., at 400 cm from the user, and varying VAC, where the depth distance of targets changed between 75 cm and 400 cm. According to our results, the varying VAC condition requires the most time and decreases the throughput performance of the participants. It also takes longer for users to select targets in the constant VAC condition than without the VAC. Our results show that in distal pointing placing objects at different depth planes has detrimental effect on the user performance.

In eye-gaze-based selection, dwell suffers from several issues, e.g., the Midas Touch problem. Here we investigate saccade-based selection techniques as an alternative to dwell. First, we designed a novel user interface (UI) for Actigaze and used it with (goal-crossing) saccades for confirming the selection of small targets (i.e., < 1.5-2°). We compared it with three other variants of Actigaze (with button press, dwell, and target reverse crossing) and two variants of target magnification (with button press and dwell). Magnification-dwell exhibited the most promising performance. For Actigaze, goal-crossing was the fastest option but suffered the most errors. We then evaluated goal-crossing as a primary selection technique for normal-sized targets (≥ 2°) and implemented a novel UI for such interaction. Results revealed that dwell achieved the best performance. Yet, we identified goal-crossing as a good compromise between dwell and button press. Our findings thus identify novel options for gaze-only interaction.

Models of lassoing time to select multiple square icons exist, but realistic lasso tasks also typically involve encircling non-rectangular objects. Thus, it is unclear if we can apply existing models to such conditions where, e.g., the width of the path that users want to steer through changes dynamically or step-wise. In this work, we conducted two experiments where the objects were non-rectangular, with path widths that narrowed or widened, smoothly or step-wise. The results showed that the baseline models for pen-steering movements (the steering and crossing law models) fitted the timing data well, but also that segmenting width-changing areas led to significant improvements. Our work enables the modeling of novel UIs requiring continuous strokes, e.g., for grouping icons.

Fitts' law and throughput based on effective measures are two mathematical models frequently used to analyze human motor performance in a standardized pointing task, e.g., to compare the performance of input and output devices. Even though pointing has been deeply studied in 2D, it is not well understood how different task execution strategies affect throughput in pointing in 3D virtual environments. In this work, we examine the effective throughput measure, claimed to be invariant to task execution strategies, in Virtual Reality (VR) systems with three such strategies, "as fast, as precise, and as fast and as precise as possible" for ray casting and virtual hand interaction, by re-analyzing data from a 3D pointing ISO 9241-411 study. Results show that effective throughput is not invariant for different task execution strategies in VR, which also matches a more recent 2D result. Normalized speed vs. accuracy curves also did not fit the data. We thus suggest that practitioners, developers, and researchers who use MacKenzie's effective throughput formulation should consider our findings when analyzing 3D user pointing performance in VR systems.

The design space for user interfaces for Immersive Analytics applications is vast. Designers can combine navigation and manipulation to enable data exploration with ego- or exocentric views, have the user operate at different scales, or use different forms of navigation with varying levels of physical movement. This freedom results in a multitude of different viable approaches. Yet, there is no clear understanding of the advantages and disadvantages of each choice. Our goal is to investigate the affordances of several major design choices, to enable both application designers and users to make better decisions. In this article, we assess two main factors, exploration mode and frame of reference, consequently also varying visualization scale and physical movement demand. To isolate each factor, we implemented nine different conditions in a Space-Time Cube visualization use case and asked 36 participants to perform multiple tasks. We analyzed the results in terms of performance and qualitative measures and correlated them with participants' spatial abilities. While egocentric room-scale exploration significantly reduced mental workload, exocentric exploration improved performance in some tasks. Combining navigation and manipulation made tasks easier by reducing workload, temporal demand, and physical effort.

Augmented reality (AR) is an efficient form of delivering spatial information and has great potential for training workers. However, AR is still not widely used for such scenarios due to the technical skills and expertise required to create interactive AR content. We developed ProcessAR, an AR-based system to develop 2D/3D content that captures subject matter expert’s (SMEs) environment-object interactions in situ. The design space for ProcessAR was identified from formative interviews with AR programming experts and SMEs, alongside a comparative design study with SMEs and novice users. To enable smooth workflows, ProcessAR locates and identifies different tools/objects through computer vision within the workspace when the author looks at them. We explored additional features such as embedding 2D videos with detected objects and user-adaptive triggers. A final user evaluation comparing ProcessAR and a baseline AR authoring environment showed that, according to our qualitative questionnaire, users preferred ProcessAR.

Reverse engineering (RE) of user interfaces (UIs) plays an important role in software evolution. However, the large diversity of UI technologies and the need for UIs to be resizable make this challenging. We propose ReverseORC, a novel RE approach able to discover diverse layout types and their dynamic resizing behaviours independently of their implementation, and to specify them by using OR constraints. Unlike previous RE approaches, ReverseORC infers flexible layout constraint specifications by sampling UIs at different sizes and analyzing the differences between them. It can create specifications that replicate even some non-standard layout managers with complex dynamic layout behaviours. We demonstrate that ReverseORC works across different platforms with very different layout approaches, e.g., for GUIs as well as for the Web. Furthermore, it can be used to detect and fix problems in legacy UIs, extend UIs with enhanced layout behaviours, and support the creation of flexible UI layouts.

Fitts' law and the associated throughput measure characterize user pointing performance in virtual reality (VR) training systems and simulators well. Yet, pointing performance can be affected by the feedback users receive from a VR application. This work examines the effect of the pitch of auditory error feedback on user performance in a Fitts' task through a distributed experiment. In our first study, we used middle- and high-frequency sound feedback and demonstrated that high-pitch error feedback significantly decreases user performance in terms of time and throughput. In the second study, we used adaptive sound feedback, where we increased the frequency with the error rate, while asking subjects to execute the task “as fast/as precise/as fast and precise as possible”. Results showed that adaptive sound feedback decreases the error rate for “as fast as possible” task execution without affecting the time. The results can be used to enhance and design various VR systems.

In this work, we evaluate two standard interaction techniques for Immersive Analytics environments: virtual hands, with actions such as grabbing and stretching, and virtual ray pointers, with actions assigned to controller buttons. We also consider a third option: seamlessly integrating both modes and allowing the user to alternate between them without explicit mode switches. Easy-to-use interaction with data visualizations in Virtual Reality enables analysts to intuitively query or filter the data, in addition to the benefit of multiple perspectives and stereoscopic 3D display. While many VR-based Immersive Analytics systems employ one of the studied interaction modes, the effect of this choice is unknown. Considering that each has different advantages, we compared the three conditions through a controlled user study in the spatio-temporal data domain. We did not find significant differences between hands and ray-casting in task performance, workload, or interactivity patterns. Yet, 60% of the participants preferred the mixed mode and benefited from it by choosing the best alternative for each low-level task. This mode significantly reduced completion times by 23% for the most demanding task, at the cost of a 5% decrease in overall success rates.

Object manipulation techniques in immersive virtual environments are either inaccurate or slow. We present a novel technique, PinNPivot, where pins are used to constrain 1DOF/2DOF/3DOF rotations. It also supports 6DOF manipulation and 3DOF translation. A comparison with three existing techniques shows that PinNPivot is significantly more accurate and faster.

Walking has always been considered as the gold standard for navigation in Virtual Reality research. Though full rotation is no longer a technical challenge, physical translation is still restricted through limited tracked areas. While rotational information has been shown to be important, the benefit of the translational component is still unclear with mixed results in previous work. To address this gap, we conducted a mixed-method experiment to compare four levels of translational cues and control: none (using the trackpad of the HTC Vive controller to translate), upper-body leaning (sitting on a "NaviChair", leaning the upper-body to locomote), whole-body leaning/stepping (standing on a platform called NaviBoard, leaning the whole body or stepping one foot off the center to navigate), and full translation (physically walking). Results showed that translational cues and control had significant effects on various measures including task performance, task load, and simulator sickness. While participants performed significantly worse when they used a controller with no embodied translational cues, there was no significant difference between the NaviChair, NaviBoard, and actual walking. These results suggest that translational body-based motion cues and control from a low-cost leaning/stepping interface might provide enough sensory information for supporting spatial updating, spatial awareness, and efficient locomotion in VR, although future work will need to investigate how these results might or might not generalize to other tasks and scenarios.

OR-constrained (ORC) graphical user interface layouts unify conventional constraint-based layouts with flow layouts, which enables the definition of flexible layouts that adapt to screens with different sizes, orientations, or aspect ratios with only a single layout specification. Unfortunately, solving ORC layouts with current solvers is time-consuming and the needed time increases exponentially with the number of widgets and constraints. To address this challenge, we propose ORCSolver, a novel solving technique for adaptive ORC layouts, based on a branch-and-bound approach with heuristic preprocessing. We demonstrate that ORCSolver simplifies ORC specifications at runtime and our approach can solve ORC layout specifications efficiently at near-interactive rates.

Auto-correction is a standard feature of mobile text entry. While the performance of state-of-the-art auto-correct methods is usually relatively high, any errors that occur are cumbersome to repair, interrupt the flow of text entry, and challenge the user's agency over the process. In this paper, we describe a system that aims to automatically identify and repair auto-correction errors. This system comprises a multi-modal classifier for detecting auto-correction errors from brain activity, eye gaze, and context information, as well as a strategy to repair such errors by replacing the erroneous correction or suggesting alternatives. We integrated both parts in a generic Android component and thus present a research platform for studying self-repairing end-to-end systems. To demonstrate its feasibility, we performed a user study to evaluate the classification performance and usability of our approach.

Previous research on eye-hand coordination training systems has investigated user performance on a wall, 2D touchscreens, and in Virtual Reality (VR). In this paper, we designed an eye-hand coordination reaction test to investigate and compare user performance in three different virtual environments (VEs) – VR, Augmented Reality (AR), and a 2D touchscreen. VR and AR conditions also included two feedback conditions – mid-air and passive haptics. Results showed that compared to AR, participants were significantly faster and made fewer errors both in 2D and VR. However, compared to VR and AR, throughput performance of the participants was significantly higher in the 2D touchscreen condition. No significant differences were found between the two feedback conditions. The results show the importance of assessing precision and accuracy in eye-hand coordination training and suggest that it is currently not advisable to use AR headsets in such systems.

A relatively recent application area for Virtual Reality (VR) systems is sports training and user performance assessment. One of these applications is eye-hand coordination training systems (EHCTSs). Previous research identified that VR-based training systems have great potential for EHCTSs. While previous work investigated 3D targets on a 2D plane, here we aim to study full 3D movements and extend the application of throughput analysis to EHCTSs. We conducted two user studies to investigate how user performance is affected by different target arrangements, feedback conditions, and handedness in VR-based EHCTSs. In the first study, we explored handedness as well as vertical and horizontal target arrangements, and showed that user performance increases with the dominant hand and a vertical target plane. In the second study, we investigated different combinations of visual and haptic feedback and how they affect user performance with different target and cursor sizes. Results illustrate that haptic feedback did not increase user performance when it is added to visual feedback. Our results inform the creation of better EHCTSs with mid-air VR systems.

Virtual environments with a wide range of scales are becoming commonplace in Virtual Reality applications. Methods to control locomotion parameters can help users explore such environments more easily. For multi-scale virtual environments, point-and-teleport locomotion with a well-designed distance control method can enable mid-air teleportation, which makes it competitive to flying interfaces. Yet, automatic distance control for point-and-teleport has not been studied in such environments. We present a new method to automatically control the distance for point-and-teleport. In our first user study, we used a solar system environment to compare three methods: automatic distance control for point-and-teleport, manual distance control for point-and-teleport, and automatic speed control for flying. Results showed that automatic control significantly reduces overshoot compared with manual control for point-and-teleport, but the discontinuous nature of teleportation made users prefer flying with automatic speed control. We conducted a second study to compare automatic-speed-controlled flying and two versions of our teleportation method with automatic distance control, one incorporating optical flow cues. We found that point-and-teleport with optical flow cues and automatic distance control was more accurate than flying with automatic speed control, and both were equally preferred to point-and-teleport without the cues.

Navigation and selection are critical in very large virtual environments, such as a model of a whole city. In practice, many VR applications require both of these modalities to work together. We compare different combinations of two navigation and two selection methods in VR on selection tasks involving distant targets in a user study. The aim of our work is to discover the trade-off between navigation and selection techniques and to identify which combination leads to better interaction performance in large virtual environments. The results showed that users could complete the task faster with the fly/drive method and traveled less, compared to the teleportation method. Additionally, raycasting exhibited a better performance in terms of time and (less) distance traveled, however, it significantly increased the error rate for the selection of targets.

Mobile users rely on typing assistant mechanisms such as prediction and autocorrect. Previous studies on mobile keyboards showed decreased performance for heavy use of word prediction, which identifies a need for more research to better understand the effectiveness of predictive features for different users. Our work aims at such a better understanding of user interaction with autocorrections and the prediction panel while entering text, in particular when these approaches fail. We present a crowd-sourced mobile text entry study with 170 participants. Our mobile web application simulates autocorrection and word prediction to capture user behaviours around these features. We found that using word prediction saves an average of 3.43 characters per phrase but also adds an average of two seconds compared to actually typing the word, resulting in a negative effect on text entry speed. We also identified that the time to fix wrong autocorrections is on average 5.5 seconds but that autocorrection does not have a significant effect on typing speed.

The “crossing time” to pass between objects in lassoing tasks is predicted by Fitts’ law. When an unwanted object, or obstacle , intrudes into the user’s path, users curve the stroke to avoid hitting that obstacle. We empirically show that, in the presence of an obstacle, modified Fitts models for pointing with obstacle avoidance can significantly improve the prediction accuracy of movement time compared with standard Fitts’ law. Yet, we also found that when an object is (only) close to the crossing path, i.e., a distractor , users still curve their stroke, even though the object does not intrude. We tested the effects of distractor proximity and length. While the crossing motion is modified by a nearby distractor, our results also identify that overall its effect on crossing times was small, and thus Fitts’ law can still be applied safely with distractors.

A Space-Time Cube enables analysts to clearly observe spatio-temporal features in movement trajectory datasets in geovisualization. However, its general usability is impacted by a lack of depth cues, a reported steep learning curve, and the requirement for efficient 3D navigation. In this work, we investigate a Space-Time Cube in the Immersive Analytics domain. Based on a review of previous work and selecting an appropriate exploration metaphor, we built a prototype environment where the cube is coupled to a virtual representation of the analyst's real desk, and zooming and panning in space and time are intuitively controlled using mid-air gestures. We compared our immersive environment to a desktop-based implementation in a user study with 20 participants across 7 tasks of varying difficulty, which targeted different user interface features. To investigate how performance is affected in the presence of clutter, we explored two scenarios with different numbers of trajectories. While the quantitative performance was similar for the majority of tasks, large differences appear when we analyze the patterns of interaction and consider subjective metrics. The immersive version of the Space-Time Cube received higher usability scores, much higher user preference, and was rated to have a lower mental workload, without causing participants discomfort in 25-minute-long VR sessions.

Virtual Reality (VR) headsets have made immersive 3D drawing available to the general public. However, compared to 2D drawing, the presence of an additional dimension makes sketching in VR challenging, since creating precise strokes that are positioned as intended in all three dimensions imposes higher demands on the users' perception, motor and spatial skills. Another challenge users face is creating accurate shapes in which strokes are positioned correctly relative to previous ones, as they may need to use different views to plan their next hand movement. In this paper, we analyze the behaviours of users with different spatial abilities while drawing in VR. Our results indicate that there are different types of behaviours that affect different aspects of the sketches. We also found that the user's spatial ability affects the shape of the drawing, but not the line precision. Finally, we give recommendations for designing 3D drawing interfaces.

We propose a novel approach for constraint-based graphical user interface (GUI) layout based on OR-constraints (ORC) in standard soft/hard linear constraint systems. ORC layout unifies grid layout and flow layout, supporting both their features as well as cases where grid and flow layouts individually fail. We describe ORC design patterns that enable designers to safely create flexible layouts that work across different screen sizes and orientations. We also present the ORC Editor, a GUI editor that enables designers to apply ORC in a safe and effective manner, mixing grid, flow and new ORC layout features as appropriate. We demonstrate that our prototype can adapt layouts to screens with different aspect ratios with only a single layout specification, easing the burden of GUI maintenance. Finally, we show that ORC specifications can be modified interactively and solved efficiently at runtime.

We present a novel, multilayer interaction approach that enables state transitions between spatially above-screen and 2D on-screen feedback layers. This approach supports the exploration of haptic features that are hard to simulate using rigid 2D screens. We accomplish this by adding a haptic layer above the screen that can be actuated and interacted with (pressed on) while the user interacts with on-screen content using pen input. The haptic layer provides variable firmness and contour feedback, while its membrane functionality affords additional tactile cues like texture feedback. Through two user studies, we look at how users can use the layer in haptic exploration tasks, showing that users can discriminate well between different firmness levels, and can perceive object contour characteristics. Demonstrated also through an art application, the results show the potential of multilayer feedback to extend on-screen feedback with additional widget, tool and surface properties, and for user guidance.

The limitations of stereo display systems affect depth perception, e.g., due to the vergence-accommodation conflict or diplopia. We performed three studies to understand how stereo display deficiencies impact 3D pointing for targets in front of a screen and close to the user, i.e., in peripersonal space. Our first two experiments compare movements with and without a change in visual depth for virtual respectively physical targets. Results indicate that selecting targets along the depth axis is slower and has less throughput for virtual targets, while physical pointing demonstrates the opposite result. We then propose a new 3D extension for Fitts' law that models the effect of stereo display deficiencies. Next, our third experiment verifies the model and measures more broadly how the change in visual depth between targets affects pointing performance in peripersonal space and confirms significant effects on time and throughput. Finally, we discuss implications for 3D user interface design.

Lassoing objects is a basic function in illustration software and presentation tools. Yet, for many common object arrangements lassoing is sometimes time-consuming to perform and requires precise pen operation. In this work, we studied lassoing movements in a grid of objects similar to icons. We propose a quantitative model to predict the time to lasso such objects depending on the margins between icons, their sizes, and layout, which all affect the number of stopping and crossing movements. Results of two experiments showed that our models predict fully and partially constrained movements with high accuracy. We also analyzed the speed profiles and pen stroke trajectories and identified deeper insights into user behaviors, such as that an unconstrained area can induce higher movement speeds even in preceding path segments.

Speech is increasingly being used as a method for text entry, especially on commercial mobile devices such as smartphones. While automatic speech recognition has seen great advances, factors like acoustic noise, differences in language or accents can affect the accuracy of speech dictation for mobile text entry. There has been some research on interfaces that enable users to intervene in the process, by correcting speech recognition errors. However, there is currently little research that investigates the effect of Automatic Speech Recognition (ASR) metrics, such as word error rate, on human performance and usability of speech recognition correction interfaces for mobile devices. This research explores how word error rates affect the usability and usefulness of touch-based speech recognition correction interfaces in the context of mobile device text entry.

We present Plane, Ray, and Point, a set of interaction techniques that utilizes shape constraints to enable quick and precise object alignment and manipulation in virtual reality. Users create the three types of shape constraints, Plane, Ray, and Point, by using symbolic gestures. The shape constraints are used like scaffoldings and limit and guide the movement of virtual objects that collide or intersect with them. The same set of gestures can be performed with the other hand, which allow users to further control the degrees of freedom for precise and constrained manipulation. The combination of shape constraints and bimanual gestures yield a rich set of interaction techniques to support object transformation. An exploratory study conducted with 3D design experts and novice users found the techniques to be useful in 3D scene design workflows and easy to learn and use.

Most AR and VR headsets use stereoscopic displays to show virtual objects in 3D. However, the limitations of current stereo display systems affect depth perception through conflicting depth cues, which then also affect virtual hand interaction in peri-personal space, i.e., within arm's reach. We performed a Fitts' law experiment to better understand the impact of stereo display deficiencies of AR and VR headsets on pointing at close-by targets arranged laterally or along the line of sight. According to our results, the movement direction and the corresponding change in target depth affect pointing time and throughput; subjects' movements towards/away from their head were slower and less accurate than their lateral movements (left/right). However, even though subjects moved faster in AR, we did not observe a significant difference for pointing performance between AR and VR headsets, which means that previously identified differences in depth perception between these platforms seem to have no strong effect on interaction. Our results also help 3D user interface designers understand how changes in target depth affect users' performance in different movement directions in AR and VR.

3D pointing is an integral part of Virtual Reality interaction. Typical pointing devices rely on 3D trackers and are thus subject to fluctuations in the reported pose, i.e., jitter. In this work, we explored how different levels of rotational jitter affect pointing performance and if different selection methods can mitigate the effects of jitter. Towards this, we designed a Fitts' Law experiment with three selection methods. In the first method, subjects used a single controller to position and select the object. In the second method, subjects used the controller in their dominant hand to point at objects and the trigger button of a second controller, held in their non-dominant hand, to select objects. Finally, subjects used the controller in their dominant hand to point the objects and pressed the space bar on a keyboard to select the object in the third condition. During the pointing task we added five different levels of jitter: no jitter, ±0.5°, ±1°, and ±2° uniform noise, as well as White Gaussian noise with 1° standard deviation. Results showed that the Gaussian noise and ±2° of jitters significantly reduced the throughput of the participants. Moreover, subjects made fewer errors when they performed the experiment with two controllers. Our results inform the design of 3D user interfaces, input devices and interaction techniques.

Although precise 3D positioning is not always necessary in virtual environments, it is still an important task for current and future applications of Virtual Reality (VR), including 3D modelling, engineering, and scientific applications. We focus on 3D positioning techniques in immersive environments that use a 6DOF controller as input device and present a new technique that improves 3D positioning performance in VR, in both speed and accuracy. Towards this goal, we adapted an extended sliding technique to VR systems with a controller as input device and compared it with previously presented 3DOF positioning techniques. The results showed that our new Extended VR Sliding technique significantly improved the accuracy for 3D positioning tasks, especially for targets in contact with the scene.

Reaction time training systems are used to improve user performance. Until now, such setups use physical 2D flat surfaces, e.g., a 2D touch screen or buttons mounted on a wall. We designed and investigated a mid-air reaction time training system with an immersive virtual reality (VR) headset. 12 participants performed an eye-hand coordination reaction test in three conditions: both in mid-air with or without VR controller as well as with passive haptic feedback through hitting a soft-surface wall. We also altered target and cursor sizes and used a Fitts’ law task to analyze user performance. According to the results, subjects were slower and their throughput was lower when they hit a solid surface to interact with virtual targets. Our results show that Fitts’s model can be applied to these systems to measure and assess participant training.

Most current commercial Virtual Reality (VR) drawing applications for creativity rely on freehand 3D drawing as their main interaction paradigm. However, the presence of the additional third dimension makes accurate freehand drawing challenging. Some systems address this problem by constraining or beautifying user strokes, which can be intrusive and can limit the expressivity of freehand drawing. In this paper, we evaluate the effectiveness of relying solely on visual guidance to increase overall drawing shape-likeness. We identified a set of common mistakes that users make while creating freehand strokes in VR and then designed a set of visual guides, the Smart3DGuides, which help users avoid these mistakes. We evaluated Smart3DGuides in two user studies, and our results show that non-constraining visual guides help users draw more accurately.

Text entry is still a challenging task in modern Virtual Reality (VR) systems. The lack of efficient text entry methods limits the applications that can be used productively in VR. Previous work has addressed this issue through virtual keyboards or showing the physical keyboard in VR. While physical keyboards afford faster text entry, they usually require a seated user and an instrumented environment. We introduce a new keyboard, worn on a hawker’s tray in front of the user, which affords a compact, simple, flexible, and efficient text entry solution for VR, without restricting physical movement. In our new video condition, we also show the keyboard only when the user is looking down at it. To evaluate our novel solution and to identify good keyboard visualizations, we ran a user study where we asked participants to enter both lowercase sentences as well as complex text while standing. The results show that text entry rates are affected negatively by simplistic keyboard visualization conditions and that our solution affords desktop text entry rates, even when standing.

Controllers are currently the typical input device for commercial Virtual Reality (VR) systems. Yet, such controllers are not as efficient as other devices, including the mouse. This motivates us to investigate devices that substantially exceed the controller’s performance, for both VR and Augmented Reality (AR) systems. We performed a user study to compare several input devices, including a mouse, controller, and a 3D pen-like device on a VR and AR pointing task. Our results show that the 3D pen significantly outperforms modern VR controllers in all evaluated measures and that it is comparable to the mouse. Participants also liked the 3D pen more than the controller. Finally, we show how 3D pen devices could be integrated into today’s VR and AR systems.

Moving objects is an important task in 3D user interfaces. In this work, we focus on (precise) 3D object positioning in immersive virtual reality systems, especially head-mounted displays (HMDs). To evaluate input method performance for 3D positioning, we focus on an existing sliding algorithm, in which objects slide on any contact surface. Sliding enables rapid positioning of objects in 3D scenes on a desktop system but is yet to be evaluated in an immersive system. We performed a user study that compared the efficiency and accuracy of different input methods (mouse, hand-tracking, and trackpad) and cursor display conditions (stereo cursor and one-eyed cursor) for 3D positioning tasks with the HTC Vive. The results showed that the mouse outperformed hand-tracking and the trackpad, in terms of efficiency and accuracy. Stereo cursor and one-eyed cursor did not demonstrate a significant difference in performance, yet the stereo cursor condition was rated more favourable. For situations where the user is seated in immersive VR, the mouse is thus still the best input device for precise 3D positioning.

We investigate stroking motions through successive objects with styli. There are several promising models for stroking motions, such as crossing tasks, which require endpoint accuracy of a stroke, or steering tasks, which require continuous accuracy throughout the trajectory. However, a task requiring users to repeatedly steer through constrained path segments has never been studied, although such operations are needed in GUIs, e.g., for selecting icons or objects on illustration software through lassoing. We empirically confirmed that the interval, trajectory width, and obstacle size significantly affect the movement speed. Existing models can not accurately predict user performance in such tasks. We found several unexpected results such as that steering through denser objects sometimes required less times than expected. Speed profile analysis showed the reasons behind such behaviors, such as participants' anticipation strategies. We also discuss the applicability of exiting performance models and revisions.

Head-Mounted Displays (HMDs) provide immersive experiences for virtual reality. However, their field of view (FOV) is still relatively small compared to the human eye, which adding sparse peripheral displays (SPDs) could address. We designed a new SPD, SparseLightVR2, which increases the HMD's FOV to 180° horizontally. We evaluated SparseLightVR2 with a study (N=29) by comparing three conditions: 1) no SPD, where the peripheral display (PD) was inactive; 2) extended SPD, where the PD provided visual cues consistent with and extending the HMD's main screen; and 3) counter-vection SPD, where the PD's visuals were flipped horizontally during VR travel to provide optic flow in the opposite direction of the travel. The participants experienced passive motion on a linear path and reported introspective measures such as sensation of self-motion. Results showed, compared to no SPD, both extended and counter-vection SPDs provided a more natural experience of motion, while extended SPD also enhanced vection intensity and believability of movement. Yet, visually induced motion sickness (VIMS) was not affected by display condition. To investigate the reason behind these non-significant results, we conducted a follow-up study and had users increase peripheral counter-vection visuals on the central HMD screen until they nulled out vection. Our results suggest extending HMDs through SPDs enhanced vection, naturalness, and believability of movement without enhancing VIMS, but reversed SPD motion cues might not be strong enough to reduce vection and VIMS.

Constrained interaction and navigation methods for virtual reality reduce the complexity of the interaction. Yet, with previously presented solutions, users need to learn new interaction tools or remember different actions for changing between different interaction methods. In this paper, we propose Fluid VR, a new 3D user interface for interactive virtual environments that lets users seamlessly transition between navigation and selection. Based on the selected object's properties, Fluid VR applies specific constraints to the interaction or navigation associated with the object. This way users have a better control of their actions, without having to change tools or activate different modes of interaction.

Virtual Reality (VR) is increasingly used in spatial cognition research, as it offers high experimental control in naturalistic multimodal environments, which is hard to achieve in real-world settings. Although recent technological advances offer a high level of photorealism, locomotion in VR is still restricted because people might not perceive their self-motion as they would in the real world. This might be related to the inability to use embodied spatial orientation processes, which support automatic and obligatory updating of our spatial awareness. Previous research has identified the roles reference frames play in retaining spatial orientation. Here, we propose using visually overlaid rectangular boxes, simulating reference frames in VR, to provide users with a better insight into spatial direction in landmark-free virtual environments. The current mixed-method study investigated how different variations of the visually simulated reference frames might support people in a navigational search task. Performance results showed that the existence of a simulated reference frame yields significant effects on participants completion time and travel distance. Though a simulated CAVE translating with the navigator (one of the simulated reference frames) did not provide significant benefits, the simulated room (another simulated reference frame depicting a rest frame) significantly boosted user performance in the task as well as improved participants preference in the post-experiment evaluation. Results suggest that adding a visually simulated reference frame to VR applications might be a cost-effective solution to the spatial disorientation problem in VR.

This paper presents two new pointing techniques for wiggle 3D displays, which present the 2D projection of 3D content with automatic (rotatory) motion parallax. Standard pointing at targets in wiggle 3D displays is challenging as the content is constantly in motion. The two pointing techniques presented here take advantage of the cursor's current position or the user's gaze direction for collocating the wiggle rotation center and potential targets. We evaluate the performance of the pointing techniques with a novel methodology that integrates 3D distractors into the ISO-9241-9 standard task. The experimental results indicate that the new techniques are significantly more efficient than standard pointing techniques in wiggle 3D displays. Given that we observed no performance variation for different targets, our new techniques seem to negate any interaction performance penalties of wiggle 3D displays.

We augment 3D user interfaces with a new technique that enables users to select objects that are invisible from the current viewpoint. We present a layer-based method for selecting invisible objects, which works for arbitrary objects and scenes. The user study shows that with our new techniques users can easily select hidden objects.

In presence of conflicting or ambiguous visual cues in complex scenes, performing 3D selection and manipulation tasks can be challenging. To improve motor planning and coordination, we explore audio-tactile cues to inform the user about the presence of objects in hand proximity, e.g., to avoid unwanted object penetrations. We do so through a novel glove-based tactile interface, enhanced by audio cues. Through two user studies, we illustrate that proximity guidance cues improve spatial awareness, hand motions, and collision avoidance behaviors, and show how proximity cues in combination with collision and friction cues can significantly improve performance.

We present a novel forearm-and-glove tactile interface that can enhance 3D interaction by guiding hand motor planning and coordination. In particular, we aim to improve hand motion and pose actions related to selection and manipulation tasks. Through our user studies, we illustrate how tactile patterns can guide the user, by triggering hand pose and motion changes, for example to grasp (select) and manipulate (move) an object. We discuss the potential and limitations of the interface, and outline future work.

Quantifying cumulative arm muscle fatigue is a critical factor in understanding, evaluating, and optimizing user experience during prolonged mid-air interaction. A reasonably accurate estimation of fatigue requires an estimate of an individual's strength. However, there is no easy-to-access method to measure individual strength to accommodate inter-individual differences. Furthermore, fatigue is influenced by both psychological and physiological factors, but no current HCI model provides good estimates of cumulative subjective fatigue. We present a new, simple method to estimate the maximum shoulder torque through a mid-air pointing task, which agrees with direct strength measurements. We then introduce a cumulative fatigue model informed by subjective and biomechanical measures. We evaluate the performance of the model in estimating cumulative subjective fatigue in mid-air interaction by performing multiple cross-validations and a comparison with an existing fatigue metric. Finally, we discuss the potential of our approach for real-time evaluation of subjective fatigue as well as future challenges.

The steering law models human motor performance and has been verified to hold for a single linear and/or circular path. Some extensions investigated steering around corners. Yet, little is known about human performance in navigating joined linear paths, i.e., successions of path segments with different widths. Such operations appear in graphical user interface tasks, including lasso operations in illustration software. In this work, we conducted several experiments involving joined paths. The results show that users significantly changed their behavior, and that this strategy change can be predicted beforehand. A simple model summing the two indexes of difficulty (IDs) for each path predicts movement time well, but more sophisticated models were also evaluated. The best model in terms of both of R2 and AIC values includes the ID of the crossing operation to enter the second path.

We present a new interface for interactive comparisons of more than two alternative documents in the context of a generative design system that uses generative data-flow networks defined via directed acyclic graphs. To better show differences between such networks, we emphasize added, deleted, (un)changed nodes and edges. We emphasize differences in the output as well as parameters using highlighting and enable post-hoc merging of the state of a parameter across a selected set of alternatives. To minimize visual clutter, we introduce new difference visualizations for selected nodes and alternatives using additive and subtractive encodings, which improve readability and keep visual clutter low. We analyzed similarities in networks from a set of alternative designs produced by architecture students and found that the number of similarities outweighs the differences, which motivates use of subtractive encoding. We ran a user study to evaluate the two main proposed difference visualization encodings and found that they are equally effective.

State-of-the-art auto-correction methods for predictive text entry systems work reasonably well, but can never be perfect due to the properties of human language. We present an approach for the automatic detection of erroneous auto-corrections based on brain activity and text-entry-based context features. We describe an experiment and a new system for the classification of human reactions to auto-correction errors. We show how auto-correction errors can be detected with an average accuracy of 85%.

Many techniques for visualization and interaction that potentially increase user performance have been studied in the growing field of virtual reality. However, the effects of virtual-arm representations on users' performance and perception in selection tasks have not been studied before. This paper presents the results of a user study of three different representations of the virtual arm: "hand only," "hand+forearm," and "whole arm" which includes the upper arm. In addition to the representations' effects on performance and perception in selection tasks, we investigate how the users' performance changes depending on whether collisions with objects are allowed or not. The relationship between the virtual-arm representations and the senses of agency and ownership are also explored. Overall, we found that the "whole arm" condition performed worst.

We present a new smart-restorable backspace technique to facilitate correction of "overlooked" errors on touchscreen-based tablets. We conducted an empirical study to compare the new backspace technique with the conventional one. Results of the study revealed that the new technique improves the overall text entry performance, both in terms of speed and operations per character, by significantly reducing error correction efforts. In addition, results showed that most users preferred the new technique to the one they use on their tablets, and found it easy to learn and use. Most of them also felt that it improved their overall text entry performance, thus wanted to keep using it.

Gestural interaction has become increasingly popular, as enabling technologies continue to transition from research to retail. The mobility of miniaturized (and invisible) technologies introduces new uses for gesture recognition. This paper investigates single-hand microgestures (SHMGs), detailed gestures in a small interaction space. SHMGs are suitable for the mobile and discrete nature of interactions for ubiquitous computing. However, there has been a lack of end-user input in the design of such gestures. We performed a user-elicitation study with 16 participants to determine their preferred gestures for a set of referents. We contribute an analysis of 1,632 gestures, the resulting gesture set, and prevalent conceptual themes amongst the elicited gestures. These themes provide a set of guidelines for gesture designers, while informing the designs of future studies. With the increase in hand-tracking and electronic devices in our surroundings, we see this as a starting point for designing gestures suitable to portable ubiquitous computing.

Ambient light is starting to be commercially used to enhance the viewing experience for watching TV. We believe that ambient light can add value in meeting and control rooms that use large vertical interactive surfaces. Therefore, we equipped a large interactive whiteboard with a peripheral ambient light display and explored its utility for different scenarios by conducting two controlled experiments. In the first experiment, we investigated how ambient light can be used for peripheral notifications, and how perception is influenced by the user's position and the type of work they are engaged in. The second experiment investigated the utility of ambient light for off-screen visualization. We condense our findings into several design recommendations that we then applied to application scenarios to show the versatility and usefulness of ambient light for large surfaces.

We propose a workshop soliciting submissions on the topics of spatial and 3D interaction in highly fidelity graphical interactive systems, especially video games and virtual reality systems. Topics will cover all aspects of interaction in these systems, including input-related issues such as efficient control mappings, spatial input device features (e.g., low latency, input noise, etc.), as well as output-related issues such as display hardware features (e.g., stereoscopic rendering, head-tracking), and information visualization (e.g., game HUDs for status information). We propose to include both position papers, as well as papers including at least preliminary studies on issues in these areas.

We present GEM-NI -- a graph-based generative-design tool that supports parallel exploration of alternative designs. Producing alternatives is a key feature of creative work, yet it is not strongly supported in most extant tools. GEM-NI enables various forms of exploration with alternatives such as parallel editing, recalling history, branching, merging, comparing, and Cartesian products of and for alternatives. Further, GEM-NI provides a modal graphical user interface and a design gallery, which both allow designers to control and manage their design exploration. We conducted an exploratory user study followed by in-depth one-on-one interviews with moderately and highly skills participants and obtained positive feedback for the system features, showing that GEM-NI supports creative design work well.

In this paper we describe a new orbiting algorithm, called SHOCam, which enables simple, safe and visually attractive control of a camera moving around 3D objects. Compared with existing methods, SHOCam provides a more consistent mapping between the user's interaction and the path of the camera by substantially reducing variability in both camera motion and look direction. Also, we present a new orbiting method that prevents the camera from penetrating object(s), making the visual feedback -- and with it the user experience -- more pleasing and also less error prone. Finally, we present new solutions for orbiting around multiple objects and multi-scale environments.

In this article, we present a new user authentication technique for touchscreen-based smartphone users that augments pseudo touch pressure as an extra security measure to the conventional digit-lock technique. The new technique enhances security by offering more unique password combinations than the

We present a study of cursors for selecting 2D-projected 3D targets. We compared a stereo- and mono-rendered (one-eyed) cursor using two mouse-based and two remote pointing techniques in a 3D Fitts' law pointing experiment. The first experiment used targets at fixed depths. Results indicate that one-eyed cursors only improve screen-plane pointing techniques, and that constant target depth does not influence pointing throughput. A second experiment included pointing between targets at varying depths and used only "screen-plane" pointing techniques. Our results suggest that in the absence of stereo cue conflicts, screen-space projections of Fitts' law parameters (target size and distance) yield constant throughput despite target depth differences and produce better models of performance.

Layout managers are used to control the placement of widgets in graphical user interfaces (GUIs). Constraint-based layout managers are among the most powerful. However, they are also more complex and their layouts are prone to problems such as over-constrained specifications and widget overlap. This poses challenges for GUI builder tools, which ideally should address these issues automatically. We present a new GUI builderthe Auckland Layout Editor (ALE)that addresses these challenges by enabling GUI designers to specify constraint-based layouts using simple, mouse-based operations. We give a detailed description of ALE's edit operations, which do not require direct constraint editing. ALE guarantees that all edit operations lead to sound specifications, ensuring solvable and non-overlapping layouts. To achieve that, we present a new algorithm that automatically generates the constraints necessary to keep a layout non-overlapping. Furthermore, we discuss how our innovations can be combined with manual constraint editing in a sound way. Finally, to aid designers in creating layouts with good resize behavior, we propose a novel automatic layout preview. This displays the layout at its minimum and in an enlarged size, which allows visualizing potential resize issues directly. All these features permit GUI developers to focus more on the overall UI design.

Stable structured layouts of buttons are a primary means of control for input in current graphical user interfaces. Such layouts are ubiquitous---from tiny iPhone screens to large kiosk screens in the malls---they are found everywhere. Yet, there has been relatively little theoretical account that compares the impact of cognitive effort on learning such stable layouts. In this paper, we demonstrate that prior empirical results on cognitive effort in learning stable layouts are theoretically predictable through the memory activation model of a cognitive architecture, ACT-R. We go beyond previous work by quantitatively comparing the level of cognitive effort in terms of a newly introduced parameter in the declarative memory model of ACT-R. We theoretically compare the cognitive effort of two different layouts of graphical buttons with respect to their label representativeness in the domains of traditional keyboard and ShapeWriter.

This demo presents a desktop VR system for evaluating human performance in 3D pointing tasks. The system supports different input devices (e.g., mouse and 6DOF remote pointer), pointing techniques (e.g., screen-plane and depth cursors), and cursor visualization styles (e.g., one-eyed and stereo 3D cursors). The objective is to comprehensively compare all combinations of these conditions. We especially focus in on fair and direct comparisons between 2D and 3D pointing tasks. Finally, our system includes a new pointing technique that outperforms standard ray pointing.

Researchers have developed many models to predict and understand human performance in text entry. Most of the models are specific to a technology or fail to account for human factors and variations in system parameters, and the relationship between them. Moreover, the process of fixing errors and its effects on text entry performance has not been studied. Here, we first analyze real-life text entry error correction behaviors. We then use our findings to develop a new model to predict the cost of error correction for character-based text entry technologies. We validate our model against quantities derived from the literature, as well as with a user study. Our study shows that the predicted and observed cost of error correction correspond well. At the end, we discuss potential applications of our new model.

We investigate the effects of input device latency and spatial jitter on 2D pointing tasks and a 3D movement. First, we characterize jitter and latency in a 3D tracking device and an optical mouse used for baseline comparison. We present an experiment based on ISO 9241-9, which measures performance of pointing devices. We added latency and jitter to the mouse and compared it to a 3D tracker. Results indicate that latency has a stronger effect on performance than small spatial jitter. A second experiment found that erratic jitter "spikes" can affect 3D movement performance.

No work on mobile text messaging so far has taken into account the effect of learning on the change in visual exploration behavior as users progress from non-expert to expert level. We discuss within the domain of multi-tap texting on mobile phone and address the process of searching versus selecting a letter on the keypad interface. We develop a simulation model that forecasts the probability of letter location recall by non-expert users and thereby models learning, as the user acquires expertise in recalling, with practice, session after session. We then plugin this probability within a model of visual strategy that combines the effect of different ways visual exploration: non-expert users search for a letter while expert users select a letter. The observed non-expert non-motor time preceding a key press (for a letter) correlates extremely well with the simulation results.

We present two studies to assess which physical factors of various input devices influence 3D object movement tasks. In particular, we evaluate the factors that seem to make the mouse a good input device for constrained 3D movement tasks. The first study examines the effect of a supporting surface across orientation of input device movement and display orientation. Surprisingly, no significant results were found for the effect of physical support for constrained movement techniques. Also, no significant difference was found between matching the orientation of the display to that of the input device movement. A second study found that the mouse outperformed all tracker conditions for speed, but the presence or absence of support had no significant effect when tracker movement is constrained to 2D.

Application designers of collaborative distributed virtual environments must account for the influence of the network connection and its detrimental effects on user performance. Based upon analysis and classification of existing latency compensation techniques, this paper introduces a novel approach to latency amelioration in the form of a two-tier predictor-estimator framework. The technique is variability-aware due to its proactive sender-side prediction of a pose a variable time into the future. The prediction interval required is estimated based on current and past network delay characteristics. This latency estimate is subsequently used by a Kalman filter-based predictor to replace the measurement event with a predicted pose that matches the event's arrival time at the receiving workstation. The compensation technique was evaluated in a simulation through an offline playback of real head motion data and network delay traces collected under a variety of real network conditions. The experimental results indicate that the variability-aware approach significantly outperforms a state-of-the-art one, which assumes a constant system delay

Conceptual design dominates the early stages of most creative design processes. During these stages, the designer makes important decisions about the parameters of a model that are aimed at satisfying a set of design criteria. To do this, the designer produces many sketches of various possible solutions. Meanwhile, computer-aided design (CAD) systems are becoming the dominant visual medium used in design practice. However, these tools evolved as documentation production tools and do not support conceptual thinking. This paper presents a list of guidelines for computer support for conceptual design activities on 3D scenes and presents SESAME (Sketch, Extrude, Sculpt, and Manipulate Easily), a system based on these guidelines. Finally, we present a user study comparing SESAME with a conventional CAD package to demonstrate the effectiveness of SESAME.

User interfaces are becoming more and more complex. Adaptable and adaptive interfaces have been proposed to address this issue and previous studies have shown that users prefer interfaces that they can adapt to self-adjusting ones. However, most existing systems provide users with little support for adapting their interfaces. Interface customization techniques are still very primitive and usually constricted to particular applications. In this paper, we present User Interface Façades, a system that provides users with simple ways to adapt, reconfigure, and re-combine existing graphical interfaces, through the use of direct manipulation techniques. The paper describes the user's view of the system, provides some technical details, and presents several examples to illustrate its potential.

Existing commercial technologies do not adequately meet the requirements for tracking in fully-enclosed VR displays. We present the Hedgehog, which overcomes several limitations imposed by existing sensors and tracking technology. The tracking system robustly and reliably estimates the 6DOF pose of the device with high accuracy and a reasonable update rate. The system is composed of several cameras viewing the display walls and an arrangement of laser diodes secured to the user. The light emitted from the lasers projects onto the display walls and the 2D centroids of the projections are tracked to estimate the 6DOF pose of the device. The system is able to handle ambiguous laser projection configurations, static and dynamic occlusions of the lasers, and incorporates an intelligent laser activation control mechanism that determines which lasers are most likely to improve the pose estimate. The Hedgehog is also capable of performing auto-calibration of the necessary camera parameters through the use of the SCAAT algorithm. A preliminary evaluation reveals that the system has an angular resolution of 0.01 degrees RMS and a position resolution of 0.2 mm RMS.

In this paper we present a new model for predicting text entry speed on a 12-button mobile phone keypad. The proposed model can predict the performance of novice users. Like other models for text entry, the proposed model includes a movement component based on Fitts' law and a linguistic component based on letter digraph probabilities. It also adds cognitive delay times before key presses and takes into account the fact that Fitts' law cannot model multiple presses of the same key accurately. Finally, we compare the prediction of our model to previously published experimental results, demonstrate that it fits observed results for novices very well, and list some observations about learning.

The dynamic range of many real-world environments exceeds the capabilities of current display technology by several orders of magnitude. In this paper we discuss the design of two different display systems that are capable of displaying images with a dynamic range much more similar to that encountered in the real world. The first display system is based on a combination of an LCD panel and a DLP projector, and can be built from off-the-shelf components. While this design is feasible in a lab setting, the second display system, which relies on a custom-built LED panel instead of the projector, is more suitable for usual office workspaces and commercial applications. We describe the design of both systems as well as the software issues that arise. We also discuss the advantages and disadvantages of the two designs and potential applications for both systems.

We describe simple techniques for object group manipulation, an important operation in user interaction with a virtual environment. All presented manipulation techniques exploit constraints to simplify user interaction. The techniques are based on how humans perceive groups and afford direct manipulation of such groups. Furthermore, we introduce two new intuitive ways to create a whole group of objects: drag-add and random drag-add. Finally, we present an evaluation of the presented techniques.

We present a system for rendering very complex 3D models at interactive rates.We select a subset of the model as preferred viewpoints and partition the space into virtual cells.Each cell contains near geometry, rendered using levels of detail and visibility culling, and far geometry, rendered as a textured depth mesh.Our system automatically balances the screen-space errors resulting from geometric simplification with those from textureddepth-mesh distortion.We describe our prefetching and data management schemes, both crucial for models significantly larger than available system memory.We have successfully used our system to accelerate walkthroughs of a 13 million triangle model of a large coal-fired power plant and of a 1.7 million triangle architectural model.We demonstrate the walkthrough of a 1.3 GB power plant model with a 140 MB cache footprint.

This paper presents a new approach to rendering triangular algebraic free-form surfaces. A hierarchical subdivision of the surface with associated tight bounding volumes provides for quick identification of the surface regions likely to be hit by a ray. For each leaf of the hierarchy, an approximation to the corresponding surface region is stored. The approximation is used to compute a good starting point for the iteration, which ensures rapid convergence. Trimming curves are described by a tree of trimming primitives, such as squares, circles, polygons, and free-form curves, combined with Boolean operations. For trimmed surfaces, an irregular adaptive subdivision is constructed to quickly eliminate all parts outside the trimming curve from consideration during rendering. Cost heuristics are introduced to optimize the rendering time further.

Abstract Despite recent advances in rendering hardware, large and complex virtual environments cannot be displayed with a sufficiently highframe rate, because of limitations in the available rendering performance. This paper presents a new approach of software accelerated rendering which draws from the concepts of impostors, hierarchical scene subdivision and levels of detail. So far software optimization in real‐time rendering has merely considered individual objects. This work is actually optimizing the rendering of the whole virtual environment by implementing a three dimensional image cache. It speeds up rendering for large portions of the scene by exploiting the coherence inherent in any smooth frame sequence. The implementation of the three dimensional image cache is discussed and the savings in rendering load achievable on a suitable hardware platform are presented.