Dance-based exergames like Just Dance can be a fun way to boost your fitness and sharpen your mind. However, designing the dance routines requires expertise in modeling and animation. We introduce an augmented reality (AR) personalized dance generation framework that synthesizes dance routines according to specified physical and cognitive intensities. Our system utilizes a curated library of motion-capture dance segments, which are intelligently combined through an optimization process to meet user-defined intensity and cognitive goals. This optimization also ensures smooth transitions between movements for natural dance flow. Users can customize routines by specifying physical constraints or injuries. Implemented in a depth-camera-based exergame that provides real-time performance feedback, our framework was evaluated through experiments and user studies confirming its effectiveness in generating personalized routines with varying levels of physical and cognitive intensity.

Cognitive load affects learning and task performance; specifically, increased cognitive load hinders an individual’s ability to process information. In augmented reality (AR) interfaces, distracting notifications can also heighten cognitive load. Integrated gaze tracking offers a non-intrusive way to monitor cognitive states and provides the opportunity to predict and adapt to changes in cognitive load. In this paper, we demonstrate how cognitive load prediction models can leverage built-in gaze-tracking data in AR to accurately predict cognitive load during search tasks. We collected gaze data from participants under both cognitively overloaded and non-overloaded states and analyzed gaze feature signatures to identify load-dependent patterns. We compared individual and group-trained models for predictive performance and generalizability. We initially used logistic regression, then tested tree-based ensemble models to improve performance. The best-performing XGBoost group model achieves a test AUC-ROC of 0.85. This work demonstrates robust cognitive load monitoring for AR tasks using built-in eye-tracking measurements.

Input false-positive errors, where a system recognizes an input action that the user did not perform, have been shown to be particularly costly for user experience. Recent work has suggested that eye-gaze behavior immediately following an input event can be used to detect whether the input was intended by a user or was the result of a false-positive error. The ability to detect these errors could enable systems that assist the user with error recovery, but little is currently known about how such error mediation techniques might be designed, or the benefits they could provide. This paper presents an initial investigation of the design of error mediation techniques, and an evaluation of their potential benefits. A controlled study demonstrated that error mediation techniques can save time when recovering from errors by helping users to notice and resolve these errors quickly when they occur.

Three-dimensional curve drawing in Augmented Reality (AR) enables users to create 3D curves that fit within the real-world scene. It has applications in 3D design, sculpting, and animation. However, the task complexity increases when the desirable path for the curve is obstructed by the physical environment or by what the camera can see. For example, it is difficult to draw a curve that wraps around an object or scales to out-of-reach places. We propose WARPY, an environment-aware 3D curve drawing tool for mobile AR. Our system enables users to draw freeform curves from a distance in AR by combining 2D-to-3D sketch inference with geometric proxies. Geometric Proxies can be obtained via 3D scanning or from a list of pre-defined primitives. WARPY also provides a multi-view mode to enable users to sketch a curve from multiple viewpoints, which is useful if the target curve cannot fit within the camera's field of view. We conducted two user studies and found that WARPY can be a viable tool to help users create complex and large curves in AR.

The recent popularity of consumer-grade virtual reality devices has enabled users to experience immersive shopping in virtual environments. As in a real-world store, the placement of products in a virtual store should appeal to shoppers, which could be time-consuming, tedious, and non-trivial to create manually. Thus, this work introduces a novel approach for automatically optimizing product placement in virtual stores. Our approach considers product exposure and spatial constraints, applying an optimizer to search for optimal product placement solutions. We conducted qualitative scene rationality and quantitative product exposure experiments to validate our approach with users. The results show that the proposed approach can synthesize reasonable product placements and increase product exposures for different virtual stores.

Virtual pets are an alternative to real pets, providing a substitute for people with allergies or preparing people for adopting a real pet. Recent advancements in mixed reality pave the way for virtual pets to provide a more natural and seamless experience for users. However, one key challenge is embedding environmental awareness into the virtual pet (e.g., identifying the food bowl’s location) so that they can behave naturally in the real world.

Character design is a lengthy process, requiring artists to iteratively alter their characters' features and colorization schemes according to feedback from creative directors or peers. Artists experiment with multiple colorization schemes before deciding on the right color palette. This process may necessitate several tedious manual re-colorizations of the character. Any substantial changes to the character's appearance may also require manual re-colorization. Such complications motivate a computational approach for visualizing characters and drafting solutions. We propose a character exploration tool that automatically colors a sketch based on a selected style. The tool employs a Generative Adversarial Network trained to automatically color sketches. The tool also allows a selection of faces to be used as a template for the character's design. We validated our tool by comparing it with using Photoshop for character exploration in our pilot study. Finally, we conducted a study to evaluate our tool's efficacy within the design pipeline.

A key challenge for virtual reality level designers is striking a balance between maintaining the immersiveness of VR and providing users with on-screen aids after designing a virtual experience. These aids are often necessary for wayfinding in virtual environments with complex paths. We introduce a novel adaptive aid that maintains the effectiveness of traditional aids, while equipping designers and users with the controls of how often help is displayed. Our adaptive aid uses gaze patterns in predicting user's need for navigation aid in VR and displays mini-maps or arrows accordingly. Using a dataset of gaze angle sequences of users navigating a VR environment and markers of when users requested aid, we trained an LSTM to classify user's gaze sequences as needing navigation help and display an aid. We validated the efficacy of the adaptive aid for wayfinding compared to other commonly-used wayfinding aids.

Eye-tracking enables researchers to conduct complex analysis on human behavior. With the recent introduction of eye-tracking into consumer-grade virtual reality headsets, the barrier of entry to visual attention analysis in virtual environments has been lowered significantly. Whether for arranging artwork in a virtual museum, posting banners for virtual events or placing advertisements in virtual worlds, analyzing visual attention patterns provides a powerful means for guiding visual element placement. In this work, we propose a novel data-driven optimization approach for automatically analyzing visual attention and placing visual elements in 3D virtual environments. Using an eye-tracking virtual reality headset, we collect eye-tracking data which we use to train a regression model for predicting gaze duration. We then use the predicted gaze duration output of our regressors to optimize the placement of visual elements with respect to certain visual attention and design goals. Through experiments in several virtual environments, we demonstrate the effectiveness of our optimization approach for predicting gaze duration and for placing visual elements in different practical scenarios. Our approach is implemented as a useful plug-in that level designers can use to automatically populate visual elements in 3D virtual environments.