We present SimXR, a methodfor controlling a simulated avatar from information (headset pose and cameras) ob-tained from AR / VR headsets. Due to the challenging view-point of head-mounted cameras, the human body is often clipped out of view, making traditional image-based ego-centric pose estimation challenging. On the other hand, headset poses provide valuable information about overall body motion, but lack fine-grained details about the hands and feet. To synergize headset poses with cameras, we control a humanoid to track headset movement while analyzing input images to decide body movement. When body parts are seen, the movements of hands and feet will be guided by the images; when unseen, the laws of physics guide the controller to generate plausible motion. We design an end-to-end method that does not rely on any intermediate representations and learns to directly map from images and headset poses to humanoid control signals. To train our method, we also propose a large-scale synthetic dataset created using camera configurations compatible with a commercially available VR headset (Quest 2) and show promising results on real-world captures. To demonstrate the applicability of our framework, we also test it on an AR headset with a forward-facing camera.

We propose RoHM, an approach for robust 3D human motion reconstruction from monocular RGB(-D) videos in the presence of noise and occlusions. Most previous approaches either train neural networks to directly regress motion in 3D or learn data-driven motion priors and com-bine them with optimization at test time. The former do not recover globally coherent motion and fail under occlusions; the latter are time-consuming, prone to local minima, and require manual tuning. To overcome these shortcomings, we exploit the iterative, denoising nature of diffusion models. RoHM is a novel diffusion-based motion model that, conditioned on noisy and occluded input data, reconstructs complete, plausible motions in consistent global co-ordinates. Given the complexity of the problem - requiring one to address different tasks (denoising and infilling) in different solution spaces (local and global motion) - we de-compose it into two sub-tasks and learn two models, one for global trajectory and one for local motion. To capture the correlations between the two, we then introduce a novel conditioning module, combining it with an iterative inference scheme. We apply RoHM to a variety of tasks from motion reconstruction and denoising to spatial and temporal infilling. Extensive experiments on three popular datasets show that our method outperforms state-of-the-art approaches qualitatively and quantitatively, while being faster at test time. The code is available at https://sanweiliti.github.io/ROHM/ROHM.html.

We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. Due to the high dimensionality of humanoids and the inherent difficulties in reinforcement learning, prior methods have focused on learning skill embeddings for a narrow range of movement styles (e.g. locomotion, game characters) from specialized motion datasets. This limited scope hampers their applicability in complex tasks. We close this gap by significantly increasing the coverage of our motion representation space. To achieve this, we first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our motion representation by distilling skills directly from the imitator. This is achieved by using an encoder-decoder structure with a variational information bottleneck. Additionally, we jointly learn a prior conditioned on proprioception (humanoid's own pose and velocities) to improve model expressiveness and sampling efficiency for downstream tasks. By sampling from the prior, we can generate long, stable, and diverse human motions. Using this latent space for hierarchical RL, we show that our policies solve tasks using human-like behavior. We demonstrate the effectiveness of our motion representation by solving generative tasks (e.g. strike, terrain traversal) and motion tracking using VR controllers.

We present a method for controlling a simulated humanoid to grasp an object and move it to follow an object's trajectory. Due to the challenges in controlling a humanoid with dexterous hands, prior methods often use a disembodied hand and only consider vertical lifts or short trajectories. This limited scope hampers their applicability for object manipulation required for animation and simulation. To close this gap, we learn a controller that can pick up a large number (>1200) of objects and carry them to follow randomly generated trajectories. Our key insight is to leverage a humanoid motion representation that provides human-like motor skills and significantly speeds up training. Using only simplistic reward, state, and object representations, our method shows favorable scalability on diverse objects and trajectories. For training, we do not need a dataset of paired full-body motion and object trajectories. At test time, we only require the object mesh and desired trajectories for grasping and transporting. To demonstrate the capabilities of our method, we show state-of-the-art success rates in following object trajectories and generalizing to unseen objects. Code and models will be released.

This paper presents a novel approach to intuitive force-controlled motion planning on collaborative robots using augmented reality (AR) technology. A user-friendly interface is developed that grants the operator complete control over the robot manipulator. By employing a mixed reality head-mounted display (HMD) as the interface, virtual content is overlaid, enabling the operator to interact seamlessly with the robotic system. The interface provides extensive data on the robot’s status, including joint position, velocity, and the applied force on the robot’s flange. Operators can issue motion commands in both joint and Cartesian space, intuitively plan robot paths using waypoints, and execute force-controlled motion by defining control points around an object. Visual feedback in the form of superimposed sliders indicates the force to be exerted by the robot on the object. These sliders allow dynamic and intuitive adjustment of forces in Cartesian space, minimizing the need for extensive programming. Safety is a primary concern, and to address it, a virtual model of the robot is superimposed on the work environment, providing a preview of the motion. This preview displays the current and final positions of each joint before execution. The human-robot interface and virtual content are built using the Unity3D game engine, while reliable data transmission and processing between the HMD and the robot controller are facilitated by the Robot Operating System (ROS). This approach offers an intuitive and safer method for controlling collaborative robots, empowering operators with greater precision and ease in robot motions. The proposed approach has significant potential to streamline robot programming, enhance efficiency, and improve safety across a wide range of applications involving collaborative robots.

We present a physics-based humanoid controller that achieves high-fidelity motion imitation and fault-tolerant behavior in the presence of noisy input (e.g. pose estimates from video or generated from language) and unexpected falls. Our controller scales up to learning ten thousand motion clips without using any external stabilizing forces and learns to naturally recover from fail-state. Given reference motion, our controller can perpetually control simulated avatars without requiring resets. At its core, we propose the progressive multiplicative control policy (PMCP), which dynamically allocates new network capacity to learn harder and harder motion sequences. PMCP allows efficient scaling for learning from large-scale motion databases and adding new tasks, such as fail-state recovery, without catastrophic forgetting. We demonstrate the effectiveness of our controller by using it to imitate noisy poses from video-based pose estimators and language-based motion generators in a live and real-time multi-person avatar use case.

In navigated surgery, physicians perform complex tasks assisted by virtual representations of anatomical structures and surgical tools. Integrating Augmented Reality (AR) in these scenarios enriches the information presented to the surgeon through a range of visualization techniques. Their selection is a crucial task as they represent the primary interface between the system and the surgeon.In this work, we present a novel approach to conveying augmented content using dynamic visualization techniques, allowing users to gather depth and shape information from both pictorial and kinetic cues. We conducted user studies comparing two novel dynamic methods – Object Flow and Wave Propagation – and three state-of-the-art static visualization techniques among medical experts. Our studies provide a detailed comparison of the visualization techniques’ efficacy in conveying shape and depth information from medical data, as well as task load and usability reported by the participants and post hoc analyses. We found that kinetic cues can assist users in understanding complex anatomical structures in medical AR.

When users create hand-drawn annotations in Virtual Reality they often reach their physical limits in terms of precision, especially if the region to be annotated is small. One intuitive solution employs magnification beyond natural scale. However, scaling the whole environment results in wrong assumptions about the coherence between physical and virtual space. In this paper, we introduce Mag-noramas, a novel interaction method for selecting and extracting a region of interest that the user can subsequently scale and transform inside the virtual space. Our technique enhances the user's capabilities to perform supernaturally precise virtual annotations on virtual objects. We explored our technique in a user study within asimplified clinical scenario of a teleconsultation-supported craniectomy procedure that requires accurate annotations on a human head. Teleconsultation was performed asymmetrically between a remote expert in Virtual Reality that collaborated with a local user through Augmented Reality. The remote expert operates inside a reconstructed environment, captured from RGB-D sensors at the local site, and is embodied by an avatar to establish co-presence. The results show that Magnoramas significantly improve the precision of annotations while preserving usability and perceived presence measures compared to the baseline method. By hiding the 3D reconstruction while keeping the Magnorama, users can intentionally choose to lower their perceived social presence and focus on their tasks.

A recurrent problem in egocentric Augmented Reality (AR) applications is the misestimation of depth. Providing alternative views from non-egocentric perspectives can convey useful information for applications that require the correct judgment of depth as it is in the case of placement and alignment of virtual and real content, but also for exploration and visualization tasks.In this paper, we introduce Augmented Mirrors. Through the integration of a real mirror, our approach is capable to reflect changes of the real and virtual content of an AR application while users benefit from the perceptual advantages of using mirrors. Our concept, simple yet effective, only requires tracking the user and mirror poses with the accuracy demanded by a specific application. To showcase the potential and flexibility of the Augmented Mirrors, we present and discuss multiple examples ranging from alignment, exploration, spatial understanding, and selective content visualization using different AR-enabled devices and tracking technologies. We envision the Augmented Mirrors as a new and valuable concept that can be used in applications that benefit from additional viewpoints and require the simultaneous visualization of real and virtual content.

In this paper, the accuracy of material decomposition (MD) using an energy discriminating photon counting detector was studied.An MD framework was established and validated using calcium hydroxyapatite (CaHA) inserts of known densities (50 mg/cm 3 , 100 mg/cm 3 , 250 mg/cm 3 , 400 mg/cm 3 ), and diameters (1.2, 3.0, and 5.0 mm).These inserts were placed in a cardiac rod phantom that mimics a tissue equivalent heart and measured using an experimental photon counting detector cone beam computed tomography (PCD-CBCT) setup.The quantitative coronary calcium scores (density, mass, and volume) obtained from the MD framework were compared with the nominal values.In addition, three different calibration techniques, signal-to-equivalent thickness calibration (STC), polynomial correction (PC), and projected equivalent thickness calibration (PETC) were compared to investigate the effect of the calibration method on the quantitative values.The obtained MD estimates agreed well with the nominal values for density (mass) with mean absolute percent errors (MAPEs) 8 ± 11% (9 ± 15%) and 4 ± 6% (9 ± 14%) for STC and PETC calibration methods, respectively.PC displayed large MAPEs for density (27 ± 9%), and mass (25 ± 12%).Volume estimation resulted in large deviations between true and measured values with notable MAPEs for STC (40 ± 90%), PC (40 ± 80%), and PETC (40 ± 90%).The framework demonstrated the feasibility of quantitative CaHA mass and density scoring using PCD-CBCT.

One major cause of preventable death is a lack of proper skills for providing critical care. Conventional training for advanced emergency medical procedures is often limited to a verbal block of instructions and/or an instructional video. In this study, we evaluate the benefits of using an optical see-through head-mounted display (OST-HMD) for training of caregivers in an emergency medical environment. A rich user interface was implemented that provides 3D visual aids including images, text and tracked 3D overlays for each task. A user study with 20 participants was conducted for two medical tasks, where each subject received conventional training for one task and HMD training for the other task. Our results indicate that using a mixed reality HMD is more engaging, improves the time-on-task, and increases the confidence level of users.

We present a unique simulator-based methodology for assessing both technical and nontechnical (cognitive) skills for surgical trainees while immersed in a complete medical simulation environment. Further, we have included two crisis scenarios which allow for the evaluation of the effect of cognitive strategy selection on the low-level surgical skills. Training these mixed-mode scenarios can thereby be evaluated on our platform, allowing for improved assessment and a stronger foundation for credentialing, with the potential to reduce the occurrence of adverse events in the operating room. Scientific evaluation and validation of our work is conducted together with 19 junior surgeons in order to achieve the following goals: 1) to provide a qualitative measure of usability, 2) to assess vertebroplasty technical performance of the surgeon, and 3) to explore the relationship between mental workload and surgical performance during crisis. Our results indicate that: 1) the surgeons scored the face validity of our modeled simulation environment very highly ( 4.68 ±0.48, using a 5-point Likert scale), 2) surgeon training enabled completion of tasks more quickly, and 3) the introduction of crisis scenarios negatively affected the surgeons' objective performance. Taken together, our results underscore the need to develop realistic simulation environments that prepare young residents to respond to emergent events in the operating room.

This article presents an algorithm for dynamic force/torque measurement and robot load identification using the so called 12DOF sensor to measure forces/torques and linear/angular accelerations. The basic equations of dynamic forces and torques arising during robot motion and acting on the end-effector were worked out. To be able to perform the experiments suitable robot system based on a six axes articulated manipulator was constituted. For this system also the appropriate software was developed. The load parameters of the tool were determined during particular motion sequence of the robot and the values were compared with values from CAD software. The compensation of dynamic forces and torques is verified using the experimental robot system and the results are presented.