Remote coaching for sports is challenged by the lack of 3D spatial communication. While athletes send live or recorded videos to their coaches, these 2D representations fail to capture the spatial relationships of the body, limiting the ability to understand timing, weight distribution, and smoothness in an athletic movement. This demonstration presents Augmented Coach, an AR sports coaching platform for coaches to remotely view, manipulate, and annotate athletic movements in 3D augmented space. Also, this demonstration provides an adaptive platform to study real-time efficient volumetric data transmission between remotely connected devices, including over 5G cellular networks.

Mimicking physical odor sensations virtually can present users with a real - time odor synthesis that approximates what users would smell in a virtual environment, e.g., as they walk around in virtual reality. To this end, we devise a Smell Engine that includes: (i) a Smell Composer framework that allows developers to configure odor sources in virtual space, (ii) a Smell Mixer that dynamically estimates the odor mix that the user would smell, based on diffusion models and relative odor source distances, and (iii) a Smell Controller that coordinates an olfactometer to physically present an approximation of the odor mix to the user’s mask from a set of odorants channeled through controllable flow valves. Through a three - part user study, we found that the Smell Engine can help measure a subject’s olfactory detection threshold and improve their ability to precisely localize odors in the virtual environment, as compared to existing trigger - based solutions.

Remote sensing is currently the primary method of obtaining knowledge about the composition and physical properties of the surface of other planets. In a commonly used technique, visible and near-infrared (VNIR) spectrometers onboard orbiting satellites capture reflectance data at different wavelengths, which in turn gives insight about the minerals present and the overall composition of the terrain. In select locations on Mars, rovers have also conducted up close in-situ investigation of the same terrains examined by orbiters, allowing direct comparisons at different spatial scales. In this work, we build Planetary Visor, a virtual reality tool to visualize orbital and ground data around NASA's Mars Science Laboratory Curiosity rover's ongoing traverse in Gale Crater. We have built a 3D terrain along Curiosity's traverse using rover images, and within it we visualize satellite data as polyhedrons, superimposed on that terrain. This system provides perspectives of VNIR spectroscopic data from a satellite aligned with ground images from the rover, allowing the user to explore both the physical aspects of the terrain and their relation to the mineral composition. The result is a system that provides seamless rendering of datasets at vastly different scales. We conduct a user study with subject matter experts to evaluate the success and potential of our tool. The results indicate that Visor assists with geometric understanding of spectral data, improved geological context, a better sense of scale while navigating terrain, and new insights into spectral data. The result is not only an immersive environment in a scientifically interesting area on Mars, but a robust tool for analysis and visualization of data that can yield improved scientific discovery. This technology is relevant to the ongoing operations of the Curiosity rover and will directly be able to represent the data collected in the upcoming Mars 2020 Perseverance rover mission.

Mixed reality mobile platforms attempt to co-locate virtual scenes with physical environments, towards creating immersive user experiences. However, to create visual harmony between virtual and physical spaces, the virtual scene must be accurately illuminated with realistic lighting that matches the physical environment. To this end, we design GLEAM, a framework that provides robust illumination estimation in real-time by integrating physical light-probe estimation with current mobile AR systems. GLEAM visually observes reflective objects to compose a realistic estimation of physical lighting. Optionally, GLEAM can network multiple devices to sense illumination from different viewpoints and compose a richer estimation to enhance realism and fidelity. Using GLEAM, AR developers gain the freedom to use a wide range of materials, which is currently limited by the unrealistic appearance of materials that need accurate illumination, such as liquids, glass, and smooth metals. Our controlled environment user studies across 30 participants reveal the effectiveness of GLEAM in providing robust and adaptive illumination estimation over commercial status quo solutions, such as pre-baked directional lighting and ARKit 2.0 illumination estimation. Our benchmarks reveal the need for situation driven tradeoffs to optimize for quality factors in situations requiring freshness over quality and vice-versa. Optimizing for different quality factors in different situations, GLEAM can update scene illumination as fast as 30ms by sacrificing richness and fidelity in highly dynamic scenes, or prioritize quality by allowing an update interval as high as 400ms in scenes that require high-fidelity estimation.

Mixed reality mobile platforms attempt to co-locate virtual scenes with physical environments, towards creating immersive user experiences. However, to create visual harmony between virtual and physical spaces, the virtual scene must be accurately illuminated with realistic lighting that matches the physical environment. To this end, we design GLEAM, a framework that provides robust illumination estimation in real-time by integrating physical light-probe estimation with current mobile AR systems. We present a demo implementation of GLEAM by means of an AR application that estimates environmental illumination and renders the scene with real-time illumination updates. We demonstrate the efficacy of GLEAM's estimation against a current commercial status quo solution, Apple's ARKit, with the same application.

Mixed-reality haptic devices introduce a gateway to otherwise intangible virtual content, creating a life-like immersive experience. Congruent haptic sensation requires faithful integration of visual stimuli and perceived tactile sensation. Unfortunately, current commercial mixed-reality systems are unable to reproduce the physical sensation of fluid vessels, due to the shifting nature of fluid motion. To this end, we introduce SWISH, a novel type of ungrounded mixed-reality system, capable of affording the users a realistic haptic sensation of fluid behavior. We also present solutions to prominent challenges of rendering haptic fluid behavior, especially in coordinate translation and virtual adaptation to physical limitation. Our virtual-to-physical coupling uses Nvidia Flex's Unreal Engine integration, wirelessly controlling a motorized mechanical actuation system housed in a plastic "vessel''. In this paper we discuss the current state of SWISH and present results from our preliminary user study, followed by a description of our future planned phases.

Current VR/AR systems are unable to reproduce the physical sensation of fluid vessels, due to the shifting nature of fluid motion. To this end, we introduce SWISH, an ungrounded mixed-reality interface, capable of affording the users a realistic haptic sensation of fluid behaviors in vessels. The chief mechanism behind SWISH is in the use of virtual reality tracking and motor actuation to actively relocate the center of gravity of a handheld vessel, emulating the moving center of gravity of a handheld vessel that contains fluid. In addition to solving challenges related to reliable and efficient motor actuation, our SWISH designs place an emphasis on reproducibility, scalability, and availability to the maker culture. Our virtual-to-physical coupling uses Nvidia Flex's Unity integration for virtual fluid dynamics with a 3D printed augmented vessel containing a motorized mechanical actuation system. To evaluate the effectiveness and perceptual efficacy of SWISH, we conduct a user study with 24 participants, 7 vessel actions, and 2 virtual fluid viscosities in a virtual reality environment. In all cases, the users on average reported that the SWISH bucket generates accurate tactile sensations for the fluid behavior. This opens the potential for multi-modal interactions with programmable fluids in virtual environments for chemistry education, worker training, and immersive entertainment.