We introduce Audio Personas, enabling users to “decorate” themselves with body-anchored sounds in audio augmented reality. Like outfits, makeup, and fragrances, audio personas offer an alternative yet dynamic channel to augment face-to-face interactions. For instance, one can set their audio persona as rain sounds to reflect a bad mood, bee sounds to establish personal boundaries, or a playful “woosh” sound to mimic passing by someone like a breeze. To instantiate the concept, we implemented a headphone-based prototype with multi-user tracking and audio streaming. Our preregistered in-lab study with 64 participants showed that audio personas influenced how participants formed impressions. Individuals with positive audio personas were rated as more socially attractive, more likable, and less threatening than those with negative audio personas. Our study with audio designers revealed that audio personas were preferred in public and semi-public-private spaces for managing social impressions (e.g., personality) and signaling current states (e.g., emotions).

While one’s facial expression and voice can be easily broadcasted from one to many via digital media, the sense of touch is limited to direct interactions. What happens if such body sensations can be shared across individuals, in which one feels a touch while watching someone else being touched? In this work, we investigated the impact of such shared body sensations on social interactions in virtual reality (VR). Building upon previous research that used psychophysics methods, our work explores the practical implications of shared body sensations in Social VR, which enables interactions beyond what’s physically possible. We conducted a withingroup user study ($\mathrm{n}=32$) in which participants observed conversations between two virtual agents and shared touch with one of the agents, as shown in Figure 1. Our results showed that even experiencing shared touch sensations several times during a conversation can affect social perception and behavior. Participants reported a stronger body illusion and empathy towards the virtual agent they shared touch with and stood closer to them. These results occurred both with and without a virtual mirror that made participants’ selfavatars more salient. The findings from this study introduce a new technique to enhance social connectedness in VR, and we discuss its applications in various contexts, such as asynchronous communication and collaboration.

Embodiment toward an avatar in virtual reality (VR) is generally stronger when there is a high degree of alignment between the user’s and self-avatar’s motion. However, one-to-one mapping between the two is not always ideal when user interacts with the virtual environment. On these occasions, the user input often leads to unnatural behavior without physical realism (e.g., objects penetrating virtual body, body unmoved by hitting stimuli). We investigate how adding physics correction to self-avatar motion impacts embodiment. Physics-aware self-avatar preserves the physical meaning of the movement but introduces discrepancies between the user’s and self-avatar’s motion, whose contingency is a determining factor for embodiment. To understand its impact, we conducted an in-lab study (n = 20) where participants interacted with obstacles on their upper bodies in VR with and without physics correction. Our results showed that, rather than compromising embodiment level, physics-responsive self-avatar improved embodiment compared to no-physics condition in both active and passive interactions.

With the proliferation of consumer-level virtual reality (VR) devices, users started experiencing VR in less controlled environments, such as in social gatherings and public areas. While the current VR hardware provides an increasingly immersive experience, it ignores stimuli originating from the physical surroundings that distract users from the VR experience. To block distractions from the outside world, many users wear noise-canceling headphones. However, this is insufficient to block loud or transient sounds (e.g., drilling or hammering) and, especially, multi-modal distractions (e.g., air drafts, temperature shifts from an A/C, construction vibrations, or food smells). To tackle this, we explore a new concept, where we directly integrate the distracting stimuli from the user's physical surroundings into their virtual reality experience to enhance presence. Using our approach, an otherwise distracting wind gust can be directly mapped to the sway of trees in a VR experience that already contains trees. Using our novel approach, we demonstrate how to integrate a range of distractive stimuli into the VR experience, such as haptics (temperature, vibrations, touch), sounds, and smells. To validate our approach, we conducted three user studies and a technical evaluation. First, to validate our key principle, we conducted a controlled study where participants were exposed to distractions while playing a VR game. We found that our approach improved users’ sense of presence, compared to wearing noise-canceling headphones. From these results, we engineered a sensing module that detects a set of simple distractive signals (e.g., sounds, winds, and temperature shifts). We validated our hardware in a technical evaluation and in an out-of-lab study where participants played VR games in an uncontrolled environment. Moreover, to gather the perspective of VR content creators that might one day utilize a system inspired by our findings, we invited game designers to use our approach and collected their feedback and VR designs. Finally, we present design considerations for mapping distracting external stimuli and discuss ethical considerations of integrating real-world stimuli into virtual reality.

Electrical muscle stimulation (EMS) is an emergent technique that miniaturizes force feedback, especially popular for untethered haptic devices, such as mobile gaming, VR, or AR. However, the actuation displayed by interactive systems based on EMS is coarse and imprecise. EMS systems mostly focus on inducing movements in large muscle groups such as legs, arms, and wrists; whereas individual finger poses, which would be required, for example, to actuate a user's fingers to fingerspell even the simplest letters in sign language, are not possible. The lack of dexterity in EMS stems from two fundamental limitations: (1) lack of independence: when a particular finger is actuated by EMS, the current runs through nearby muscles, causing unwanted actuation of adjacent fingers; and, (2) unwanted oscillations: while it is relatively easy for EMS to start moving a finger, it is very hard for EMS to stop and hold that finger at a precise angle; because, to stop a finger, virtually all EMS systems contract the opposing muscle, typically achieved via controllers (e.g., PID)—unfortunately, even with the best controller tuning, this often results in unwanted oscillations. To tackle these limitations, we propose dextrEMS, an EMS-based haptic device featuring mechanical brakes attached to each finger joint. The key idea behind dextrEMS is that while the EMS actuates the fingers, it is our mechanical brake that stops the finger in a precise position. Moreover, it is also the brakes that allow dextrEMS to select which fingers are moved by EMS, eliminating unwanted movements by preventing adjacent fingers from moving. We implemented dextrEMS as an untethered haptic device, weighing only 68g, that actuates eight finger joints independently (metacarpophalangeal and proximal interphalangeal joints for four fingers), which we demonstrate in a wide range of haptic applications, such as assisted fingerspelling, a piano tutorial, guitar tutorial, and a VR game. Finally, in our technical evaluation, we found that dextrEMS outperformed EMS alone by doubling its independence and reducing unwanted oscillations.

We propose a haptic device that alters the perceived softness of real rigid objects without requiring to instrument the objects. Instead, our haptic device works by restricting the user's fingerpad lateral deformation via a hollow frame that squeezes the sides of the fingerpad. This causes the fingerpad to become bulgier than it originally was—when users touch an object's surface with their now-restricted fingerpad, they feel the object to be softer than it is. To illustrate the extent of softness illusion induced by our device, touching the tip of a wooden chopstick will feel as soft as a rubber eraser. Our haptic device operates by pulling the hollow frame using a motor. Unlike most wearable haptic devices, which cover up the user's fingerpad to create force sensations, our device creates softness while leaving the center of the fingerpad free, which allows the users to feel most of the object they are interacting with. This makes our device a unique contribution to altering the softness of everyday objects, creating “buttons” by softening protrusions of existing appliances or tangibles, or even, altering the softness of handheld props for VR. Finally, we validated our device through two studies: (1) a psychophysics study showed that the device brings down the perceived softness of any object between 50A-90A to around 40A (on Shore A hardness scale); and (2) a user study demonstrated that participants preferred our device for interactive applications that leverage haptic props, such as making a VR prop feel softer or making a rigid 3D printed remote control feel softer on its button.

We present a personalized, comprehensive eye-tracking solution based on tracking higher-order Purkinje images, suited explicitly for eyeglasses-style AR and VR displays. Existing eye-tracking systems for near-eye applications are typically designed to work for an on-axis configuration and rely on pupil center and corneal reflections (PCCR) to estimate gaze with an accuracy of only about 0.5°to 1°. These are often expensive, bulky in form factor, and fail to estimate monocular accommodation, which is crucial for focus adjustment within the AR glasses.Our system independently measures the binocular vergence and monocular accommodation using higher-order Purkinje reflections from the eye, extending the PCCR based methods. We demonstrate that these reflections are sensitive to both gaze rotation and lens accommodation and model the Purkinje images' behavior in simulation. We also design and fabricate a user-customized eye tracker using cheap off-the-shelf cameras and LEDs. We use an end-to-end convolutional neural network (CNN) for calibrating the eye tracker for the individual user, allowing for robust and simultaneous estimation of vergence and accommodation. Experimental results show that our solution, specifically catering to individual users, outperforms state-of-the-art methods for vergence and depth estimation, achieving an accuracy of 0.3782° and 1.108cm respectively.