The integration of generative AI into immersive prototyping tools offers unprecedented potential to accelerate VR content creation. However, a critical gap exists in understanding the most effective interaction paradigms. We present DreamCraft, a novel VR prototyping system designed to integrate two fundamental workflows: scene-level creation, where users generate entire, navigable environments, and object-level creation, where scenes are iteratively populated with AI-generated assets. A formal user study with 12 VR designers was conducted to evaluate these workflows, evaluating both workflows through qualitative feedback and quantitative metrics, including task completion time, creative exploration, and perceived cognitive load. Our primary contribution is a set of empirically-grounded design guidelines derived from this analysis. These guidelines provide a crucial framework for the development of next-generation, user-centric generative tools for immersive virtual environments, directly informing future research and system design in the field.

Previous works have shown the potential of immersive technologies to make physical activities a more engaging experience. With encountered-type haptic feedback, users can perceive a more realistic sensation for exertion interaction in substitutions reality. Although substitutional reality has utilized physical environments, props, and devices to provide encountered-type haptic feedback, these cannot withstand the fierce force of humans and do not give feedback when users move around simultaneously, such as in combat sports. In this work, we present Hit Around, a substitutional moving robot for immersive and exertion interaction, in which the user can move and punch the virtual opponent and perceive encountered-type haptic feedback anywhere. We gathered insight into immersive exertion interaction from three exhibitions with iterative prototypes, then designed and implemented the hardware system and application. To understand the ability of mobility and weight loading, we conducted two technical evaluations and a laboratory experiment to validate the feasibility. Finally, a field deployment study explored the limitations and challenges of developing immersive exertion interaction with encountered-type haptics.

Cinematic virtual reality (CVR) brings viewers a novel and immersive movie-watching experience. However, they may miss story events and scene transitions that the director has designed as key points hidden in the VR scene. In this paper, we introduce implicit gaze-based interaction for enhancing the exploration experience in CVR. In contrast to most research on gaze-based selection of objects or explicit guidance of attention using visual cues, we focus on implicit interaction that utilizes the user’s natural gaze and attention to explore the scene. We design and implement different gaze trigger methods for implicit interaction, making the interaction more intuitive and natural when users reveal the invisible objects. We implemented the adaptive collider technique, offering users a better sense of exploration than raycasting and spotlight techniques. We have also conducted user studies to compare animation sequences for visual feedback, with each animation sequence offering different storytelling techniques. One of the sequences is better suited for describing spaces in the virtual world, while the other sequence offers users the feeling of constructing a world through their gaze.

To enrich the emotional experiences of virtual reality (VR) online audiences in music performances, we developed TelEmoScatter, a system that facilitates remote interaction between music performers and onsite audiences. Our system also fosters emotional connections for online audiences through sound-visualization conversion, which is influenced by the state of the onsite audiences using computer vision techniques. In this work, we generate a 3D space using real-time sound-visualization techniques by converting MIDI signals from musical instruments into dynamic animations. Additionally, we employ video analysis to predict the emotions of the onsite audience, allowing seamless integration of emotional visual cues into the virtual scene. With our system, users can effortlessly immerse themselves in the emotional expressions of performers through music and experience the unique atmosphere of a live performance venue simply by wearing a VR headset.

Utilizing haptic devices to enhance the immersive experience is a direct approach in virtual reality (VR) applications. Various studies develop haptic feedback using force, wind, and thermal mechanisms. However, most haptic devices simulate feedback in dry environments such as the living room, prairie, or city. Water-related environments are thus less explored, for example, rivers, beaches, and swimming pools. In this paper we present GroundFlow, a liquid-based haptic floor system for simulating fluid on the ground in VR. We discuss design considerations and propose a system architecture and interaction design. We conduct two user studies to assist in the design of a multiple-flow feedback mechanism, develop three applications to explore the potential uses of the mechanism, and consider the limitations and challenges thereof to inform VR developers and haptic practitioners.

This paper explores directional kinesthetic force feedback on feet for enhancing foot-related experiences in virtual reality. We present Push-Ups, a pair of pneumatic shoes capable of outputting motions to users’ feet soles via the shape-forming shoe bases, to enhance users’ kinesthetic experience in a virtual ski. Each Push-Ups shoe comprises four pneumatic muscles in the base, allowing to present ten shape patterns and thus ten different motion outputs. Our first study validated the effectiveness of the motion output through a recognition study. The second study evaluated the user experience of the system on a virtual ski experience, which demonstrated that users perceived higher enjoyment and realism with the kinesthetic force feedback provided by Push-Ups.

Virtual reality (VR) has been a promising technique to provide an immersive experience. Comparing to traditional multimedia, when users want to take a rest or change the position such as sitting down, the chair might not be sittable because of the inconsistency between the physical and the virtual chair. In this work, we utilized a tracker attached to a physical chair and conducted a user study to explore the sitting behavior when they interact with different forms of the virtual chair. Results indicate that visualizing each part of the chair could provide different information and affect trust and preference.

Tai Chi Chuan (TCC) is a famous Chinese martial art. In addition to reading instruction books and watching demonstration videos, the traditional way of learning TCC for most people is to observe and mimic the movements of the coach. However, it is not easy for a novice to determine the weight distribution of the coach, and it is also difficult for a novice to strike a pose with the correct weight distribution. To help people to learn correct weight distribution when practicing TCC, we proposed a TCC augmented reality (AR)/mixed reality (MR) learning system that consists of a mixed reality head-mounted display (MR-HMD) and a pair of pressure sensing insoles. We have designed three kinds of visual hints to help people strike poses with correct weight distributions. Two user studies have shown that our TCC learning system is helpful for people to learn the weight distribution correctly and helpful for people to improve their proprioceptive sensitivity on weight distribution.

We propose a real-time excavation simulation system for interactive gameplay in Virtual Reality. In order to increase the player's immersion, our simulation system will produce actual potholes and clods according to the depth and angle of the players excavation. We divide the process into three phases: ground deformation, clod generation and clod fragmentation. In ground deformation, we describe how to simulate the topographic changes before and after excavation. In clod generation, we describe how to generate the clod which mesh matches the depth and angle of the players excavation action. In clod fragmentation, the clods are broken and fall as the shovel lifts. This simulation system can create excavation effects on different geologies by changing the material of the ground and clods.

During interaction in a virtual environment, haptic displays provide users with sensations such as vibration, texture simulation, and electrical muscle stimulation. However, as humans perceive object weights naturally in daily life, objects picked up in virtual reality feel unrealistically light. To create an immersive experience in virtual reality that includes weight sensation, we propose GravityCup, a liquid-based haptic feedback device that simulates realistic object weights and inertia when moving virtual handheld objects. In different scenarios, GravityCup uses liquid to provide users with a dynamic weight sensation experience that enhances interaction with handheld objects in virtual reality.

In this paper, we present Haptic Around, a hybrid-haptic feedback system, which utilizes fan, hot air blower, mist creator and heat light to recreate multiple tactile sensations in virtual reality for enhancing the immersive environment and interaction. This system consists of a steerable haptic device rigged on the top of the user head and a handheld device also with haptics feedbacks to simultaneously provide tactile sensations to the users in a 2m x 2m space. The steerable haptic device can enhance the immersive environment for providing full body experience, such as heat in the desert or cold in the snow mountain. Additionally, the handheld device can enhance the immersive interaction for providing partial body experience, such as heating the iron or quenching the hot iron. With our system, the users can perceive visual, auditory and haptic when they are moving around in virtual space and interacting with virtual object. In our study, the result has shown the potential of the hybrid-haptic feedback system, which the participants rated the enjoyment, realism, quality, immersion higher than the other.

When people eat, the taste is very complex and be influenced easily by other senses. Such as visual, olfactory, and haptic, even past experiences, can affect the human perception, which in turn creates more taste possibilities. We present TransFork, an eating tool with olfactory feedback, which augments the tasting experience with video see-through head-mounted display. Additionally, we design a recipe via preliminary experiments to find out the taste conversion formula, which could enhance the flavor of foods and change the user perception to recognize the food. In this demonstration, we prepare a mini feast with bite-sized fruit, the participants use the TransFork to eat food A and smell the scent of food B stored at the aromatic box via airflow guiding. Before they deliver the food to their mouth, the head-mounted display augmented the color of food B on food A by the QR code on the aromatic box. With this augmented reality techniques and the recipe, the tasting experience could be augmented or enhanced, which is a potential approach and could be a playful used for eating.