We address the challenge of multi-agent cooperation, where agents achieve a common goal by cooperating with decentralized agents under complex partial observations. Existing cooperative agent systems often struggle with efficiently processing continuously accumulating information, managing globally suboptimal planning due to lack of consideration of collaborators, and addressing false planning caused by environmental changes introduced by other collaborators. To overcome these challenges, we propose the RElevance, Proximity, and Validation-Enhanced Cooperative Language Agent (REVECA), a novel cognitive architecture powered by GPT-4o-mini. REVECA enables efficient memory management, optimal planning, and cost-effective prevention of false planning by leveraging Relevance Estimation, Adaptive Planning, and Trajectory-based Validation. Extensive experimental results demonstrate REVECA's superiority over existing methods across various benchmarks, while a user study reveals its potential for achieving trustworthy human-AI cooperation.

In virtual reality (VR), it is widely assumed that increased realism in hand-object interactions enhances user immersion and overall experience. However, recent studies challenge this assumption, suggesting that faithfully replicating real-world physics and visuals is not always necessary for improved usability or immersion. This has led to ambiguity for developers when choosing optimal hand interaction methods for different applications. Currently, there is a lack of comprehensive research to resolve this issue. This study aims to fill this gap by evaluating three contemporary VR hand interaction methods—Attachment, Penetration, and Torque—across two distinct task scenarios: simple manipulation tasks and more complex, precision-driven tasks. By examining key technical features, we identify the strengths and limitations of each method and propose development guidelines for future advancements. Our findings reveal that while Attachment, with its simplified control mechanisms, is well-suited for commercial applications, Penetration and Torque show promise for next-generation interactions. The insights gained from our study provide practical guidance for developers and researchers seeking to balance realism, usability, and user satisfaction in VR environments.

With the increase of virtual reality (VR) applications in daily life, protecting the comfort and health of VR users has become increasingly important. The immersive nature of VR often results in decreased eye blinking and movement, putting users at risk of developing conditions such as dry eye syndrome and eye strain. In this article, we propose visual interfaces to induce temporary eye blinks or movements by drawing users' attention temporarily in order to mitigate the negative effects of VR on eye health. Our proposed interfaces can induce eye blinking and movement, which are known to mitigate eye problems in VR. The experimental results confirmed that our interfaces increase the frequency of eye blinking and movement in VR users.

Marker-based optical motion capture (mocap) systems are increasingly utilized for acquiring 3D human motion, offering advantages in capturing the subtle nuances of human movement, style consistency, and ease of obtaining desired motion. Motion data acquisition via mocap typically requires laborious marker labeling and motion reconstruction, recent deep-learning solutions have aimed to automate the process. However, such solutions generally presuppose a fixed marker configuration to reduce learning complexity, thereby limiting flexibility. To overcome the limitation, we introduce DAMO, an end-to-end deep solver, proficiently inferring arbitrary marker configurations and optimizing pose reconstruction. DAMO outperforms state-of-the-art like SOMA and MoCap-Solver in scenarios with significant noise and unknown marker configurations. We expect that DAMO will meet various practical demands such as facilitating dynamic marker configuration adjustments during capture sessions, processing marker clouds irrespective of whether they employ mixed or entirely unknown marker configurations, and allowing custom marker configurations to suit distinct capture scenarios. DAMO code and pretrained models are available at https://github.com/CritBear/damo .

Abstract Holography stands at the forefront of visual technology, offering immersive, three‐dimensional visualizations through the manipulation of light wave amplitude and phase. Although generative models have been extensively explored in the image domain, their application to holograms remains relatively underexplored due to the inherent complexity of phase learning. Exploiting generative models for holograms offers exciting opportunities for advancing innovation and creativity, such as semantic‐aware hologram generation and editing. Currently, the most viable approach for utilizing generative models in the hologram domain involves integrating an image‐based generative model with an image‐to‐hologram conversion model, which comes at the cost of increased computational complexity and inefficiency. To tackle this problem, we introduce P‐Hologen, the first end‐to‐end generative framework designed for phase‐only holograms (POHs). P‐Hologen employs vector quantized variational autoencoders to capture the complex distributions of POHs. It also integrates the angular spectrum method into the training process, constructing latent spaces for complex phase data using strategies from the image processing domain. Extensive experiments demonstrate that P‐Hologen achieves superior quality and computational efficiency compared to the existing methods. Furthermore, our model generates high‐quality unseen, diverse holographic content from its learned latent space without requiring pre‐existing images. Our work paves the way for new applications and methodologies in holographic content creation, opening a new era in the exploration of generative holographic content. The code for our paper is publicly available on https://github.com/james0223/P-Hologen .

This study aims to allow users to perform dexterous hand manipulation of objects in virtual environments with hand-held VR controllers. To this end, the VR controller is mapped to the virtual hand and the hand motions are dynamically synthesized when the virtual hand approaches an object. At each frame, given the information about the virtual hand, VR controller input, and hand-object spatial relations, the deep neural network determines the desired joint orientations of the virtual hand model in the next frame. The desired orientations are then converted into a set of torques acting on hand joints and applied to a physics simulation to determine the hand pose at the next frame. The deep neural network, named VR-HandNet, is trained with a reinforcement learning-based approach. Therefore, it can produce physically plausible hand motion since the trial-and-error training process can learn how the interaction between hand and object is performed under the environment that is simulated by a physics engine. Furthermore, we adopted an imitation learning paradigm to increase visual plausibility by mimicking the reference motion datasets. Through the ablation studies, we validated the proposed method is effectively constructed and successfully serves our design goal. A live demo is demonstrated in the supplementary video.

This article introduces an interaction method allowing virtual reality (VR) users to interact with virtual objects by blowing air. The proposed method allows users to interact with virtual objects in a physically plausible way by recognizing the intensity of the wind generated by the user's actual wind blowing activity in the physical world. This is expected to provide immersed VR experience since it enables users to interact with virtual objects in the same way they do in the real world. Three experiments were carried out to develop and improve this method. In the first experiment, we collected the user's blowing data and used it to model a formula to estimate the speed of the wind from the sound waves obtained through a microphone. In the second experiment, we investigated how much gain can be applied to the formula obtained in the first experiment. The aim is to reduce the lung capacity required to generate wind without compromising physical plausibility. In the third experiment, the advantages and disadvantages of the proposed method compared to the controller-based method were investigated in two scenarios of blowing a ball and a pinwheel. According to the experimental results and participant interview, participants felt a stronger sense of presence and found the VR experience more fun with the proposed blowing interaction method.

Abstract The rapid evolution of augmented reality (AR) technology has presented new opportunities in the domain of education. Acting as a bridge between the virtual and real worlds, AR technology overcomes the physical limitations of our classrooms at a low cost and provides an interactive learning experience. In this study, we present a smartphone AR application, named the AR‐E‐Helper, which assists the learning of students in higher education lectures. Our goal is to provide an AR‐enhanced learning experience for students. To validate the effectiveness of the AR‐E‐Helper, we conducted an experiment that compares three classes: AR‐enhanced, smartphone‐enhanced, and nontechnology‐enhanced classes. Through the experiment, we observed that our application was helpful in maintaining student's focus in class, promoting their interest, and increasing their satisfaction. Furthermore, we also found how to improve our application based on the observations that the application brought some downsides to the learning activities. We expect that this study will be helpful to design AR learning tools in the future.

In a one-to-many mixed reality collaboration environment, where multiple local users wearing AR headsets are supervised by a remote expert wearing a VR HMD, we evaluated three view-sharing techniques: 2D video, 360 video, and 3D model augmented with 2D video. Through a pilot test, the weaknesses of the techniques were identified, and additional features were integrated into them. Then, their performances were compared in two different collaboration scenarios based on search and assembling. In the first scenario, a local user performed both search and assembling. In the second scenario, two local users had dedicated roles, one for search and the other for assembling. The experiment results showed that the 3D model augmented with 2D video was time-efficient, usable, less demanding and most preferred in one-to-many mixed reality collaborations.

In this paper, we propose to reproduce drag forces in a virtual underwater environment. To this end, we first compute the drag forces to be exerted on human limbs in a physically correct way. Adopting a pseudo-haptic approach that generates visual discrepancies between the real and virtual limb motions, we compute the extent of drag forces that are applied to the virtual limbs and can be naturally perceived. Through two tests, our drag force simulation method is compared with others. The results show that our method is effective in reproducing the sense of being immersed in water. Our study can be utilized for various types of virtual underwater applications such as scuba diving training and aquatic therapy.

In a cable-driven suspension system developed to simulate the reduced gravity of lunar or Martian surfaces, we propose to manipu-late/reduce the physical cues of forward jumps so as to overcome the limited workspace problem. The physical cues should be manipulated in a way that the discrepancy from the visual cues provided through the HMD is not noticeable by users. We identified the extent to which forward jumps can be manipulated naturally. We combined it with visual gains, which can scale visual cues without being noticed by users. The test results obtained in a prototype application show that we can use both trajectory manipulation and visual gains to overcome the spatial limit. We also investigated the user experiences when making significantly high and far jumps. The results will be helpful in designing astronaut-training systems and various VR entertainment content.

This paper presents an obstacle detection and alert system for the pedestrians who use smartphone AR applications. The system analyzes the input camera image to extract feature points and determines whether the feature points come from obstacles ahead in the path. With the obstacle detector, two experiments were made. The first investigated the obstacle alert interfaces, and the second investigated the orientation guide interfaces that instruct users to hold their smartphones with some angles/orientations appropriate to capture the environment. Then, the best interfaces identified from the experiments were integrated and tested to examine their usability and user experiences.

This paper presents flotation simulation in a cable-driven virtual environment. For this, a virtual parasailing system was developed, where the visual stimulus was provided through a VR headset and the physical stimulus was given by wires. In order to prevent the user from moving out of the limited workspace of the cable-driven system, the visual acceleration was washout-filtered to produce the physical acceleration. In the parasailing trajectory, we focused on the stages of vertical acceleration/deceleration and conducted an experiment to identify how much gain can be applied to the visual acceleration, which makes the user feel the natural self-motion when integrated with physical stimulus. Then, the results were tested using several types of full-course virtual parasailing. The results showed that fairly large differences between visual and physical stimuli would be accepted and different gains could be assigned depending on the user's altitudes.