ABSTRACT Co‐speech gestures are essential for natural human communication, yet existing synthesis methods fall short in delivering semantically aligned and contextually appropriate motions. In this paper, we present RIDGE, a hybrid system that combines rule‐based and deep learning approaches to generate realistic gestures for virtual avatars and human‐computer interaction. RIDGE employs a high‐fidelity rule base, generated from motion capture data with the assistance of large language models, to select reliable gesture mappings. When a high‐confidence match is not available, a contrastively trained deep learning model steps in to produce semantically appropriate gestures. Evaluated using a novel Gesture Cluster Affinity (GCA) metric, our system outperforms existing baselines, achieving a GCA score of 0.73 compared to a rule‐based baseline of 0.6 and an end‐to‐end: 0.52, while the ground truth score was 0.90. Detailed analyses of system architecture, data preprocessing, and evaluation methodologies demonstrate RIDGE's potential to enhance gesture synthesis. Project Url: https://www.mrlab.co.kr/research/ridge .

ABSTRACT In this article, we propose an AI‐agent that integrates a large language model (LLM) with a retrieval‐augmented generation (RAG) system to deliver reliable historical information from the Annals of the Joseon Dynasty through both objective facts and contextual analysis, achieving significant performance improvements over existing models. For an AI‐agent using the Annals of the Joseon Dynasty to deliver reliable historical information, clear source citations and systematic analysis are essential. The Annals, an official record spanning 472 years (1392–1897), offer a dense, chronological account of daily events and state administration that shaped Korea's cultural, political, and social foundations. We propose integrating a LLM with a RAG system to generate highly accurate responses based on this extensive dataset. This approach provides both objective information about historical figures and events from specific periods and subjective contextual analysis of the era, helping users gain a broader understanding. Our experiments demonstrate improvements of approximately 23 to 50 points on a 100‐point scale compared with the GPT‐4o and OpenAI AI‐Assistant v2 models.

This study investigates how different virtual agent (VA) behaviors influence subjects’ perceptions and group decision-making. Participants carried out experimental group discussions with a VA exhibiting varying levels of engagement and affective behavior. Engagement refers to the VA’s focus on the group task, whereas affective behavior reflects the VA’s emotional state. The findings revealed that VA’s engagements effectively captured participants’ attention even in the group setting and enhanced group synergy, thereby facilitating more in-depth discussion and producing better consensus. On the other hand, VA’s affective behavior negatively affected the perceived social presence and trustworthiness. Consequently, in the context of group discussion, participants preferred the engaged and non-affective VA to the non-engaged and affective VA. The study provides valuable insights for improving the VA’s behavioral design as a team member for collaborative tasks.

Abstract Paying close attention to facial expressions, gestures, and communication techniques is essential when creating animated physician characters that are realistic and captivating when describing surgical procedures. This paper emphasizes the integration of appropriate emotions, co‐speech gestures when medical experts explain the medical procedure, and designing animated characters. We can achieve healthy doctor‐patient relationships and improvement of patients' understanding by depicting these components truthfully. We suggest two critical approaches to developing virtual medical experts by incorporating these elements. First, doctors can generate the contents of the surgical procedure with a virtual doctor. Second, patients can listen to the surgical procedure described by the virtual doctor and ask if they have any questions. Our system helps patients by considering their psychology and adding medical professionals' opinions. These improvements ensure the animated virtual agent is comforting, reassuring, and emotionally supportive. Through a user study, we evaluated our hypothesis and gained insight into improvements.

In this paper, we applied the concept of diminished reality to remove content-irrelevant pedestrian (i.e., real object) in the context of handheld augmented reality (AR). We prepared three view conditions: in Transparent (TP) condition, we removed the pedestrian entirely; in Semi-transparent (STP) condition, the pedestrian became semi-transparent; lastly, in Default (DF) condition, the pedestrian appeared as is. We conducted a user study to compare the effects of the three conditions on users’ engagement and perception of a virtual pet in the AR content. Our findings revealed that users felt less distracted to the AR content in TP and STP conditions, compared to the DF condition. Furthermore, users felt the virtual pet as more life-like, its behavior more plausible, and felt a higher spatial presence in the real environment, in the TP condition.

We investigate the effects of different ways of visualizing the virtual gait of the avatar in the context of Walk-in-Place (WIP) based navigation in a virtual environment (VE). In Study 1, participants navigated through a VE using the WIP method while inhabiting an avatar. We varied the visualization of the avatar’s leg motion while performing the WIP gesture: (1) Fixed Body: the legs stood still; (2) Pre-recorded Animation: the legs moved in a fixed predetermined pace (plausible but not in accordance to that of the user in general); (3) Synchronized Motion the legs moved according (synchronized) to those of the user. Our results indicated that the sense of presence and body ownership improved significantly when the leg motion was rendered synchronized to that of the user (Synchronized Motion). In addition, we developed a deep neural network (DNN) that predicted the users’ leg postures only with the head position tracking, eliminating the need for any external sensors. We carried out Study 2, to assess the effects of different gait visualizations, under two new factors: (1) virtual gait seen directly by the user looking down, or already visible by one’s shadow (i.e., no need to look down); and (2) playing a pre-recorded animation, or pre-recorded animation whose playback speed was adjusted to match with pace of the users’ actual leg motions as predicted by the DNN. The results of Study 2 showed that the virtual gait temporally synchronized with that of the user greatly improved the sense of body ownership, whether it was witnessed directly or indirectly with the shadow. However, the effect of virtual gait on presence was less marked when indirectly observed. We discuss our findings and the implications for representing the avatar locomotion in immersive virtual environments.

Abstract Interactions with embodied conversational agents can be enhanced using human‐like co‐speech gestures. Traditionally, rule‐based co‐speech gesture mapping has been utilized for this purpose. However, the creation of this mapping is laborious and often requires human experts. Moreover, human‐created mapping tends to be limited, therefore prone to generate repeated gestures. In this article, we present an approach to automate the generation of rule‐based co‐speech gesture mapping from publicly available large video data set without the intervention of human experts. At run‐time, word embedding is utilized for rule searching to get the semantic‐aware, meaningful, and accurate rule. The evaluation indicated that our method achieved comparable performance with the manual map generated by human experts, with a more variety of gestures activated. Moreover, synergy effects were observed in users' perception of generated co‐speech gestures when combined with the manual map.

In this paper, we present the design of a multimodal interaction framework for intelligent virtual agents in wearable mixed reality environments, especially for interactive applications at museums, botanical gardens, and similar places. These places need engaging and no-repetitive digital content delivery to maximize user involvement. An intelligent virtual agent is a promising mode for both purposes. Premises of framework is wearable mixed reality provided by MR devices supporting spatial mapping. We envisioned a seamless interaction framework by integrating potential features of spatial mapping, virtual character animations, speech recognition, gazing, domain-specific chatbot and object recognition to enhance virtual experiences and communication between users and virtual agents. By applying a modular approach and deploying computationally intensive modules on cloud-platform, we achieved a seamless virtual experience in a device with limited resources. Human-like gaze and speech interaction with a virtual agent made it more interactive. Automated mapping of body animations with the content of a speech made it more engaging. In our tests, the virtual agents responded within 2-4 seconds after the user query. The strength of the framework is flexibility and adaptability. It can be adapted to any wearable MR device supporting spatial mapping.

Navigation in the virtual environment is one of the interactions which happens very frequently. Physical locomotion method is popular due to its naturalness in VR but it requires wide area. To use minimum space for the navigation, walk-in-place method is suggested as an alternative way for the virtual environment navigation. In this tutorial, we will cover the following topics. 1) Treadmill based walk-in-place methods 2) Non-treadmill based walk-in-place methods

To improve the telepresence sense of a local HMD user, a high-resolution view of the remote environment is necessary. However, current commodity omnidirectional camera could not support enough resolution for the human eye. Using a higher resolution omnidirectional camera is also infeasible because it will increase the streaming bandwidth. We propose a hybrid camera system that can convey a higher resolution for the HMD user viewport ROI region in available bandwidth range. The hybrid camera consists of a pair of omnidirectional and PTZ camera which is close to each other. The HMD user head orientation controls the PTZ camera orientation. The HMD user also controls the zooming level of the PTZ camera to achieve higher resolution up to PTZ camera maximum optical zoom level. The remote environment view obtained from each camera is streamed to the HMD user and then stitched into one combined view. This combined view simulates human visual system (HVS) phenomenon called foveation, where only a small part in the human view is in high resolution, and the rests are in low resolution.

In this paper, we propose to use the deep learning technique to estimate and predict the torso direction from the head movements alone. The prediction allows to implement the walk-in-place navigation interface without additional sensing of the torso direction, and thereby improves the convenience and usability. We created a small dataset and tested our idea by training an LSTM model and obtained a 3-class prediction rate of about 90%, a figure higher than using other conventional machine learning techniques. While preliminary, the results show the possible inter-dependence between the viewing and torso directions, and with richer dataset and more parameters, a more accurate level of prediction seems possible.

Abstract In this paper, we introduce a bare‐hand interaction method for controlling an augmented virtual character. Our method is utilizing a dynamic hierarchical structure of virtual buttons on the natural marker of the environment. Unlike the existing virtual button method, the proposed method can be used to select a very fine level. This method is adequate for controlling an augmented virtual character with a users' various intentions. We implemented several examples of manipulating an augmented virtual character such as guiding the character, selecting part of the character, and selecting a target object for the character. We also compared the performance of full arrangement and dynamic hierarchical arrangement of virtual buttons. Our method outperforms the full arrangement method, especially in the low index of difficulty. In the user study with 15 participants, the users responded that the proposed method was significantly better than the existing full arrangement method regarding efficiency and satisfaction in the case of the fine level selection.

This paper presents a novel bare-hand interaction method for wearable AR (augmented reality). The suggested method is using hierarchical virtual buttons which are placed on the image target. Therefore, we can provide precise hand interaction on the image target surface (while using wearable AR). The method operates on a wearable AR system and uses an image target tracker to make occlusion-based interaction button. We introduce the hierarchical virtual button method which is adequate for more precise and faster interaction with augmented objects.

There are growing needs for the 3D environment. The 3D devices, such like 3D television, have already started to be sold in markets to the general public. In other words, it means that 3D environment is becoming accustomed to everything. In the same manner, the ways for the 3D manipulation are also getting more important, and there are lots of ways for the 3D manipulation on large screen, which has been used so far, such like 6DOF tracking devices. The problem, however, is that they require high cost, and appropriate place with good environment should be prepared. In this paper, another easier and cheaper way for the 3D manipulation is introduced. The method which is introduced is a way using a mobile device instead of the expensive trackers. Using the method that this paper shows, it is expected that this method is accustomed to everybody and can be used easily.

In this paper, we present "PODIUM (POstech Distributed virtual Music environment)", a distributed virtual environment that allows users to participate in a shared space and play music with other participants in a collaborative manner. In addition to playing virtual instruments, users can communicate and interact in various ways to enhance the collaboration and, thus, the quality of the music played together. Musical messages are generated note by note through interaction with the keyboard, mouse, and other devices, and transmitted through an IP-multicasting network among participants. In addition to such note-level information, additional messages for visualization, and interaction are supported. Real world based visualization has been chosen, against, for instance, abstract music world based visualization, to promote "co-presence" (e.g. recognize and interact with other players), which is deemed important for collaborative music production. In addition to the entertainment purpose, we hope that DVME will find great use in casual practice sessions for even professional performers/orchestras/bands.Since even a slight interruption in the flow of the music or out - of-synch graphics and sound would dramatically decrease utility of the system, we employ various techniques to minimize the network delay. An adapted server-client architecture and UDP' s are used to ensure fast packet deliveries and reduce the data bottleneck problem. Time-critical messages such as MIDI messages are multicasted among clients, and the less time-critical and infrequently updated messages are sent through the server. Predefined animations of avatars are invoked by interpreting the musical messages. Using the latest graphics and sound processing hardware, and by maintaining an appropriate scene complexity, and a frame rate sufficiently higher than the fastest note duration, the time constraint for graphics and sound synchronization can be met. However, we expect the network delay could cause considerable problems when the system is scaled up for many users and processing simultaneous notes (for harmony). To assess the scalability, we carried out a performance analysis of our system model to derive the maximum number of simultaneous participants. For example, according to our data, about 50 participants should be able to play together without significant disruption, each using one track with five simultaneous notes and for playing a musical piece at a speed of 16 ticks per second in a typical PC/LAN environment.In hopes of enhancing the feeling of "co-presence" among participants, a simple sound localization technique is used to compute panning and relative volumes from positions and orientations of participants. This reduced sound localization model is used also in order to minimize the computational cost and the network traffic. Participants can send predefined messages by interacting with the keyboard, mouse, and other input devices. All of the predefined messages are mapped into simple avatar motions, such as playing various types of instruments (players), making applause (audience), and conducting gestures (conductors). We believe that for coordinated music performance, indirect interaction will be the main interaction method, for example, exchanging particular gestures, signals, and voice commands to synchronize music, confirming and reminding expression of the upcoming portion of the music, and just exchanging glances to enjoy each others' emotion. In this view, there would be mainly three groups of participants: conductor, players, and the audience, playing different roles, but creating co-presence together through mutual recognition. We ran a simple experiment comparing the music performance of two groups of participants, one provided with co-presence cues and the other without, and found no performance edge by the group with the co-presence cues. Such a result can serve as one guideline for building music-related VR applications.