With recent advances in information and communication technologies, Hybrid meetings, where local attendees are physically present and remote participants join virtually, have become increasingly common. However, remote participants often experience reduced contextual awareness and a sense of isolation. To address these issues, we propose HybridSphere, a hybrid meeting system that reconstructs a shared virtual reality environment for remote participants. The system employs a 360-degree camera and pose estimation to generate real-time avatar representations of local attendees to allow remote users with head-mounted displays to experience the meeting as if all participants are in the same virtual space. We conducted a user study comparing HybridSphere to a baseline condition in which remote participants viewed an unmodified 360-degree video. Although the avatars were rated lower in perceived trustworthiness and likability due to limited visual fidelity, participants appreciated the seated-6DoF function. These results suggest that immersive and spatially flexible VR representations can enhance remote engagement in hybrid meetings.

Autonomous robot manipulation is a complex and continuously evolving robotics field. This paper focuses on data augmentation methods in imitation learning. Imitation learning consists of three stages: data collection from experts, learning model, and execution. However, collecting expert data requires manual effort and is time-consuming. Additionally, as sensors have different data acquisition intervals, preprocessing such as downsampling to match the lowest frequency is necessary. Downsampling enables data augmentation and also contributes to the stabilization of robot operations. In light of this background, this paper proposes the Data Augmentation Method for Bilateral Control-Based Imitation Learning with Images, called “DABI”. DABI collects robot joint angles, velocities, and torques at 1000 Hz, and uses images from gripper and environmental cameras captured at 100 Hz as the basis for data augmentation. This enables a tenfold increase in data. In this paper, we collected just 5 expert demonstration datasets. We trained the bilateral control Bi-ACT model with the unaltered dataset and two augmentation methods for comparative experiments and conducted real-world experiments. The results confirmed a significant improvement in success rates, thereby proving the effectiveness of DABI. For additional material, please check: https://mertcookimg.github.io/dabi

We present Bi-LAT, a novel imitation learning framework that unifies bilateral control with natural language processing to achieve precise force modulation in robotic manipulation. Bi-LAT leverages joint position, velocity, and torque data from leader-follower teleoperation while also integrating visual and linguistic cues to dynamically adjust applied force. By encoding human instructions such as "softly grasp the cup" or "strongly twist the sponge" through a multimodal Transformer-based model, Bi-LAT learns to distinguish nuanced force requirements in real-world tasks. We demonstrate Bi-LAT’s performance in (1) unimanual cup-stacking scenario where the robot accurately modulates grasp force based on language commands, and (2) bimanual sponge-twisting task that requires coordinated force control. Experimental results show that Bi-LAT effectively reproduces the instructed force levels, particularly when incorporating SigLIP among tested language encoders. Our findings demonstrate the potential of integrating natural language cues into imitation learning, paving the way for more intuitive and adaptive human–robot interaction. For additional material, please visit the website: https://mertcookimg.github.io/bi-lat/

Mobile robot navigation systems are increasingly relied upon in dynamic and complex environments, yet they often struggle with map inaccuracies and the resulting inefficient path planning. This paper presents MRHaD, a Mixed Reality-based Hand-drawn Map Editing Interface that enables intuitive, real-time map modifications through natural hand gestures. By integrating the MR head-mounted display with the robotic navigation system, operators can directly create hand-drawn restricted zones (HRZ), thereby bridging the gap between 2D map representations and the real-world environment. Comparative experiments against conventional 2D editing methods demonstrate that MRHaD significantly improves editing efficiency, map accuracy, and overall usability, contributing to safer and more efficient mobile robot operations. The proposed approach provides a robust technical foundation for advancing human-robot collaboration and establishing innovative interaction models that enhance the hybrid future of robotics and human society. For additional material, please check: https://mertcookimg.github.io/mrhad/

Aerial investigation is used for surveying damage and identifying post-disaster events through imagery data. However, the challenge lies in detecting disaster-related areas within aerial or shipborne images, as these can appear as minor regions, making recognition difficult. To address this challenge, we introduce the Detective Network (DeNet), designed to optimally mask images, thereby shifting the attention of machine learning models towards these small yet crucial regions. Utilizing the concepts of patch and anchor box, DeNet incorporates a masking candidate layer and a masking layer to facilitate optimal masking. Our experimental findings are compelling; by preprocessing images with DeNet before analysis using an image captioning model, we achieved a remarkable accuracy of 92.91% in landslide detection from side-view image captions and 87.50% for shipborne view detection. The result demonstrates the efficacy of DeNet in enhancing the recognition of disaster-related areas in challenging imaging conditions.

Virtual reality (VR) video games that are played on a VR headset are becoming increasingly common in households, and though many games require players to navigate vast virtual spaces, most homes cannot provide a large enough physical space to encompass the entire virtual space. Thus, VR video games that require locomotion often provide users with alternative locomotion techniques. While teleportation or steering is typically used as a standard, new techniques can overcome remaining problems, such as motion sickness. However, a holistic perspective of user needs and issues regarding these techniques in practical situations has not been studied on a broad basis. To address this gap in the literature and contribute to future VR video game development and research, we conducted 16 semi-structured interviews and surveyed 88 participants to help explore issues regarding existing locomotion techniques. Our results revealed preferences related to teleportation versus steering and the postures that users adopt while playing VR video games, along with user needs for locomotion techniques in each posture.

Head tracking allows users of Virtual Reality (VR) to freely rotate their heads 360 degrees while exploring virtual environments. When using VR in a limited space, the ability to physically rotate one's head is limited to a specific range. To address this issue, previous studies have proposed methods to employ distinct rotation factors for real and imaginary rotations. However, its primary usage lies in redirected walking; thus, it is unsuitable for seated VR. In this article, we propose CaliView, which consistently adjusts the user's perspective to always face an optimal direction in VR, all while ensuring a comfortable posture. CaliView continuously controls the rotation gain to ensure that the disparity between the present body orientation and the optimal orientation is reduced to zero, encouraging the user to assume a forward-facing position with the target orientation. To assess the suggested approach, CaliView, we experimented to compare three conditions: CaliView, snap turning only (SnapTurn), and hybrid of CaliView and snap turning (Hybrid). The research findings suggest that CaliView functions as a useful reorientation technique, enabling implicit reorientation without sacrificing the user experience. Additionally, this study showcases its compatibility with various other techniques, such as the traditional snap-turn, thus emphasizing its versatility.

To date, virtual and augmented reality technologies enable users to view multiple, large virtual screens in their workspaces. However, users must frequently rotate their heads to shift focus among these screens. This paper presents GlanXR, a fast and robust handsfree approach for screen switching in virtual reality. GlanXR incorporates a peripheral interface that remains fixed within the user’s view, in which screens can be dynamically selected based on the user’s eye-head position beyond an adaptive range. Additionally, the user triggers the switch to the screen chosen by making an opposing head rotation in the direction of the eye-head position to minimize false triggers. We conducted an experiment including a fast-switching scenario and a working simulation scenario with 24 participants to assess the effectiveness of GlanXR as compared to a baseline (taskbar), an expansive multi-screen setup, and a gazebased screen selection method. The results indicate that GlanXR facilitates precise screen-switching, minimizes the necessity for head rotation, and allows users to maintain a neutral head position.

The COVID-19 pandemic has catalyzed a rapid transformation in higher education, prompting institutions to embrace online learning and e-learning as essential mechanisms for academic continuity. In this context, Interactive Learning Environments (ILEs), augmented with generative AI chatbots, represent a promising approach to enhancing the effectiveness of virtual education and interactive learning. This paper investigates the efficacy of an ILE integrated with ChatGPT in the context of computer networking education. A pilot experiment was conducted with three graduate students with basic IT networking background. The study examined the impact of integrating ChatGPT within the ILE on students’ engagement, comprehension, and overall learning outcomes. Methodological details, including learning program design, learning outcomes assessment, ILE settings, and ChatGPT integration are presented. Results from pre- and post-assessment tests, quizzes, and students’ feedback on ChatGPT’s utility as a learning aid are discussed. The findings suggest a significant improvement in students’ comprehension and performance following their engagement with the ILE while using ChatGPT as a support tool. Despite a few drawbacks reported by the students in terms of the interfaces’ ease of use, and the timely response and appropriate content delivered by ChatGPT, they were overall satisfied with the experience. The students would recommend ChatGPT for learning purposes under certain controlled environments. This study contributes to the growing body of literature on interactive learning technologies and highlights the potential of generative AI chatbots such as ChatGPT to revolutionize computer networking education in the digital age.

Image-to-image translation methods have progressed from only considering the image-level information to integrating the global- and instance-level information. However, only the foreground instances are refined, and the background semantics are taken as an entire feature, which causes a substantial loss of the semantic information in the translation. Additionally, the insufficient quality of the translated semantic regions also leads to an unsatisfactory performance of the object recognition or visual odometry tasks in which the translated images/videos are further used. In this paper, we propose a novel generative adversarial network for panoptic-level image-to-image translation (PanopticGAN). The proposed method has three advantages: 1) the extracted panoptic perception (i.e., the foreground instances and background semantic regions) as content codes are aligned with the sampled panoptic style codes, which considers the panoptic-level information to avoid the semantic information loss, and the latent space of each object has a rich fusion of content and style codes to generate the higher-fidelity results; 2) a feature masking module is proposed to extract the representations within each object contour by masks for sharpening the object boundaries; 3) the improved fidelity of the translated semantic regions further contributes to enhancing the performance of the object recognition or visual odometry tasks that the translated images/videos are used in. In this paper, we also annotate a compact panoptic segmentation dataset for the thermal-to-color translation task. Extensive experiments are conducted to demonstrate the effectiveness of our PanopticGAN over the latest methods.

Despite remarkable progress in image translation, the complex scene with multiple discrepant objects remains a challenging problem. The translated images have low fidelity and tiny objects in fewer details causing unsatisfactory performance in object recognition. Without thorough object perception (i.e., bounding boxes, categories, and masks) of images as prior knowledge, the style transformation of each object will be difficult to track in translation. We propose panoptic-aware generative adversarial networks (PanopticGAN) for image-to-image translation together with a compact panoptic segmentation dataset. The panoptic perception (i.e., foreground instances and background semantics of the image scene) is extracted to achieve alignment between object content codes of the input domain and panoptic-level style codes sampled from the target style space, then refined by a proposed feature masking module for sharping object boundaries. The image-level combination between content and sampled style codes is also merged for higher fidelity image generation. Our proposed method was systematically compared with different competing methods and obtained significant improvement in both image quality and object recognition performance.

In virtual reality, VR sickness resulting from continuous locomotion via controllers or joysticks is still a significant problem. In this article, we present a set of algorithms to mitigate VR sickness that dynamically modulate the user's field of view by modifying the contrast of the periphery based on movement, color, and depth. In contrast with previous work, this vision modulator is a shader that is triggered by specific motions known to cause VR sickness, such as acceleration, strafing, and linear velocity. Moreover, the algorithm is governed by delta velocity, delta angle, and average color of the view. We ran two experiments with different washout periods to investigate the effectiveness of dynamic modulation on the symptoms of VR sickness, in which we compared this approach against a baseline and pitch-black field-of-view restrictors. Our first experiment made use of a just-noticeable-sickness design, which can be useful for building experiments with a short washout period.

This paper presents a 3D pointing interface application to signal a UAV’s target in a large-scale environment. This system enables UAVs equipped with a monocular camera to determine which window of a building is selected by a human user in large-scale indoor or outdoor environments. The 3D pointing interface consists of three parts: YOLO, Open- Pose, and ORB-SLAM. YOLO detects the target objects, e.g., windows, OpenPose extracts the user pose, and ORB-SLAM builds a scale-dependent 3D map, a set of 3D sparse feature points. To obtain the visual scale, it performs a calibration step with the user standing in front of the UAV at a certain distance. We detail how we chose the gesture, localize and detect objects, and transform between coordinate systems. The real- world experiment results showed that the 3D pointing interface obtained a 0.73 F1-score average and a 0.58 F1-Score at the maximum distance of 25 meters between UAV and building.

In this paper, we present a spherical magnetic joint for the inverted locomotion of a multi-legged robot. The permanent magnet’s spherical shape allows the robot to attach its foot to a steel surface without energy consumption. However, the robot’s inverted locomotion requires foot flexibility for placement and gait construction of the robot. Therefore, the spherical magnetic joint mechanism was designed and implemented for the robot feet to deal with angular placement. For decoupling the foot from the steel surface, the attractive force is adjusted by tilting the adjustable sleeve mechanism at an adequate angle between the surface and foot tip. Experimental results show that the spherical magnetic joint can maintain the attractive force at any angle, and the sleeve mechanism can reduce 20% of the reaction force for pulling the legs from the steel surfaces. Furthermore, the designed gait for inverted locomotion with a spherical magnetic joint was tested and compared to prove the concept of the spherical magnetic joint and sleeve mechanism.

In educational settings such as art galleries or museums, Augmented Reality (AR) has the potential to provide detailed information about exhibits. However, dealing with items that contain information in multiple sections or areas is still a significant challenge. For example, a large painting may contain many minute details, which requires a system that can explain its broader features rather than just a generic description. To address this challenge, we introduce an AR guidance system that uses an embodied agent to point out items and explain each piece and part of exhibit items in detail. We also designed and tested 3 different pointing interfaces for the embodied agent: gesture only, gesture with a dot laser, and gesture with line laser. To evaluate this interface, we conducted a user experiment simulating painting guidance to test interest and exhibit memory. During the experiment, the agent pointed to various areas of interest in the painting and provided a detailed description to participants. The result shows that the search times for target positions were the fastest with the line laser. However, no particular interface outperformed others in memory recall of exhibit content.

Drone navigation in complex environments poses many problems to teleoperators. Especially in three dimensional (3D) structures such as buildings or tunnels, viewpoints are often limited to the drone’s current camera view, nearby objects can be collision hazards, and frequent occlusion can hinder accurate manipulation. To address these issues, we have developed a novel interface for teleoperation that provides a user with environment-adaptive viewpoints that are automatically configured to improve safety and provide smooth operation. This real-time adaptive viewpoint system takes robot position, orientation, and 3D point-cloud information into account to modify the user’s viewpoint to maximize visibility. Our prototype uses simultaneous localization and mapping (SLAM) based reconstruction with an omnidirectional camera, and we use the resulting models as well as simulations in a series of preliminary experiments testing navigation of various structures. Results suggest that automatic viewpoint generation can outperform first- and third-person view interfaces for virtual teleoperators in terms of ease of control and accuracy of robot operation.

We present a human-centered designed social drone aiming to be used in a human crowd environment. Based on design studies and focus groups, we created a prototype of a social drone with a social shape, face and voice for human interaction. We used the prototype for a proxemic study, comparing the required distance from the drone humans could comfortably accept compared with what they would require for a nonsocial drone. The social shaped design with greeting voice added decreased the acceptable distance markedly, as did present or previous pet ownership, and maleness. We also explored the proximity sphere around humans with a social shaped drone based on a validation study with variation of lateral distance and heights. Both lateral distance and the higher height of 1.8 m compared to the lower height of 1.2 m decreased the required comfortable distance as it approached.

Low light situations pose a significant challenge to individuals working in a variety of different fields such as firefighting, rescue, maintenance and medicine. Tools like flashlights and infrared (IR) cameras have been used to augment light in the past, but they must often be operated manually, provide a field of view that is decoupled from the operator's own view, and utilize color schemes that can occlude content from the original scene. To help address these issues, we present VisMerge, a framework that combines a thermal imaging head mounted display (HMD) and algorithms that temporally and spectrally merge video streams of different light bands into the same field of view. For temporal synchronization, we first develop a variant of the time warping algorithm used in virtual reality (VR), but redesign it to merge video see-through (VST) cameras with different latencies. Next, using computer vision and image compositing we develop five new algorithms designed to merge non-uniform video streams from a standard RGB camera and small form-factor infrared (IR) camera. We then implement six other existing fusion methods, and conduct a series of comparative experiments, including a system level analysis of the augmented reality (AR) time warping algorithm, a pilot experiment to test perceptual consistency across all eleven merging algorithms, and an in-depth experiment on performance testing the top algorithms in a VR (simulated AR) search task. Results showed that we can reduce temporal registration error due to inter-camera latency by an average of 87.04%, that the wavelet and inverse stipple algorithms were perceptually rated the highest, that noise modulation performed best, and that freedom of user movement is significantly increased with visualizations engaged.

Drone navigation in complex environments poses many problems to teleoperators. Especially in 3D structures like buildings or tunnels, viewpoints are often limited to the drone's current camera view, nearby objects can be collision hazards, and frequent occlusion can hinder accurate manipulation. To address these issues, we have developed a novel interface for teleoperation that provides a user with environment-adaptive viewpoints that are automatically configured to improve safety and smooth user operation. This real-time adaptive viewpoint system takes robot position, orientation, and 3D pointcloud information into account to modify user-viewpoint to maximize visibility. Our prototype uses simultaneous localization and mapping (SLAM) based reconstruction with an omnidirectional camera and we use resulting models as well as simulations in a series of preliminary experiments testing navigation of various structures. Results suggest that automatic viewpoint generation can outperform first and third-person view interfaces for virtual teleoperators in terms of ease of control and accuracy of robot operation.

This paper proposes The Rainbow Marker, a planar marker for estimating the direction of a light source using a structural color. A structural color is a color produced by microscopically structured surfaces that vary in appearance according to the viewpoint, the direction and the spectrum of the light source. The proposed marker contains a planar material which causes structural coloration. The direction of the light source is estimated by structural color pattern matching between an input pattern and referential color patterns. In this paper, two types of the marker were implemented, with a grating sheet and with a holographic sheet, to demonstrate that the proposed method is applicable in the field of augmented reality.

This study proposes a method to estimate elastic deformation using silhouettes obtained from multiple endoscopic images. Our method can estimate the intraoperative deformation of organs using a volumetric mesh model reconstructed from preoperative CT data. We use this elastic body silhouette information of elastic bodies not to model the shape but to estimate the local displacements. The model shape is updated to satisfy the silhouette constraint while preserving the shape as much as possible. The result of the experiments showed that the proposed methods could estimate the deformation with root mean square (RMS) errors of 5.0–10 mm.

A photomosaic is a type of decorative art made up from various other photographs. We present a method for quickly generating photomosaics and propose an interactive recursive photomosaic system. Users can operate the system by using a large display with a touch input function, which allows them to alter the appearance of the image dynamically.

Computer-aided design (CAD) system conventionally have been widely used to support designers for creating, modifying, adding something to or removing something from objects by showing simulated objects on computer screen. These virtual, non-physical, objects have been, however, known as imperfect imitation of reality. The impression of shape is highly related to the second order derivative of geometric feature of the shape. Conventional CAD systems, including AutoCAD, usually have visualization feature of the first derivative (normal) and the second derivative (curvature) of given surfaces. There, however, still have been problems in curvature visualization on the screen. First, it lacks true feeling of physical objects. Second, even if designers were given a physical mock-up object in hand, they wouldn't precisely recognize minute change of curvatures — few designers can sense small differences of curvature and most others need a special device to check the curvature. For solving these problem, the authors propose a novel curvature visualization system based on mixed reality technology. The color mapping according to the Gaussian curvature calculated via a time-of-flight camera provides the observers with intuitively understanding the object's curvature information.

This paper describes a novel game system which provides multiple players with a sense of touching fire with their own hands. Players in this game are divided into two teams in front of a wall-type flat display and try to score points by grounding a fireball on the other team's court like volleyball. The players can recognize their contacts with the fireball from mirrored image of their own appearance and the superimposed fireball in the display. Computer detects those contacts by using a real-time time-of-flight camera and renders flame of the fireball based on fluid simulation. The reason why we choose fire as a ball is that several characteristics of fire are advantages in interactions between users and virtual objects. This game can provide players with enough enjoyment and reality due to these advantages even if the implemented human detection algorithm is quite simple.

This paper proposes a method for rendering an inter-reflection between a marker and a glossy floor in Augmented Reality (AR). At first, a reflectance ratio of the floor is estimated from the reflection on the floor and from the marker box directly. Then the roughness of the floor is estimated based on the sharpness of the reflected marker box image on the floor. Lastly, the marker box reflection is eliminated based on the surrounding colors of the marker box reflection. Rendered images show that natural inter-reflection can be achieved by using the proposed method.