Navigating robots in dynamic environments, such as human crowds, is a major challenge due to the trade-off between performance and robustness. Traditional reinforcement learning methods, such as Proximal Policy Optimization (PPO), have shown strong adaptation capabilities but require extensive training and lack explicit mechanisms for collision avoidance. On the other hand, rule-based approaches, such as the Dynamic Window Approach (DWA), offer computational efficiency but struggle with generalization to unseen crowd behaviors. The proposed SafePCA framework aims to address this trade-off by integrating Cellular Automata (CA) into PPO-based navigation. CA enhances robustness by predicting high-risk areas based on pedestrian movement patterns, reducing unnecessary collisions. However, this approach may lead to conservative behavior, potentially affecting navigation performance in reaching the goal efficiently. The core research question addressed in this work is whether SafePCA can balance these trade-offs to ensure safe yet efficient robot navigation in dynamic crowds. Experiments demonstrate that SafePCA outperforms traditional PPO by providing superior risk assessment and avoidance strategies, achieving optimal performance with fewer training episodes. SafePCA's real-time adaptability ensures robust navigation in dynamic environments. By leveraging PPO's adaptive learning and CA's risk analysis, SafePCA offers an efficient solution for autonomous robot navigation in crowded environments, advancing the field and broadening application possibilities.

In this paper, we present a new approach for improving 3D point and line mapping regression for camera re-localization. Previous methods typically rely on feature matching (FM) with stored descriptors or use a single network to encode both points and lines. While FM-based methods perform well in large-scale environments, they become computationally expensive with a growing number of mapping points and lines. Conversely, approaches that learn to encode mapping features within a single network reduce memory footprint but are prone to overfitting, as they may capture unnecessary correlations between points and lines. We propose that these features should be learned independently, each with a distinct focus, to achieve optimal accuracy. To this end, we introduce a new architecture that learns to prioritize each feature independently before combining them for localization. Experimental results demonstrate that our approach significantly enhances the 3D map point and line regression performance for camera re-localization. The implementation of our method will be publicly available at: https://github.com/ais-lab/pl2map/.

Accurate pain expression synthesis is essential for improving clinical training and human-robot interaction. Current Robotic Patient Simulators (RPSs) lack realistic pain facial expressions, limiting their effectiveness in medical training. In this work, we introduce PainDiffusion, a generative model that synthesizes naturalistic facial pain expressions. Unlike traditional heuristic or autoregressive methods, PainDiffusion operates in a continuous latent space, ensuring smoother and more natural facial motion while supporting indefinite-length generation via diffusion forcing. Our approach incorporates intrinsic characteristics such as pain expressiveness and emotion, allowing for personalized and controllable pain expression synthesis. We train and evaluate our model using the BioVid HeatPain Database. Additionally, we integrate PainDiffusion into a robotic system to assess its applicability in real-time rehabilitation exercises. Qualitative studies with clinicians reveal that PainDiffusion produces realistic pain expressions, with a 31.2% ± 4.8% preference rate against ground-truth recordings. Our results suggest that PainDiffusion can serve as a viable alternative to real patients in clinical training and simulation, bridging the gap between synthetic and naturalistic pain expression. Code and videos are available at: https://damtien444.github.io/paindf/.

Effective human behavior modeling is critical for successful human–robot interaction. Current state-of-the-art approaches for predicting listening head behavior during dyadic conversations employ continuous-to-discrete representations, where continuous facial motion sequence is converted into discrete latent tokens. However, nonverbal facial motion presents unique challenges owing to its temporal variance and multimodal nature. State-of-the-art discrete motion token representation struggles to capture underlying nonverbal facial patterns making training the listening head inefficient with low-fidelity generated motion. This study proposes a novel method for representing and predicting nonverbal facial motion by encoding long sequences into a sparse sequence of keyframes and transition frames. By identifying crucial motion steps and interpolating intermediate frames, our method preserves the temporal structure of motion while enhancing instance-wise diversity during the learning process. Additionally, we apply this novel sparse representation to the task of listening head prediction, demonstrating its contribution to improving the explanation of facial motion patterns.

Creating a human-like interface in human-robot interaction is a formidable challenge. Many efforts have been made to mimic the human ability of attentive listening and synchronous participation in conversations, especially in terms of facial expressions and head movements. By taking advantage of transformer-based sequence generation models and quantization techniques, this advantage is further enhanced in the areas of text, video, and audio generation. Using Finite Scalar Quantization, we develop a facial expression tokenization module that is able to encode facial expressions in a finite, semantically meaningful vocabulary. Using this module, we establish a more powerful cross-modality transformer-based, non-deterministic model that is able to learn multiple appropriate facial responses in a dyadic conversational context. 1

Classical structural-based visual localization methods offer high accuracy but face trade-offs in terms of storage, speed, and privacy. A recent innovation, keypoint scene coordinate regression (KSCR) named D2S addresses these issues by leveraging graph attention networks to enhance keypoint relationships and predict their 3D coordinates using a simple multilayer perceptron (MLP). Camera pose is then determined via PnP+RANSAC, using established 2D-3D correspondences. While KSCR achieves competitive results, rivaling state-of-the-art image-retrieval methods like HLoc across multiple benchmarks, its performance is hindered when data samples are limited due to the deep learning model’s reliance on extensive data. This paper proposes a solution to this challenge by introducing a pipeline for keypoint descriptor synthesis using Neural Radiance Field (NeRF). By generating novel poses and feeding them into a trained NeRF model to create new views, our approach enhances the KSCR’s generalization capabilities in data-scarce environments. The proposed system could significantly improve localization accuracy by up to 50% and cost only a fraction of time for data synthesis. Furthermore, its modular design allows for the integration of multiple NeRFs, offering a versatile and efficient solution for visual localization. The implementation is publicly available at: https://github.com/ais-lab/DescriptorSynthesis4Feat2Map.

Recent advancements in visual localization and mapping have demonstrated considerable success in integrating point and line features. However, expanding the localization framework to include additional mapping components frequently results in increased demand for memory and computational resources dedicated to matching tasks. In this study, we show how a lightweight neural network can learn to represent both 3D point and line features, and exhibit leading pose accuracy by harnessing the power of multiple learned mappings. Specifically, we utilize a single transformer block to encode line features, effectively transforming them into distinctive point-like descriptors. Subsequently, we treat these point and line descriptor sets as distinct yet interconnected feature sets. Through the integration of self- and cross-attention within several graph layers, our method effectively refines each feature before regressing 3D maps using two simple MLPs. In comprehensive experiments, our indoor localization findings surpass those of Hloc and Limap across both point-based and line-assisted configurations. Moreover, in outdoor scenarios, our method secures a significant lead, marking the most considerable enhancement over state-of-the-art learning-based methodologies. The source code and demo videos of this work are publicly available at: https://thpjp.github.io/pl2map/.

In this paper, we propose a method to improve the performance of deep gaze estimation using face-shape parameters adapted to a specific target person based on multiple observations. Our gaze estimation network contains a predefined computation module that calculates gaze directions using known geometric relationships among head poses, eye-ball positions, and gaze directions. Updated face-shape parameters contribute to improving the performance of the process. In addition, the computation module enables a network to acquire the ability to induce hidden parameters such as eyeball position and eyeball radius from observed information through a training process. Experimental results reveal improvement in gaze estimation accuracy by introducing a sequential update process for face-shape parameters and a predefined computation module.

Extensive research has been conducted on autonomous surface robots and underwater robots for various tasks in aquatic environments. The duration of the operation of autonomous field robots depends on the capacity of the mounted battery, as they are not typically connected to an external power supply. Therefore, smart strategies which are optimized for each task are required to extend the working time of autonomous field robots. We have developed a symmetrically-shaped au-tonomous surface robot for the long-term monitoring of water quality. In this study, we propose position-keeping strategies to prolong the duration of the symmetrically-shaped surface robot for in-situ monitoring. The proposed position-keeping strategies are evaluated in terms of the power consumption and mean error distance in both practical and simulation environments. The experimental results demonstrate that a robot placed on a water surface with disturbance determines the best course of action to maintain its position based on the environmental conditions and application.

A human patient simulator (HPS) can achieve effective visual-, auditory-, text-, and alarm-based feedback methods in care or nursing education. Among these, the method of visual feedback is important to design an HPS that can express emotions or feelings of pain like an actual human does because this method allows an immediate reaction between robots and humans. This study aims to develop an avatar-based visual feedback method for a care training assistant robot that can express pain states in joint care education. First, this study introduces its own pain facial expression database from Ritsumeikan University (RU-PITENS) for an avatar with pain expression. The RU-PITENS database contains pain images of 41 Japanese people in their 20s, 30s, 40s, and 60s, and an experiment of pain stimulus is conducted based on transcutaneous electrical nerve stimulation, which is low-cost and easy to use in daily life. Based on the pain images in the RU-PITENS database, we generated an avatar with pain expression to achieve the goal of our study. Since the RUPITENS database does not contain the quantitative pain level, the Siamese network was used to calculate the pain intensity. In addition, the care training assistant robot (CaTARo) developed in our previous study reproduces symptoms of musculoskeletal diseases, and the pain of CaTARo was measured using fuzzy logic theory. As a result, a visual feedback system was constructed to express five types of pain (no pain at all, very faint, weak, moderate, and strong pain) with avatars according to the intensity of the pain output of CaTARo in care training environments.

The papers in this special section examine robotic technologies of the fourth industrial revolution or Industry 4.0 that will impact manufacturing industries. The concept of the fourth industrial revolution has drawn attention throughout the world and many efforts to define the concept in diverse fields have continued. Generally, the concept can be summarized as the technology convergence through hyper-intelligence and hyper-connectivity. The core technologies providing the thrust of the fourth industrial revolution, especially in the industrial informatics field, are Internet of Things, robotics, virtual reality, and artificial intelligence. As one of the most critical characteristics of the fourth industrial revolution technology is that the boundary between cyber space and physical space becomes unclear, innovations in industry and business initiate through the fusion of these two spaces. It is the robotic system that plays the key role as a physical medium linking cyber and physical spaces and even changing the physical space through direct interactions. In this sense, robotic system should be recognized as a crucial platform in performing tasks in the cyber-physical space.

We propose a method to restore aspect ratio distortion of images using convolutional neural network (CNN). The “aspect ratio,” which is focused on this research, means degree of horizontal stretching of images. Indeed an image can be distorted by vertical or horizontal stretching, which does not maintain the aspect ratio. In the proposed method, we construct an aspect ratio estimator whose input is a (possibly distorted) image and output is a scalar value of aspect ratio. Since estimation of aspect ratio from image can be regarded as regression problem, we modeled the estimator by CNN. Once we have a reliable estimate of aspect ratio of an image, the restoration can be done straightforwardly by inverse stretching. In the experiments, we evaluated performance of the model trained on Pascal VOC natural image dataset. Our method can precisely restore the distortion within 1.4% of stretch from original images on average, which outperforms average human performance (i.e., about 13%). In terms of accuracy, 99.86% of distorted images are successfully restored. We also propose training methods to enhance the robustness of the CNN against particular types of disturbance.

Marine and lake monitoring have been received a lot of attention. We are developing a sensing device that can keep a fixed position on water autonomously for long term environmental measurement. The concepts of the device is low power consumption, low cost, omni-directional move and portable design. In this paper, we present a long-term surface monitoring system and a new prototype design of the sensing device for the monitoring system. The proposed device is aiming to move to arbitrary directions and keep its positions autonomously on ocean or lake. In the several modular experiments, we confirmed the availability and some problems of the device.

In this paper, we present two new methods to integrate latent Dirichlet allocation (LDA) which is a topic model for surgical workflow phase estimation with a hidden Markov model (HMM). The proposed methods are able to detect surgical phases automatically based on codebook which is built by quantizing the extracted optical flow vectors from the recorded videos of surgical processes. To detect the current phase at a given time point of an operation, some sets of training data with correct phase labels need to be learned by LDA. All documents which are actually short clips divided from the recorded videos are presented as mixtures over learned latent topics. These presentations are then quantized as observed values of a HMM. The major difference between two proposed methods is that while the first method quantizes all topic-based presentations based on k-means, the second method does this based on multivariate Gaussian mixture model. A Left to Right HMM is appropriate for this work because there is no switching the order between surgical phases.

We propose a camera-projector module based space figure understanding support system. The system is able to create the 3D model of a space figure automatically by visual sensing of the drawn space figure on a plane such as a paper. Furthermore, the system adopted a new hardware and an user interface unlike existing system. The system is composed of following three parts; Recognition part, User Interface part and Display part. In recognition part, the system captures the space figure image and builds 3D model by estimating the figure. The 3D model is built by optimization after detecting apparent vertices, edges and faces. In user interface part, the system detects the positions of tangible interfaces placed by a user. In display part, the system determines where to project the information autonomously and projects 3D images. In this paper, we confirmed that users don't feel displeasure at the proposed system, and the system supports user's understanding of space figures.

Reconfigurable Intelligent Space is a smart environment that can rearrange all the devices on the wall and the ceiling since the devices are mounted on the mobile modules and the smart space controls them. However to rearrange the devices optimally, the optimal position and the collision-free path to rearrange the devices are very important. In this paper, the fundamental algorithms, which find the optimal position and generate the path to optimal position, are proposed. The algorithm is verified by simulation experiment with camera mounted mobile module.

In this video, the FRC (Future Robotic Computer) and its demonstrations are introduced. The FRC is a new concept of computer by combining cameras as input device and image projectors as output device. It also has actuators to control the input and the output devices. The selected demonstrations will show the applications of various situations.

In this video, we propose a new interaction method using mobile devices in Intelligent Space (iSpace). This interaction, called R-Fii (Real-world Flexible Interaction Interface), uses mobile devices, mainly cellular phones, as stable interface channels to iSpace. The users are able to control any objects and get information of the objects in iSpace with R-Fii.

In this paper, the concept of R+iSpace and the mechanical architecture of mobile module MoMo for the R+iSpace is introduced. The R+iSpace denotes `Reconfigurable Intelligent Space'. The R+iSpace is able to rearrange every devices in the space according to situation of the space, and it reconfigures the system itself. For rearranging devices, which includes sensors, projectors, etc, the mobile module MoMo is proposed. With the MoMo and a little modification of environment, the R+iSpace can be achieved.

A spatial history storing system is proposed in this paper. In usual cases, what we experienced is left in our brain memories or other static media such as diary, video tape, CD, etc. However, our memories are too inaccurate to put confidence in and static media stored information is fragmentary and hard to search. The proposed system store analytic information of what happened in a space and the space based on the intelligent space functions is able to solve above problems.

This paper describes a projection based information display system which is called Ubiquitous Display. In usual cases, a man should approach information sources which are located around our living environment; e.g. bulletin boards, artificial signs, local maps, etc. However, the Ubiquitous Display is able to afford a human with relevant information by projecting it on where the human is looking such as a wall, a floor and a door so that the human does not need to move for seeking information. The Ubiquitous Display can make a user see 3D objects as well as 2D objects with naked eyes. Anamorphosis technique is adopted for a human to perceive a 3D structure from a 2D image by psychological effect. In this paper and video, how the Ubiquitous Display has been developed and its functions are introduced.

The Intelligent Space is a space where we can easily interact with computers and robots, and get useful service from them. To achieve such a space, a distributed intelligent networked device (DIND) has been proposed. Many DINDs are installed in a space and they cooperate each other to make the space to become an Intelligent Space. In this paper, an optimal DIND placement, self-calibration of DIND and handover protocol for cooperation among DINDs, are described.

In this paper, the Intelligent Space concept is applied for helping disabled or blind persons in crowded environments such as train stations, or airports. The main contribution of this paper is a general mathematical (fuzzy-neuro) description of obstacle avoidance method (walking habit) of moving objects (human beings) in a limited area scanned by the Intelligent Space. A mobile robot with extended functions is introduced as a mobile haptic interface, which is assisted by the Intelligent Space. The mobile haptic interface can guide and protect a blind person in a crowded environment with the help of the Intelligent Space. The Intelligent Space learns the obstacle avoidance method (walking habit) of dynamic objects (human beings) by tracing their movements and helps to the blind person to avoid the collision. The prototype of the mobile haptic interface and simulations of some basic types of obstacle avoidance method (walking habit) are presented.

A resource sharing architecture (RSA) is a robot architecture for a mobile robot in a networked and intelligent space. A robot based on a RSA performs its tasks by sharing many external resources around it and thus has a simple, flexible and low priced structure. However, RSA has a problem as it must guarantee a real-time condition, because it is designed to use the network protocol JINI. For an example, a time delay occurs in transferring data from a vision service to a robot control service. In this paper, we propose a data accessing service (DAS) to solve this problem. It is a service of a network protocol JINI, which transfers data between two services. Its basic function is to find a data source service that a data destination service needs, and after doing this, the DAS transfers data between two services using not JINI, but another network protocol such as TCP/IP or UDP. As a result, we can reduce the time delay to access data. Some experimental results show the validity of our proposition.

Intelligent space is a space where many sensors and intelligent devices are distributed. Mobile robots exist in this space as physical agents, which provide human with services. To realize this, human and mobile robots have to approach each other as much as possible. Moreover, it is necessary for them to perform interactions naturally. It is desirable for a mobile robot to carry out human-affinitive movement. In this research, a mobile robot is controlled by the Intelligent Space through its resources. The mobile robot is controlled to follow walking human as stably and precisely as possible.

The intelligent space is a space where we can easily interact with computers and robots, and get useful service from them. In such a space, location information is very important, since the agents cannot provide proper service to a proper people at a proper location without location information. Our positioning system uses CCD cameras. It is important to determine where to put the cameras for the best localization depending on the tasks in the space. The main purpose of this paper is to consider how to arrange cameras for the best localization in the intelligent space.

Intelligent Space is a space where many sensors and intelligent devices are distributed. Mobile robots exist in this space as physical agents, which provide human with services. To realize this, human and mobile robots have to approach each other as much as possible. Moreover, it is necessary for them to perform interactions naturally. It is desirable for a mobile robot to carry out human-affinitive movement. In this research, a mobile robot is controlled by the Intelligent Space through its resources. The mobile robot is controlled to follow a walking human as stably and precisely as possible.

In this paper, by using network protocol Jini, we propose an advanced resource sharing architecture (RSA) for a mobile robot. We made several services with all resources around the robot by using Jini. There is a task managing service for a certain task, and the service uses other service to perform a task as well as make a new service of other services. To verify our architecture we built a simple experimental space. In this space we performed the experiments such as a detection of desired objects, path generation and motion control for a robot, obstacle avoidance, and so on. Experimental results show that the mobile robot with our advanced RSA performs various tasks successfully.

We propose a method of guiding mobile robots in a networked space. To watch human and robots, distributed sensor devices with processors are located around the networked space. In this space, robots as well as the human are supported informatively and physically. The distributed sensor devices guide mobile robots in this space and navigation with high adaptability is realized. The simulation and experimental results including the camera arrangement and robot guidance with distributed sensor devices are shown.

This paper describes our concept on the intelligent space. The intelligent spaces are rooms or areas that are equipped with sensors, which enable the spaces to perceive and understand what is happening in them. By using such features, people or systems in the intelligent space are supported and can use additional functions. The intelligent spaces are expected to have a broad range of applications such as in homes, offices, factories etc. The basic components, which compose our experimental system, are shown in this paper with descriptions.

Usually localization in mobile robots is interested in only the geographical position of the robot in space. However, to utilize mobile robots in general environments such as a hospital, an office, a school etc., we need more than conventional localization. We propose a new localization method that watches the entire space as well as the mobile robot. Some experimental results are shown to show the performance and the merits of proposed localization method.

A new concept for constructing an intelligent mobile system is proposed. We describe reasons for the necessity of a new architecture for mobile systems in intelligent spaces. The intelligent spaces are room or area that are equipped with sensors, network and computers. The intelligent spaces are expected to be authentic future environment. If environment gets intelligence, it is not a distinct part of an intelligent mobile system any more and also the mobile system becomes a physical agent of the intelligent space. In this paper, we do some experiments to show what are possible for a mobile robot in an intelligent space.

Intelligent spaces are rooms or areas that are equipped with sensors such as microphones or cameras that enable them to perceive what is happening in them. In such spaces that have an intelligence of their own a world model no longer is something the robot has alone but a service offered by the information infrastructure of the space. In this article we show how such an intelligent space can generate a topological map for robots by looking at the movements of people in the room. We describe the stereo vision system that is capable of tracking the 3D movements of several humans in real time and give experimental results obtained in a real-world environment with several people.