In artificial intelligence (AI), many legal conflicts have arisen, especially concerning privacy and copyright associated with training data. When an AI model's training data incurs privacy concerns, it becomes imperative to develop a new model devoid of influences from such contentious data. However, retraining from scratch is often not viable due to the extensive data requirements and heavy computational costs. Machine unlearning presents a promising solution by enabling the selective erasure of specific knowledge from models. Despite its potential, many existing approaches in machine unlearning are based on scenarios that are either impractical or could lead to unintended degradation of model performance. We utilize the concept of weight prediction to approximate the less-learned weights based on observations about further training. By repetition of 1) finetuning on specific data and 2) weight prediction, our work gradually eliminates knowledge about the specific data. We verify its ability to eliminate side effects caused by problematic data and show its effectiveness across various architectures, datasets, and tasks.

Guide dog robots offer promising solutions to enhance mobility and safety for visually impaired individuals, addressing the limitations of traditional guide dogs, particularly in perceptual intelligence and communication. With the emergence of Vision-Language Models (VLMs), robots are now capable of generating natural language descriptions of their surroundings, aiding in safer decision-making. However, existing VLMs often struggle to accurately interpret and convey spatial relationships, which is crucial for navigation in complex environments such as street crossings. We introduce the SpaceAware Instruction Tuning (SAIT) dataset and the Space-Aware Benchmark (SA-Bench) to address the limitations of current VLMs in understanding physical environments. Our automated data generation pipeline focuses on the virtual path to the destination in 3D space and the surroundings, enhancing environmental comprehension and enabling VLMs to provide more accurate guidance to visually impaired individuals. We also propose an evaluation protocol to assess VLM effectiveness in delivering walking guidance. Comparative experiments demonstrate that our space-aware instruction-tuned model outperforms state-of-the-art algorithms. We have fully opensourced the SAIT dataset and SA-Bench, along with the related code, at https://github.com/byungokhan/Space-awareVLM.

Robotic catching has traditionally focused on single-handed systems, which are limited in their ability to handle larger or more complex objects. In contrast, bimanual catching offers significant potential for improved dexterity and object handling but introduces new challenges in coordination and control. In this paper, we propose a novel framework for learning dexterous bimanual catching skills using Heterogeneous-Agent Reinforcement Learning (HARL). Our approach introduces an adversarial reward scheme, where a throw agent increases the difficulty of throws-adjusting speed-while a catch agent learns to coordinate both hands to catch objects under these evolving conditions. We evaluate the framework in simulated environments using 15 different objects, demonstrating robustness and versatility in handling diverse objects. Our method achieved approximately a$2 x$increase in catching reward compared to single-agent baselines across$\mathbf{1 5}$diverse objects.

Large language models (LLMs) have recently received considerable attention as alternative solutions for task planning. However, comparing the performance of language-oriented task planners becomes difficult, and there exists a dearth of detailed exploration regarding the effects of various factors such as pre-trained model selection and prompt construction. To address this, we propose a benchmark system for automatically quantifying performance of task planning for home-service embodied agents. Task planners are tested on two pairs of datasets and simulators: 1) ALFRED and AI2-THOR, 2) an extension of Watch-And-Help and VirtualHome. Using the proposed benchmark system, we perform extensive experiments with LLMs and prompts, and explore several enhancements of the baseline planner. We expect that the proposed benchmark tool would accelerate the development of language-oriented task planners.

We investigate the potential of the belief-desire-intention (BDI) model for enhancing proactive action planning and transparency in large language models (LLMs). Our proposed method, BDIPrompting, integrates the knowledge representation framework of the BDI model into prompt design. This allows agents to generate motivational and goal-directed service plans proactively while offering human users insights into the rationale behind the decision-making process. Through preliminary experiments with OpenAI’s GPT-4, we highlight the effectiveness of our approach in planning motivational actions and providing improved explanations during human-agent interactions.

Recent complicated problems require large-scale datasets and complex model architectures, however, it is difficult to train such large networks due to high computational issues. Significant efforts have been made to make the training more efficient such as momentum, learning rate scheduling, weight regularization, and meta-learning. Based on our observations on 1) high correlation between past eights and future weights, 2) conditions for beneficial weight prediction, and 3) feasibility of weight prediction, we propose a more general framework by intermittently skipping a handful of epochs by periodically forecasting near future weights, i.e., a Weight Nowcaster Network (WNN). As an add-on module, WNN predicts the future weights to make the learning process faster regardless of tasks and architectures. Experimental results show that WNN can significantly save actual time cost for training with an additional marginal time to train WNN. We validate the generalization capability of WNN under various tasks, and demonstrate that it works well even for unseen tasks. The code and pre-trained model are available at https://github.com/jjh6297/WNN.

The development of artificial intelligence has led to significant progress in activity detection; however, a proper evaluation of the activity detection performance is not guaranteed and is still lacking in real-life environments. In this study, to verify the stability and usefulness of activity detection in a real-world setting, we analyzed the activity detection performance for 40 elderly people using a human care robot.

Non-verbal behavior is essential for embodied agents like social robots, virtual avatars, and digital humans. Existing behavior authoring approaches including keyframe animation and motion capture are too expensive to use when there are numerous utterances requiring gestures. Automatic generation methods show promising results, but their output quality is not satisfactory yet, and it is hard to modify outputs as a gesture designer wants. We introduce a new gesture generation toolkit, named SGToolkit, which gives a higher quality output than automatic methods and is more efficient than manual authoring. For the toolkit, we propose a neural generative model that synthesizes gestures from speech and accommodates fine-level pose controls and coarse-level style controls from users. The user study with 24 participants showed that the toolkit is favorable over manual authoring, and the generated gestures were also human-like and appropriate to input speech. The SGToolkit is platform agnostic, and the code is available at https://github.com/ai4r/SGToolkit.

Deep learning, based on which many modern algorithms operate, is well known to be data-hungry. In particular, the datasets appropriate for the intended application are difficult to obtain. To cope with this situation, we introduce a new dataset called ETRI-Activity3D, focusing on the daily activities of the elderly in robot-view. The major characteristics of the new dataset are as follows: 1) practical action categories that are selected from the close observation of the daily lives of the elderly; 2) realistic data collection, which reflects the robot's working environment and service situations; and 3) a large-scale dataset that overcomes the limitations of the current 3D activity analysis benchmark datasets. The proposed dataset contains 112,620 samples including RGB videos, depth maps, and skeleton sequences. During the data acquisition, 100 subjects were asked to perform 55 daily activities. Additionally, we propose a novel network called four-stream adaptive CNN (FSA-CNN). The proposed FSA-CNN has three main properties: robustness to spatio-temporal variations, input-adaptive activation function, and extension of the conventional two-stream approach. In the experiment section, we confirmed the superiority of the proposed FSA-CNN using NTU RGB+D and ETRI-Activity3D. Further, the domain difference between both groups of age was verified experimentally. Finally, the extension of FSA-CNN to deal with the multimodal data was investigated.

Social robots should understand the user's nonverbal behavior and respond appropriately. Machine learning is one way of implementing the social intelligence. It provides the ability to automatically learn and improve from experience instead of explicitly telling the robot what to do. This paper proposes an end-to-end machine learning method to learn social behaviors for humanoid robots. We adapt sequence-to-sequence architecture consisting of two long short-term memory (LSTM) units. One is an LSTM encoder for encoding the previous sequence of human poses, and the other is an LSTM decoder for generating the next sequence of robot poses. The weights of the LSTMs are trained using human-human interaction data such as greeting and handshaking. The trained model is implemented in a humanoid robot, Pepper, to show its feasibility. Experimental results show that the robot can generate gestures appropriate to the situation and recognize subtle differences in user behavior. In addition, when a user's behavior changes, the transition to another behavior occurs naturally.

Automatic engagement recognition is a technique that is used to measure the engagement level of people in a specific task. Although previous research has utilized expensive and intrusive devices such as physiological sensors and pressure-sensing chairs, methods using RGB video cameras have become the most common because of the cost efficiency and noninvasiveness of video cameras. Automatic engagement recognition methods using video cameras are usually based on hand-crafted features and a statistical temporal dynamics modeling algorithm. This paper proposes a data-driven convolutional neural networks (CNNs)-based engagement recognition method that uses only facial images from input videos. As the amount of data in a dataset of children's engagement is insufficient for deep learning, pre-trained CNNs are utilized for low-level feature extraction from each video frame. In particular, a new layer combination for temporal dynamics modeling is employed to extract high-level features from low-level features. Experimental results on a database created using images of children from kindergarten demonstrate that the performance of the proposed method is superior to that of previous methods. The results indicate that the engagement level of children can be gauged automatically via deep learning even when the available database is deficient.

For human-like agents, including virtual avatars and social robots, making proper gestures while speaking is crucial in human-agent interaction. Co-speech gestures enhance interaction experiences and make the agents look alive. However, it is difficult to generate human-like gestures due to the lack of understanding of how people gesture. Data-driven approaches attempt to learn gesticulation skills from human demonstrations, but the ambiguous and individual nature of gestures hinders learning. In this paper, we present an automatic gesture generation model that uses the multimodal context of speech text, audio, and speaker identity to reliably generate gestures. By incorporating a multimodal context and an adversarial training scheme, the proposed model outputs gestures that are human-like and that match with speech content and rhythm. We also introduce a new quantitative evaluation metric for gesture generation models. Experiments with the introduced metric and subjective human evaluation showed that the proposed gesture generation model is better than existing end-to-end generation models. We further confirm that our model is able to work with synthesized audio in a scenario where contexts are constrained, and show that different gesture styles can be generated for the same speech by specifying different speaker identities in the style embedding space that is learned from videos of various speakers. All the code and data is available at https://github.com/ai4r/Gesture-Generation-from-Trimodal-Context.

Co-speech gestures enhance interaction experiences between humans as well as between humans and robots. Most existing robots use rule-based speech-gesture association, but this requires human labor and prior knowledge of experts to be implemented. We present a learning-based co-speech gesture generation that is learned from 52 h of TED talks. The proposed end-to-end neural network model consists of an encoder for speech text understanding and a decoder to generate a sequence of gestures. The model successfully produces various gestures including iconic, metaphoric, deictic, and beat gestures. In a subjective evaluation, participants reported that the gestures were human-like and matched the speech content. We also demonstrate a co-speech gesture with a NAO robot working in real time.

This paper proposes a new dentity, gender, and age recognition convergence system for robot environments. In a robot environment, it is difficult to apply deep learning based methods because of various limitations. To overcome the limitations, we propose a shallow deep-learning fusion model that can calculate identity, gender, and age at once, and a technique for improving recognition performance. Using convergence network, we can obtain three pieces of information from a single input through a single operation. In addition, we propose a 2D / 3D augmentation method to generate virtual additional datasets for learning data. The proposed method has a smaller model size and faster computation time than existing methods and uses a very small number of parameters. Through the proposed method, we finally achieved 99.35%, 90.0%, and 60.9% / 94.5% of performance in identity recognition, gender recognition, and age recognition. In all experiments, we did not exceed the state-of-the-art results, but compared to other studies, we obtained performance similar to the previous study using only less than 10% parameters. In some experiments, we also achieved state-of-the-art result.

This paper presents a recurrent learning-based facial attribute recognition method that mimics human observers' visual fixation. The concentrated views of a human observer while focusing and exploring parts of a facial image over time are generated and fed into a recurrent network. The network makes a decision concerning facial attributes based on the features gleaned from the observer's visual fixations. Experiments on facial expression, gender, and age datasets show that applying visual fixation to recurrent networks improves recognition rates significantly. The proposed method not only outperforms state-of-the-art recognition methods based on static facial features, but also those based on dynamic facial features.

Autonomous mobile robots have been deployed at hospitals to cope with numerous delivery services. Robots deliver items from point to point through autonomous navigation. Conventional robots work individually conducting only one task at a time. This paper proposes a fleet optimization method to maximize the advantages of operating a group of robots concurrently by assigning multiple tasks to a robot. The server selects an appropriate robot for a delivery task based on the cost for conducting the delivery task of each robot. The robot with the minimum cost is assigned for the new delivery task. This way, a robot is assigned to conduct multiple tasks. The scheduler finds the minimum routing path for visiting multiple delivery locations. The shortest path for visiting multiple delivery locations is derived from a combinatorial search approach. This paper proposes an algorithm that reduces the computational burden for finding the path combinations. Because this procedure takes a short amount of time, the allocation of a robot can be conducted in real-time. The proposed algorithm is tested in the simulation, and the results show that it increases the efficiency when using multiple robots. In addition, by running the simulation and reviewing the performance results, the management group of a hospital can determine the number of robots required at their site.

In this paper, we identify the effects of robot social skills for enhancing social interaction in a physical training. To that end, we designed a physical training scenario and conducted an experiment with 28 participants using the humanoid robot NAO. As a result, there were significant differences between the control group where social skills were not used, and the experimental group where social skills were used by the robot.

We present a movable spatial augmented reality (SAR) system that can be easily installed in a user workspace. The proposed system aims to dynamically cover a wider projection area using a portable projector attached to a simple robotic device. It has a clear advantage than a conventional SAR scenario where, for example, a projector should be installe1d with a fixed projection area in the workspace. In the previous research [1], we proposed a data-driven kinematic control method for a movable SAR system. This method targets a SAR system integrated with a user-created robotic (UCR) device where an explicit kinematic configuration such as CAD model is unavailable. Our contribution in this paper is to show the feasibility of the data-driven control method by developing a practical application where dynamic change of projection area matters. We outline the control method and demonstrate an assembly guide example using a casually installed movable SAR system.

This paper presents a process to build a classifier in a data-driven way for recognizing engagement of children in a robot-based math quiz game. The process consists of collecting video recordings from HRI experiments; annotating the social signals and engagement states via video analysis; extracting feature vectors from the annotations and training classifiers. We conducted an experiment with 7 participants of 10 -- 11 years of age using an android robot EveR-4. With three coders annotating the video recordings and extracting features by snapshot model with 1-second time window, we achieved 84.83% recall performance.

A horseback riding simulator is a robotic machine that simulates the motion of horseback riding. In this paper, we present an interaction control system for a horseback riding simulator. The proposed system provides a postural correction function suited to the user based on their historical log and specialized posture coaching data. The system has adopted certain schemes for posture detection and recognition of the identified user as a way to recommend customized exercise modes. Our experiments show that including these techniques will help users maintain good posture while riding.

In this paper, we present an affect recognition system for measuring the engagement level of children using the Kinect while performing a multiple intelligence test on a computer. First of all, we recorded 12 children while solving the test and manually created a ground truth data for the engagement levels of each child. For a feature extraction, Kinect for Windows SDK provides support for a user segmentation and skeleton tracking so that we can get 3D joint positions of an upper-body skeleton of a child. After analyzing movement of children, the engagement level of children’s responses is classified into two classes: High or Low. We present the classification results using the proposed features and identify the significant features in measuring the engagement.

This paper describes a novel RGB-D-based visual target tracking method for person-following robots. We enhance a single-object tracker, which combines RGB and depth information, by exploiting two different types of distracters. First set of distracters includes objects existing near-by the target, and the other set is for objects looking similar to the target. The proposed algorithm reduces tracking drifts and wrong target re-identification by exploiting the distracters. Experiments on real-world video sequences demonstrating a person-following problem show a significant improvement over the method without tracking distracters and state-of-the-art RGB-based trackers. A mobile robot following a person is tested in real environment.

Providing robots with social intelligence is critical for making entertaining and sustainable human-robot interactions. The first step to get good social intelligence is to appropriately understand the meaning of social signals emitted by interactors. In this paper, we introduce a preliminary study on identifying principal social signals in interpreting participant's engagement and confirmation intention in 1:1 interactions. We annotated 6 video recordings of private teacher-student interactions with 20 social signals and their interpretations, and built pattern data sets with different subsets of social signals. C4.5 based decision trees were generated using the pattern data sets and the recall rates were compared. Also attribute selection was performed to find principal social signals. The results showed that verbal signal was the most principal for determining engagement, and the combination of gaze and verbal signal for confirmation intention.

IT technologies have started to be applied to many fields such as sports science, robotics or BT. Especially, object detection and recognition using vision sensor has intensively studied due to diverse application fields. But, various illumination condition, pose or time progress have put object detection and recognition to challenge task in the real world. In this paper, face detection and recognition using vision sensor applied to sports simulator which has been proposed. Face detection has been processed to identify straight in the face that has used to detect head pose of riders. Face recognition has used to identify user, who has tried to take horse riding simulator. Gabor wavelet and face graph has used to recognize low quality face image, which has acquired under poor illumination environment and movement of simulator. We have simulated on FERET and ETRI database. ETRI database has acquired on horse riding simulator under poor illumination condition. The accuracy of 91% face recognition rate on FERET and encouraging result of 82% recognition rate on ETRI DB have been obtained.

This paper describes the reason that perception technology in HRI has not given high performance enough to be used for commercial service robots. As a practical solution for better performance of perception technology, we propose a perception framework with a dedicated perception engine called a perception demon. The demon in the proposed framework constantly collects evidences and analyses them better by combining various types of individual perception components. The proposed framework enables robot makers to easily get more reliable information on humans without concerns about optimizing perception components to their robots.

We propose a user pose verification technique using depth information to compare their pose with expert riders. Xtion sensor by Asus is used for gathering a depth data in real world. The user pose verification algorithm is divided into two categories: user segmentation and user pose verification. In user segmentation step, body parts are segmented based on the region growing algorithm from a head point (i.e. seed point). Then, a simple algorithm is used to generate skeletal joints in the segmented body parts. Finally, the user pose is investigated by the standard pose of experts in the same situation.

In everyday lives, so many events happen around us. Although not all of them are important to us, we have to process them all to be able to isolate the significant events. Consciousness is a constant awareness of the environment, which can be acquired only by such an exhaustive approach. As an autonomous agent, a robot is also required to be conscious, which is not yet properly realized. In this paper, we discuss the strategy to organize various perception technologies to achieve the consciousness of the robots. Especially, an independent process that constantly monitors its surroundings without the intervention of higher processes is proposed. This process, which is called a perception daemon, is discriminated from the traditional component-based approach, where individual perception technology is provided as a passive function so that the service applications should call whenever it is necessary.

This paper presents a person following robot equipped a RGB-D imaging sensor. We introduce three modules of a visual target tracking, target detection, and robot control. For the tracking module, distracters existing near-by the target are explicitly tracked to support target tracking. Preliminary tests showed that the robot robustly follows the target in uncontrolled environment with cluttered background and uneven illumination.

Robotic person-following is an essential component for natural human robot interaction. To follow a person, the robot should track the target person robustly and in real time. Object tracking algorithms in the computer vision field typically require abundant features and heavy computing power, and thus cannot be directly applied to person-following robots due to the problems arising in practical robotic environments. This paper proposes a robotic person-tracking algorithm based on modified multiple instance learning. In order to resolve the problems raised by the rearward view of the target person, the tracker is modified to be guided by color histogram back-projection. Additionally, the search area model is modified from circle to ellipse and the number of features is reduced so that the tracker should adapt the robotic environment in real-time. The algorithm is validated through system integration and experiments.

Automatic facial expression recognition is an important technique for interaction between humans and machines such as robots or computers. In particular, pose invariant facial expression recognition is needed in an automatic facial expression system because frontal faces are not always visible in real situations. The present paper introduces a multi-view method for recognizing facial expressions using a parametric kernel eigenspace method based on class features (pKEMC). We first describe pKEMC that finds the manifold of data patterns in each class on a non-linear discriminant subspace for separating multiple classes. Then, we apply pKEMC for pose-invariant facial expression recognition. We also utilize facial-component-based representation to improve the robustness to pose variation. We carried out the validation of our method on a Multi-PIE database. The results show that our method has high discrimination accuracy and provides an effective means to recognize multi-view facial expressions.

This paper presents a character segmentation method to address automatic number plate recognition problem. The method considered pixel intensity, character appearance, and arrangement of characters altogether to segment character regions. The method firstly discovers candidate blobs of characters by using connected component analysis and appearance-based character detection. A character recognizer is used for removing redundant and noisy blobs. Then, a trained classifier selects character blobs among the candidates by examining arrangement of the blobs. Experimental results show an achievement of 98.3% of segmentation rate, which prove the effectiveness of our method.

This paper presents a user recognition system, using face, height, and clothes color features under the special assumption that is a user is monitored and tracked. In real human-robot interaction situation, all information cannot be provided at the same time and some parts of frames in a video have no clues at all. In the proposed system, tracking is an important feature to recognize a user because data in the previous frames can be utilized. We propose an information update method that efficiently updates similarity results. This system is tested using the movie clips acquired under the unconstrained environment including illumination variation, several distance from a camera to the user, and various view types of human body.

Ubiquitous location-based services have become a popular trend these days as a next-generation paradigm. Various research studies are underway to provide the location-based services through wireless sensor networks in an indoor environment. In this paper, we present a user's location recognition system based on environmental sensor data in an indoor living space. The system gathers the environmental data and extracts the feature data from them. It also recognizes and classifies the current location information of the user. We describe the configuration of the sensor module used for location recognition, and present a recognition system that is designed and implemented to recognize the location based on data from the sensor signals, and describe the pertinent aspects of our experimental results.

This paper describes a location and emergency monitoring system, which aims to monitor an emergency situation, activities and location for elder care. The watch-type device detect emergency such as emergency call or fall. The collected data through the ZigBee network is send to a DB server and stored in a DB. A care staff recognizes an emergency message via SMS or monitoring interface. At the same time, robot moves to an emergency location and sends video stream to a caregiver using tele-presence. A caregiver copes with an emergency. This system was tested in a retirement village.

A character segmentation algorithm for automatic license plate recognition is presented in this paper. Character regions are selected through binarization, connected component analysis, and character recognition. A blob analysis operation excludes noisy blobs, merges fragmented blobs, and splits clumped blobs. A character segmentation module achieved an accuracy rate of 97.2%. The recognition accuracy of the complete system with license plate localization was 90.9%. In depth analysis of failure cases is also provided for better understanding of the algorithm and a future development direction.

We describe in this paper a framework, called URSF, for simulating ubiquitous computing environment and ubiquitous robots. URSF provides in/out channel with which ubiquitous robot platforms can be plugged. Once connected to the framework, ubiquitous robot platform can percept and affect the world simulated in the framework. The simulated world is built by composing a simulation space and placing in it operational components such as appliances, sensors, persons, robots etc. Each operational component continuously generates context data, which are fed to the plugged platform. The platform builds world model by interpreting the context data stream. Operational components in the simulation space expose services through semantic Web service scheme, through which plugged platform can affect the simulated world. Semantic Web service scheme enables automated discovery of services dynamically coming and going in the simulation space. The framework can be used to observe how robots interact with environment via ubiquitous network. We discuss two advanced usages of URSF: intelligent robot manipulation and mixed robot-reality manifestation.