Social robots have been extensively studied in recent decades, with many researchers exploring the use of modalities such as facial expressions to achieve more natural emotions in robots. Various methods have been attempted to generate and express robot emotions, including computational models that define an affect space and show dynamic emotion changes. However, the implementation of multimodal expression in previous models is ambiguous, and the generation of emotions in response to stimuli relies on heuristic methods. In this paper, we present a framework that enables robots to naturally express their emotions in a multimodal way, where the emotion can change over time based on the given stimulus values. By representing the robot's emotion as a position in an affect space of a computational emotion model, we consider the given stimuli values as driving forces that can shift the emotion position dynamically. In order to examine the feasibility of our proposed method, a mobile robot prototype was implemented that can recognize touch and express different emotions with facial expressions and movements. The experiment demonstrated that the emotion elicited by a given stimulus is contingent upon the robot's previous state, thereby imparting the impression that the robot possesses a distinctive emotion model. Furthermore, the Godspeed survey results indicated that our model was rated significantly higher than the baseline, which did not include a computational emotion model, in terms of anthropomorphism, animacy, and perceived intelligence. Notably, the unpredictability of emotion switching contributed to a perception of greater lifelikeness, which in turn enhanced the overall interaction experience.

Social robots utilize facial expressions to convey emotions, yet most rely on predefined animation sequences for each emotion, which can result in abrupt or unnatural transitions. To address this, we employ a Control Point (CP)-based approach that dynamically adjusts facial features, enabling seamless expression transitions without predefined animations. Our study explores whether a CP-based approach enables smoother and more natural facial expressions compared to commonly used facial expression methods. Furthermore, we validate its applicability by demonstrating its effectiveness on two distinct facial designs highlighting versatility across both tested designs.

We explore the potential of mobile motion as an expression modality for home social robots with a low expressivity. To gauge the potential of mobile motion, we examine its relative efficacy in terms of emotion perception accuracy, emotion intensity, and impression (anthropomorphism and animacy) compared to screen-based facial expressions. Additionally, we explore how users perceive the emotional intensity of expressions based on different degrees of expression by manipulating emotional features such as motion size and speed. Motion expressions are less accurate than facial expressions, but perform on par with facial expressions in other metrics. Its dynamic and expressive features elicit powerful emotional conveyance, in contrast to the low emotional impact associated with the monotonous nature of screen-based facial expressions. Further research is needed on ME for specific emotions, but in general, the higher the degree of expression, the more intense the emotion conveyed. We show that the degree of expression, i.e., the combination of emotional features and their modulation, can be utilized to express the emotional intensity, situational dependence, and personality of robots. In conclusion, we argue that mobile motion is a promising method to compensate for the weaknesses of screen-based facial expression, which is the dominant expression modality for low expressivity robots.

Researchers have explored the effects of expressing emotions using various modalities in the field of social robots. Prior studies have demonstrated that the use of color in emotional expressions can enhance user acceptance, relationship building, and communication effectiveness. This study aims to validate the effectiveness of different modalities in expressing Ekman’s six basic emotions. Specifically, four modalities were compared: face expression (F), face and LED color expression (FL), face and LED color expression with blinking (FLB), and face and eye color expression (FE). To accomplish this, we developed a small social robot prototype and used a computational emotion model to improve robot dynamics and interactivity. The findings revealed that, although the F modality effectively expressed emotions, ambiguous emotions were better perceived when color or blinking was incorporated. Emotions such as anger, sadness, disgust, surprise, and fear were better conveyed to the participants when the FL, FLB, and FE modalities were utilized. For happiness, F alone was sufficient for recognition. This study provides empirical evidence on the effectiveness of different modalities for expressing emotions in social robots and offers valuable insights gathered from participants’ feedback and reflections.

In this paper, we present ALH-E (ALternative Healthcare for Expressing ache), an assistive robot with a deformable and flexible interface for pain communication. It consists of two components: a squeezable device for inputting the patient’s pain intensity and a flexible output device that expresses the pain by twisting-bending movements in response to the input signals. The interconnectivity between the devices allows for communication of pain intensity between patients and caregivers, anywhere and anytime. A field study in the hospital (dental clinic and orthopedic) was conducted to verify the usability and effectiveness of ALH-E in pain communication. Our field study results demonstrated the unique advantages of ALH-E over conventional methods, providing significant assistance to patients and caregivers.

Describing pain intensity constitutes an essential part of pain communication. A medical practitioner cannot depend on pain scales because of the criteria differences between the patient and caregiver. However, online pain communication is dependent on a patient’s description and an assessment on a pain scale. This paper proposes a robot-assisted pain communication system with a tangible user interface that enables non-numerical pain communication. The SQTT interface is proposed using design processes consisting of a novel squeezable input device and a twisting robot. The twisting expression of the robot represents the pain intensity, which is gauged from the squeezing power on the input device. Integrating input and output requires defining how the twisting motion of the robot is rendered from the squeezing input. An experiment was conducted to evaluate the four methods of rendering: raw, smoothing (moving average), dynamic smoothing, and updating peak. The result of a non-parametric one-way analysis of variance indicates a significant difference between the rendering methods. As a result, an appropriate rendering method is proposed based on the ranges: smoothing for mild pain, vibrating for moderate pain, and exaggerating for severe pain. In conclusion, the robot-assisted pain communication system can be implemented with intuitive interaction primitives for online context. This paper contributes a lesson in designing a robot-assisted online pain communication, which uses a new method of measuring deformation.

This paper details the development of a Shared Indoor Smart Mobility device called AngGo. As a precursor to the development process, we conducted user research on three kinds of outdoor personal mobility. Our goal was to determine the major differences between outdoor and indoor personal mobility and to ensure that AngGo would meet the requirements of indoor personal mobility in a practical way, as informed by the results of surveys and interviews. Tests were conducted on the time-of-flight sensors to be used for indoor autonomous driving. Manual mode as well as the experiment-based equations governing the sensors were optimized through user testing. Our observational experiments, which were carried out in the lobby of a building, showed that both autonomous and manual modes functioned as designed. This study makes a contribution to the literature by describing how our AngGo device features an autonomous driving platform that can transport riders around an indoor environment.

In the field of HRI, to take advantage of people’s innate ability of communication, researchers have thus far concentrated on facial expression in facilitating human robot communication. However, for the robot to express emotional intensity, other modalities such as gestures, movement, sound, and color are also needed. This paper suggests that the intensity of emotion can be expressed with color and blinking, which are applicable on robots via LED. Although color and emotion have certain relations, it is difficult to implement the findings of previous research in this area due to a lack of quantitative data. In this paper, we determined RGB values of color and blinking period to effectively express intensities of six basic emotions based on quantitative data. Color and blinking are implemented on an avatar and the intensities of emotions are evaluated through surveys with the avatar. It was found that color and blinking helped the avatar express emotional intensity of surprise, happiness, disgust, and anger. For fear, and sadness, the color and blinking did not play a significant role.

Recently, in the field of HRI, multimodal expression has been an issue. Synchronizing modalities and determining what modality to use are important aspect of multimodal expression. For example, when robots express emotional states, they may use only facial expressions or facial expressions with gestures, neck motions, sounds, etc. In this paper, emotional boundaries are proposed for multimodal expression in a three-dimensional affect space. The simultaneous expression of facial expression and gestures was demonstrated using proposed emotional boundaries on a simulator.

This paper introduces an emotion interaction system for a service robot. The purpose of emotion interaction systems in service robots is to make people feel that the robot is not a mere machine, but reliable living assistant in the home. The emotion interaction system is composed of the emotion recognition, generation, and expression systems. A user's emotion is recognized by multi-modality, such as voice, dialogue, and touch. The robot's emotion is generated according to a psychological theory about emotion: OCC (Ortony, Clore, and Collins) model, which focuses on the user's emotional state and the information about environment and the robot itself. The generated emotion is expressed by facial expression, gesture, and the musical sound of the robot. Because the proposed system is composed of all the three components that are necessary for a full emotional interaction cycle, it can be implemented in the real robot system and be tested. Even though the multi- modality in emotion recognition and expression is still in its rudimentary stages, the proposed system is shown to be extremely useful in service robot applications. Furthermore, the proposed framework can be a cornerstone for the design of emotion interaction and generation systems for robots.

It is expected that robots will be widely exposed to humans in the near future. Emotional communication is very important in human-robot interaction, as well as human- human interaction. Facial expression is an emotional expression method between humans, and it also enables human to recognize a robot's emotional state. Although there are lots of previous research regarding facial expressions, it is not easily applicable to different shapes of robot faces, if the number and types of the robot's control points are varied. In addition, the natural connection between emotions has not been considered or has been inefficiently implemented in previous research. In this paper, we propose a linear dynamic affect-expression model for continuous change of expression and diversifying characteristics of expressional changes. Therefore, the proposed model allows a robot's facial expression and expression changes to more closely resemble those of humans, and this model is applicable to various mascot-type robots irrespective of the number and types of robot's control points.

A robot's face is its symbolic feature, and its facial expressions are the best method for interacting with people with emotional information. Moreover, a robot's facial expressions play an important role in human-robot emotional interactions. This paper proposes a general rule for the design and realization of expressions when some mascot-type facial robots are developed. Mascot-type facial robots are developed to enable friendly human feelings. The number and type of control points for six basic expressions or emotions were determined through a questionnaire. A linear affect-expression space model is provided to realize continuous and various expressions effectively, and the effects of the proposed method are shown through experiments using a simulator and an actual robot system.

In the last decade, face analysis, e.g. face recognition, face detection, face tracking and facial expression recognition, is a very lively and expanding research field. As computer animated agents and robots bring a social dimension to human computer interaction, interest in this research field is increasing rapidly. In this paper, we introduce an artificial facial expression mimic system which can recognize facial expressions of human and also imitate the recognized facial expressions. We propose a classifier that is based on weak classifiers obtained by using modified rectangular features to recognize human facial expression in real-time. Next, we introduce our robot that is manipulated by a distributed control algorithm and that can make artificial facial expressions. Finally, experimental results of facial expression recognition and facial expression generation are shown for the validity of our artificial facial expression imitator