Engagement is a key concept in Human-Robot Interaction (HRI), as high engagement often leads to improved user experience and task performance. However, accurately estimating engagement during interactions is challenging. In this study, we propose a Dynamic Bayesian Network (DBN) to infer user engagement from various modalities, including head rotation, eye movements, facial expressions captured through visual sensors, as well as facial temperature variations measured by a thermal camera. Data was gathered from a human-robot interaction (HRI) experiment, where a robot guided participants and encouraged them to share their thoughts and insights on environmental issues. Our approach successfully combines these diverse features to offer a thorough assessment of user engagement. The network was tested on its capacity to classify participants as either engaged or not engaged, achieving an accuracy of 0.83 and an Area Under the Curve (AUC) of 0.82. These findings underscore the strength of our DBN in detecting user engagement during interactions.

Humor is a key element in human interactions, essential for building connections and rapport. To enhance human-robot communication, we developed a humor-aware chat framework that enables robots to deliver contextually appropriate humor. This framework takes into account the interaction environment, and user's profile as well as emotional state. Two GPT models are used to generate responses. The initial one, named sensor-GPT, processes contextual data from the sensor along with the user's response and conversation history to create prompts for the second one, chat-GPT. These prompts can guide the model on how to integrate appropriate humor elements into the conversation, ensuring that the dialogue is both contextually relevant and humorous. Our experiment compared the effectiveness of humor expression between our framework and the GPT-40 model. The results demonstrate that robots using our framework significantly outperform those using GPT-4o in humor expression, extending conversations, and improving overall interaction quality.

Social robots have shown significant potential in enhancing learning experiences, and humor has been proven to be beneficial for learning. This study investigates the impact of both the presence and timing of humor on students’ learning outcomes and overall learning experience. A total of 24 participants were randomly assigned to one of the three conditions: (C1) interact with a robot with no humor, (C2) interact with a robot with humor at pre-defined moments during the lesson, and (C3) interact with a robot that triggers humor based on engagement levels. The results revealed that the humor at pre-defined moments condition (C2) led to significantly better learning outcomes and longer interaction times compared to the other two conditions. While the adaptive humor in Condition C3 did not significantly outperform Condition C1, it showed positive effects on participants’ perceived learning effectiveness and engagement. These findings contribute to the understanding of how humor, when strategically timed, can enhance the effectiveness of social robots in educational settings.

Social robots have become increasingly prevalent in the context of one-on-one tutoring, serving as effective educational aids. In response to this trend, the present study was devised to conduct a comparative analysis between human tutors and robot tutors. Additionally, the study aims to investigate how varying previous knowledge in robots influence students’ tendencies for help-seeking. By examining the performance and physiological signals of participants, this research seeks to provide valuable insights into the effectiveness of social robots in educational contexts. 21 participants were divided into three groups, each seeking assistance from a human tutor (HT), seeking help from a robot without any prior knowledge of robots (RT1), and seeking help from a robot after gaining some understanding of its capabilities (RT2). Our results demonstrated that participants sought more help from robot than from human and participants in Group RT2 performed better than participants in Group RT1. However, participants experienced greater cognitive load when interacting with a robot tutor compared to interacting with a human tutor. Future work could focus on developing interventions to alleviate students’ cognitive load during interactions with robot tutors.

Expressing humor in social interactions presents a significant challenge for humans due to its intricate linguistic nature. This complexity is further magnified when teaching robots to express humor appropriately. Among the various expressions of humor, jokes are one of the most commonly used. Therefore, a well-annotated joke dataset holds significant promise in enhancing a robot’s ability to express humor effectively. This paper introduces a dataset comprising over two thousand jokes, with the aim of providing rich material and multidimensional selection criteria for the humor expression of the robot. The creation of this joke dataset involved a collaborative effort among robot experts studying HRI, psychologists with rich humor research experience, and GPT-3. The annotation process primarily concentrated on four dimensions within the dataset: the humor style of jokes, semantic words (aligned with semantic gestures), keywords, and ratings of joke funniness. We additionally outline several prospective applications of this dataset. We introduce a BERT-based neural network model trained on the dataset with semantic word labels. This model aims to empower robots to choose suitable semantic words from jokes and articulate them alongside corresponding semantic gestures. Moreover, we offer suggestions for utilizing jokes from this dataset to facilitate the adaptive expression of humor by social robots. These endeavors will further enhance the multi-modal humor expression capability of social robots.

Laughter serves as a subtle social signal in human interaction, playing an essential role in expressing emotions and facilitating social connections. However, laughter comes in various forms and is usually accompanied by different non-verbal expressions, such as facial expressions and gestures. These accompanying factors can significantly influence the effect of laughter in diverse contexts, thus complicating the research on laughter, especially in understanding its role in social dynamics. Consequently, endowing robots with the ability to appropriately use laughter in interactions with humans is still a big challenge. Our current study focuses on the effect of robot laughter on robot humor expression. Our objective is to investigate whether and how two factors, the type of laughter and the robot laughter gesture, impact the overall humor performance. In this study, we selected four types of laughter (sarcastic, joyful, embarrassed, and relieved laughter) from a laughter corpus based on four specific types of jokes (Affiliative, Aggressive, Self-enhancing, and Self-defeating). For each type of laughter, we designed distinct robot gestures. During the humor performance, the robot NAO delivered jokes accompanied by matching or mismatching laughter, with or without corresponding gestures. To enhance the quantity and diversity of experimental data, we conducted an online survey utilizing recordings of the robot’s humor performance. The experimental findings indicate that when the robot’s laughter matches the type of humor in the joke, participants rate the humor performance significantly higher compared to situations where there is a mismatch. Additionally, the results confirm the positive impact of robot laughter gestures on humor performance.

The Covid-19 pandemic has massively developed the use of distance learning. The limits of this practice have gradually come to light, both for students and for teachers. It is now crucial to design alternative solutions to overcome the shortcomings of videoconferencing in terms of involvement, concentration, learning, and equity. Social robots are increasingly used as tutors in the educational context and help improve teaching efficiency. Many psychology-based principles have been applied in education to guide instructional strategies, motivate students, and create a positive and productive learning environment. In this work, we use Regulatory Focus Theory (RFT), which categorizes an individual’s motivation into two types: Promotion and Prevention. Promotion-focused individuals are motivated by the potential for growth and achievement, whereas prevention-focused individuals are motivated by the potential for avoiding negative outcomes. Based on RFT, we aim to explore if and how the regulatory-focused behavior of the tutor robot can affect participants’ learning outcomes. In this work, a language learning scenario was designed with two conditions: (1) a robot tutor with promotion-focused behavior, (2) a robot tutor with prevention-focused behavior. The results are encouraging and support that promotion robot tutor can increase the learning efficiency of promotion participants and prevention robot tutor will enhance the learning interest of prevention participants.

Humor usually plays a positive role in social activities. We posit that endowing a social robot with humor ability can enhance expressive human-robot interaction. People’s perception on humor is different, and therefore, making the robot expressing humor in an appropriate way is a challenge. The main aim of this paper is to explore the correlation between people’s perception on different types of jokes and their humor styles (Affiliative, Self-enhancing, Self-defeating, Aggressive). In the experiment, we used the humanoid robot Pepper to perform different types of jokes. Both subjective (jokes rating) and objective measures (RGB and thermal images) were used. The latter method was employed to extract facial features (facial action unit and facial temperature). After extracting and analyzing the data of both measurement methods, we found that the Self-defeating humor style positively affects people’s rating on all types of jokes. In addition, there is also a positive correlation between people’s humor style scores and the degree of happiness.

Grasping unstructured objects and sensing the contact force are two vital issues for grippers. However, it is still difficult for most existing grippers to realize these two functions simultaneously. In this article, we revise the traditional fin-ray finger by inserting a series of rigid nodes into the compliant structure and develop an adaptive two-finger gripper. This design linearizes the gripper's deformation-force relationship and enables an intrinsic force sensing ability without any tactile sensor. Experimental results show that the finger has high accuracy in sensing the external force applied at its middle part (average error less than 3%) but much larger errors appear near its two ends. Further experiments indicate that the gripper functions well in sensing the total grasping force (average error less than 8%). Although larger errors are observed in estimating the force distribution at each node, the variation tendency of the sensed force coincides well with the ground truth. Experiments are also carried out on grasping free-form objects and performing pick-and-place operations to further prove the gripper's adaptive grasping and intrinsic force sensing abilities.