We introduce HRI-SENSE, a multimodal dataset of Human-Robot Interactions (HRI) studying users' social, phys-ical (e.g. facial expressions, body movements) and emotional, psychological (e.g. frustration, satisfaction) responses to robot behaviour. The dataset captures participants collaborating with a TIAGo humanoid robot following various behaviour models on a manipulation-based “Burger Assembly“ task, eliciting different user reactions. HRI -SENSE contains over 6 hours of verbal and physical interactions taking place over 146 sessions with 18 participants, recording multiple modalities captured simultaneously by RGB and Depth cameras from three angles and one microphone. The time-synchronized multimodal data include non-verbal behaviours (e.g. facial landmarks, expressions, pose landmarks), explicit feedback signals (e.g. verbal interactions), robot movements and self-assessed questionnaires on sociodemo-graphics and user impressions (e.g. frustration, satisfaction) on robot interactions. HRI-SENSE is expected to facilitate further research into modelling non-verbal behaviour and advancing the development of user-aware interaction models in HRI domain.

Our work investigates how social robots can efficiently collaborate with human users in a user-aware manner, minimising the generated frustration in human colleagues, thus enhancing their experience. As part of this, we develop a useraware framework for human-robot collaborative learning. We model users' frustration during human-robot interactions based on recent interactions inspired by Psychological principles and develop different frustration-aware interactive preference learning and decision-making models using multi-armed bandit and knapsack methods. Evaluating our approach, 1) we conducted simulated experiments on realistic human-behaviour datasets and 2) a user-study in which participants worked with a TIAGo Steel humanoid robot on a collaboration task using frustration- aware and non frustration-aware (Upper Confidence Bounds and Instruction-based) models. We demonstrate that when collaborating with the frustration-aware robot, users completed the collaboration task 9.04% faster and using 20.54% less number of verbal interactions, with user questionnaire responses reporting less frustration experienced compared to the baseline approaches. Additionally, we create a multimodal dataset containing over 6 hours of human-robot interactions displaying various explicit and implicit user responses.

In human-robot interactions (HRI), it is crucial for robots to be accepted by users and that they find robotic assistance attempts helpful rather than frustrating. Working towards this goal, we investigate the problem of frustration-aware robot behaviour planning in human-robot interaction contexts without continuous user contact or live feedback. Specifically, we address the question of how social robots can efficiently localise users and assist them with errands of various importance in office environments, while minimizing the frustration experienced by their human colleagues to enhance the overall interaction experience. Doing so, we design a frustration-aware decision-making and learning framework building on multiarmed bandit approaches and knapsack algorithms, in addition to developing a Psychology-based model of frustration tailored for HRI settings with limited user contact. Then we evaluate our approach on realistic user behaviour datasets, simulating the interactions’ robotic components in Gazebo with a TIAGo robot, and perform further scalability analysis in graph-based simulations. The experimental results demonstrate that the proposed framework achieves localisation success rates and travel times that converge towards oracle values (outperforming other structured learning benchmarks) while yielding an estimated up to 75% less frustration – indicating the proposed framework’s suitability for advancing to user studies and deployment in real-world scenarios.

Socially assistive robots and smart home devices are increasingly integrated into daily life, offering emotional and social support in a modern society where many individuals live alone. Inspired by this context, this paper presents the SYNERGY framework for managing smart home environments comprising multiple smart devices and assistive robots. SYNERGY is designed to handle a broad spectrum of user requests, from simple queries to complex tasks requiring multi-step reasoning, and assigns them to appropriate agents for optimal execution. We conducted a series of experiments under various configurations to evaluate the framework’s performance in processing user queries across socially relevant contexts and its effectiveness in task allocation. Experimental results demonstrate that incorporating contextual retrieval into our designed decision-making module significantly improves the system’s understanding of user intent and is crucial in handling complex, multi-step tasks. Additionally, the designed task allocation module proves its effectiveness in optimizing assignments using cost matrices, enabling flexible and efficient multi-agent coordination.

Nonverbal communication has the potential to enable robots to interact with customers in service environments efficiently. While previous efforts in this domain have been paid to the understanding of customers’ interaction experience from different aspects, there is a lack of studies on the configuration of multimodal interaction (i.e., the combination of nonverbal gestures, voice, and touch) in service environments and the effect of nonverbal communication styles when performed in this setting. This paper aims to address the gap in the literature by introducing a multimodal HRI framework operated in a cafe shop setting. A systematic study is conducted with 171 customers. It is followed by an in-depth analysis based on objective and subjective measurements to build an understanding of customers’ attitudes towards the robot’s nonverbal behaviours.

Humans naturally produce nonverbal signals such as facial expressions, body movements, hand gestures, and tone of voice, along with words, to communicate their messages, opinions, and feelings. Considering robots are progressively moving out from research laboratories into human environments, it is increasingly desirable that they develop a similar social intelligence. Therefore, equipping social robots with nonverbal communication skills has been an active research area for decades, where data-driven, end-to-end learning approaches have become predominant in recent years, offering scalability and generalisability. However, most of these approaches consider a single character, modelling intrapersonal dynamics only. In this paper, we propose a method based on conditional Generative Adversarial Networks, intending to generate behaviours for a robot in affective dyadic interactions. Our method takes as an input the audio of a target person together with the nonverbal signals of their interacting partner, modelled by a novel Context Encoder, to generate appropriate body gestures. We evaluate our method on the multimodal JESTKOD dataset that comprises dyadic interactions under agreement and disagreement scenarios. The experimental results show that Context Encoder can better contribute to the prediction of co-speech gestures in agreement situations.

Non-verbal behaviors have an indispensable role for social robots, which help them to interact with humans in a facile and transparent way. Especially, communicative gestures allow robots to have the capability of using bodily expressions for emphasizing the meaning of their speech, describing something, or showing clear intention. This paper presents an approach to learn the synthesis of human actions and natural language. The generative framework is inspired by Conditional Generative Adversarial Network (CGAN), and it makes use of the Convolutional Neural Network (CNN) with the Action Encoder/Decoder for action representation. The experimental and comparative results verified the efficiency of the proposed approach to produce human actions synthesized with text descriptions. Finally, through the Transformation model, the generated data were converted to a set of joint angles of the target robot, being the robot's communicative gestures. By employing the generated human-like actions for robots, it suggests that robots' social cues could be more understandable by humans.

Generating emotional bodily expressions for culturally competent robots has been gaining increased attention to enhance the engagement and empathy between robots and humans in a multi-culture society. In this paper, we propose an incremental learning model for selecting the user's representative or habitual emotional behaviors which place emphasis on individual users' cultural traits identified through long term interaction. Furthermore, a transformation model is proposed to convert the obtained emotional behaviors into a specific robot's motion space. To validate the proposed approach, the models were evaluated by two example scenarios of interaction. The experimental results confirmed that the proposed approach endows a social robot with the capability to learn emotional behaviors from individual users, and to generate its emotional bodily expressions. It was also verified that the imitated robot motions are rated emotionally acceptable by the demonstrator and recognizable by the subjects from the same cultural background with the demonstrator.

Cultural differences may influence interactions between humans with different social norms and cultural traits, incurring different emotional and behavioral responses. The same applies to human-robot interaction (HRI). We believe that controlling robot emotions based on the cultural context can help robots adapt to humans from culturally diverse backgrounds. Such culturally aligned robots are expected to be easily accepted by humans as part of daily life. In this paper, we aim at investigating the role of culture in representing robot emotions which are injected by humans during its early stage of development and subject to change through their own experience thereafter. Several public data sets of pictures labeled with affective ratings by Indian, American, and European subjects are presented to social humanoid Pepper robots. The result shows that robots can learn to behave socially in alignment with an individual's cultural background. Moreover, we have demonstrated that robots under the effect of different cultures can generate different behavioral responses to the same stimuli, which is considered one of the most important issues in socially assitive robotics.