Studies in visual cognition highlight the importance of visual, spatial, and auditory cues in influencing human attention. Such cues often tend to be indicative of actions or events, thereby serving as predictive indicators in both passive observation as well as in interactive engagement. Our research focuses on visual attention in passive observation, particularly examining the manner in which visual, spatial, and auditory cues—henceforth visuo-auditory (shorthand) cues—influence attention on everyday multimodal interaction. We systematically develop a visuo-auditory event model for investigating visual attention in naturalistic embodied settings. Rooted in this event model, we explore the influence of five select visuo-auditory cues—namely, speaking, gaze, relative motion, hand action, and visibility—on visual attention. Our analysis utilizes eye-tracking data from 90 participants observing 27 carefully designed naturalistic event scenarios and correlating their attentional metrics with the select visuo-auditory cues in the backdrop of the developed event model. Findings reveal strong associations between attention and both intra-modal (irrespective of other cues) and cross-modal (combined with other cues) cueing effects, thereby highlighting the nuanced interplay amongst the cues influencing attentional patterns. We develop a systematic and generalized method for analyzing interactions and behavioral parameters, thereby characterizing the impact of visuo-auditory cues on attentional dynamics. Our methodology, combined with the obtained insights into the attentional cueing effects, provides an analytical framework explicating the manner in which everyday (interactive) events directly drive attention under naturalistic conditions. This facilitates not only the precise modeling of behavior and attention allocation but also offers a high-level “experimental lens” for examining interactions in relation to behavioral parameters. Taken together, our methodological and behavioral findings are well-positioned to benefit multiple fields, particularly by advancing human-centered design across diverse application domains. Lastly, towards promoting open-science and for wider dissemination, the complete experimental basis of this research—e.g., event-scenarios, high-quality annotated data, dataset supplementary—has been documented together with instructions on how to use and access experimental data ( https://codesign-lab.org/cognitive-vision/multimodal-cues ).

The use of social robots in many sectors of society is predicted to progressively increase. Therefore, exploring how expectations play a role in and change users’ experiences when interacting with these robots over time is necessary. From an interpretative and insight-driven approach, our aim was to explore how humans experience in-person interactions with the social robot Pepper, which was equipped with the OpenAI GPT-3 language model. Qualitative data from 62 video recordings of the interactions with Pepper and post-test interviews were collected from 31 participants. An experiential reflexive thematic analysis was applied. The main findings include various levels of interaction quality, different interaction strategies, and elements influencing the users’ expectations and experiences, which were synthesized into a coherent framework. It appears that the participants adapted their interaction strategies based on their expectations and the perceived capability of the robot, which influenced their experiences. This reveals that positive user experience is not solely determined by interaction quality, showing the interplay among these aspects when interacting with a social robot. To conclude, our findings underscore the intricate nature of the role of user expectations and experiences in social human–robot interaction. The work adds complementary qualitative approaches to the Human–Robot Interaction community to provide additional insights on interacting with social robots.

Robots are increasingly used in shared environments with humans, making effective communication a necessity for successful human-robot interaction. In our work, we study a crucial component: active communication of robot intent. Here, we present an anthropomorphic solution where a humanoid robot communicates the intent of its host robot acting as an “Anthropomorphic Robotic Mock Driver” (ARMoD). We evaluate this approach in two experiments in which participants work alongside a mobile robot on various tasks, while the ARMoD communicates a need for human attention, when required, or gives instructions to collaborate on a joint task. The experiments feature two interaction styles of the ARMoD: a verbal-only mode using only speech and a multimodal mode, additionally including robotic gaze and pointing gestures to support communication and register intent in space. Our results show that the multimodal interaction style, including head movements and eye gaze as well as pointing gestures, leads to more natural fixation behavior. Participants naturally identified and fixated longer on the areas relevant for intent communication, and reacted faster to instructions in collaborative tasks. Our research further indicates that the ARMoD intent communication improves engagement and social interaction with mobile robots in workplace settings.

The Social Robots in Therapy workshop series aims at advancing research topics related to the use of robots in the contexts of Social Care and Robot-Assisted Therapy (RAT). Robots in social care and therapy have been a long time promise in HRI as they have the opportunity to improve patients life significantly. Multiple challenges have to be addressed for this, such as building platforms that work in proximity with patients, therapists and health-care professionals; understanding user needs; developing adaptive and autonomous robot interactions; and addressing ethical questions regarding the use of robots with a vulnerable population. The full-day workshop follows last year's edition which centered on how social robots can improve health-care interventions, how increasing the degree of autonomy of the robots might affect therapies, and how to overcome the ethical challenges inherent to the use of robot assisted technologies. This 2ndedition of the workshop will be focused on the importance of equipping social robots with socio-emotional intelligence and the ability to perform meaningful and personalized interactions. This workshop aims to bring together researchers and industry experts in the fields of Human-Robot Interaction, Machine Learning and Robots in Health and Social Care. It will be an opportunity for all to share and discuss ideas, strategies and findings to guide the design and development of robot-assisted systems for therapy and social care implementations that can provide personalize, natural, engaging and autonomous interactions with patients (and health-care providers).

This paper presents an ongoing study on affective human-robot interaction. In our previous research, touch type is shown to be informative for communicated emotion. Here, a soft matrix array sensor is used to capture the tactile interaction between human and robot and 6 machine learning methods including CNN, RNN and C3D are implemented to classify different touch types, constituting a pre-stage to recognizing emotional tactile interaction. Results show an average recognition rate of 95% by C3D for classified touch types, which provide stable classification results for developing social touch technology.

In the present study, 64 users were asked to convey eight distinct emotion to a humanoid Nao robot via touch, and were then asked to evaluate their experiences of performing that task. Large differences between emotions were revealed. Users perceived conveying of positive/pro-social emotions as significantly easier than negative emotions, with love and disgust as the two extremes. When asked whether they would act differently towards a human, compared to the robot, the users' replies varied. A content analysis of interviews revealed a generally positive user experience (UX) while interacting with the robot, but users also found the task challenging in several ways. Three major themes with impact on the UX emerged; responsiveness, robustness, and trickiness. The results are discussed in relation to a study of human-human affective tactile interaction, with implications for human-robot interaction (HRI) and design of social and affective robotics in particular.

Over the last few years, technological developments in semi-autonomous machines have raised awareness about the strategic importance of human-robot interaction (HRI) and its technical and social implications. At the same time, HRI still lacks an established pedagogic tradition in the coordination of its intrinsically interdisciplinary nature. This scenario presents steep and urgent challenges for HRI education. Our contribution presents a normative interdisciplinary dialogic framework for HRI education, denoted InDia wheel, aimed toward seamless and coherent integration of the variety of disciplines that contribute to HRI. Our framework deemphasizes technical mastery, reducing it to a necessary yet not sufficient condition for HRI design, thus modifying the stereotypical narration of HRI-relevant disciplines and creating favorable conditions for a more diverse participation of students. Prospectively, we argue, the design of an educational 'space of interaction’ that focuses on a variety of voices, without giving supremacy to one over the other, will be key to successful HRI education and practice.

Two methods for behavior recognition are presented and evaluated. Both methods are based on the dynamic temporal difference algorithm Predictive Sequence Learning (PSL) which has previously been proposed as a learning algorithm for robot control. One strength of the proposed recognition methods is that the model PSL builds to recognize behaviors is identical to that used for control, implying that the controller (inverse model) and the recognition algorithm (forward model) can be implemented as two aspects of the same model. The two proposed methods, PSLE-Comparison and PSLH-Comparison, are evaluated in a Learning from Demonstration setting, where each algorithm should recognize a known skill in a demonstration performed via teleoperation. PSLH-Comparison produced the smallest recognition error. The results indicate that PSLH-Comparison could be a suitable algorithm for integration in a hierarchical control system consistent with recent models of human perception and motor control.