Robotic systems for household object rearrangement often rely on latent preference models inferred from human demonstrations. While effective at prediction, these models offer limited insight into the interpretable factors that guide human decisions. We introduce an explicit formulation of object arrangement preferences along four interpretable constructs: spatial practicality (putting items where they naturally fit best in the space), habitual convenience (making frequently used items easy to reach), semantic coherence (placing items together if they are used for the same task or are contextually related), and commonsense appropriateness (putting things where people would usually expect to find them). To capture these constructs, we designed and validated a self-report questionnaire through a 63-participant online study. Results confirm the psychological distinctiveness of these constructs and their explanatory power across two scenarios (kitchen and living room). We demonstrate the utility of these constructs by integrating them into a Monte Carlo Tree Search (MCTS) planner and show that when guided by participant-derived preferences, our planner can generate reasonable arrangements that closely align with those generated by participants. This work contributes a compact, interpretable formulation of object arrangement preferences and a demonstration of how it can be operationalized for robot planning.

In human–robot teams, human situational awareness is the operator’s conscious knowledge of the team’s states, actions, plans and their environment. Appropriate human situational awareness is critical to successful human–robot collaboration. In human–robot teaming, it is often assumed that the best and required level of situational awareness is knowing everything at all times. This view is problematic, because what a human needs to know for optimal team performance varies given the dynamic environmental conditions, task context, and roles and capabilities of team members. We explore this topic by interviewing 16 participants with active and repeated experience in diverse human–robot teaming applications. Based on analysis of these interviews, we derive a framework explaining the dynamic nature of required situational awareness in human–robot teaming. In addition, we identify a range of factors affecting the dynamic nature of required and actual levels of situational awareness (i.e., dynamic situational awareness), types of situational awareness inefficiencies resulting from gaps between actual and required situational awareness, and their main consequences. We also reveal various strategies, initiated by humans and robots, that assist in maintaining the required situational awareness. Our findings inform the implementation of accurate estimates of dynamic situational awareness and the design of user-adaptive human–robot interfaces. Therefore, this work contributes to the future design of more collaborative and effective human–robot teams.

Traditional machine learning methods for movement recognition often struggle with limited model interpretability and a lack of insight into human movement dynamics. This study introduces a novel representation learning framework based on causal inference to address these challenges. Our twostage approach combines the Peter-Clark (PC) algorithm and Kullback-Leibler (KL) divergence to identify and quantify causal relationships between human joints. By capturing joint interactions, the proposed causal Graph Convolutional Network (GCN) produces interpretable and robust representations. Experimental results on the EmoPain dataset demonstrate that the causal GCN outperforms traditional GCNs in accuracy, F1 score, and recall, particularly in detecting protective behaviors. This work contributes to advancing human motion analysis and lays a foundation for adaptive and intelligent healthcare solutions.

This tutorial aims to equip researchers with the knowledge and skills to leverage multimodal data in human-robot interaction (HRI) studies. It covers the HRI study cycle, from sensor selection to data analysis, introducing commonly used sensors, pre-processing data, feature extraction techniques, fusion techniques and analysis techniques: both frequentist and Bayesian. Hands-on exercises using public datasets are designed to provide practical experience. The concluding panel discussion on ethics and bias in HRI is focused on fostering broader ethical considerations of HRI studies. Website tutorial is found online at https://sites.google.com/monash.edu/multimodal-hri-study-tutorial.

The first workshop on human-robot interaction in extreme and challenging environments (exactingHRI: https://sites.google.com/monash.edu/exactinghril) focuses on the forefront of HRI research in applications where robots are working with diverse users in uncertain, unknown, or risky environments to deliver reliable outcomes in repeated sessions. In these scenarios, a robot's autonomous and interactive functions are put to the test, with errors likely to arise. Such HRI systems require design and evaluation in-situ with target users, i.e., “exacting” HRI. The exactingHRI 2025 workshop aims to bring together researchers that investigate the diverse human, robot, task, environment, and interaction factors that are challenging for state-of-the-art HRI systems, as well as innovative designs, theories, models, and methods that equip people and robots with the ability to address these challenges. Workshop presenters will share lessons they have learned from successful or failed attempts in testing their work in such difficult settings, in a bid to encourage and guide the necessary efforts that progress our field to solve real-world problems.

The ICMI 2025 Workshop on Holistic and Responsible Affective Intelligence (HRAI 2025) aims to advance research in affective intelligence by fostering discussions on the holistic development of affective computing and the ethical challenges it entails. The workshop aims to strengthen interdisciplinary connections within the affective computing community, promoting better integration of methodologies and enhancing real-world applicability. By tackling both technical and ethical issues, HRAI 2025 aspires to shape the future of affective AI, ensuring it is not only powerful but also fair, safe, and socially responsible.

Online evaluation is increasingly adopted in robotics research, providing an efficient approach to collect data from large and diverse populations. However, there have been ongoing debates about online studies as a proxy for in-person studies, especially where a participant passively observes video of robot behaviours or interaction. We conduct an online video comparison study (N=178) evaluating three robot handover policies in a collaborative assembly task, namely an adaptive autonomous policy, a non-adaptive scripted policy, and teleoperation. Participants watched three sets of videos in third-person view, each consisting of 9 sequential handovers executing one of the policies. Compared to in-person participants in two previous studies who evaluated handovers as users, online participants were observant of different robot behaviours and human-robot collaboration contexts, with 76.4% and 71.9% recognising the adaptive handovers exhibited by the teleoperated and autonomous robot, respectively. However, as observers, online participants showed more critical subjective perceptions compared to the in-person participants with a user’s perspective. They valued efficiency over adaptation with twice more autonomous handovers rated as being too late compared to scripted handovers. Our work highlights the need to consider user contexts when evaluating human-robot collaboration.

Facial expression recognition (FER) has emerged as a promising approach to the development of emotion-aware intelligent systems. The performance of FER in multiple domains is continuously being improved, especially through advancements in data-driven learning approaches. However, a key challenge remains in utilizing FER in real-world contexts, namely ensuring user understanding of these systems and establishing a suitable level of user trust towards this technology. We conducted an empirical user study to investigate how explanations of FER can improve trust, understanding and performance in a human-computer interaction task that uses FER to trigger helpful hints during a navigation game. Our results showed that users provided with explanations of the FER system demonstrated improved control in using the system to their advantage, leading to a significant improvement in their understanding of the system, reduced collisions in the navigation game, as well as increased trust towards the system.

Service robots are increasingly employed in the hospitality industry for delivering food orders in restaurants. However, in current practice the robot often arrives at a fixed location for each table when delivering orders to different patrons in the same dining group, thus requiring a human staff member or the customers themselves to identify and retrieve each order. This study investigates how to improve the robot’s service behaviours to facilitate clear intention communication to a group of users, thus achieving accurate delivery and positive user experiences. Specifically, we conduct user studies (N=30) with a Temi service robot as a representative delivery robot currently adopted in restaurants. We investigated two factors in the robot’s intent communication, namely visualisation and movement trajectories, and their influence on the objective and subjective interaction outcomes. A robot personalising its movement trajectory and stopping location in addition to displaying a visualisation of the order yields more accurate intent communication and successful order delivery, as well as more positive user perception towards the robot and its service. Our results also showed that individuals in a group have different interaction experiences.

We study human-robot handovers in a naturalistic collaboration scenario, where a mobile manipulator robot assists a person during a crafting session by providing and retrieving objects used for wooden piece assembly (functional activities) and painting (creative activities). We collect quantitative and qualitative data from 20 participants in a Wizard-of-Oz study, generating the Functional And Creative Tasks Human-Robot Collaboration dataset (the FACT HRC dataset), available to the research community. This work illustrates how social cues and task context inform the temporal-spatial coordination in human-robot handovers, and how human-robot collaboration is shaped by and in turn influences people's functional and creative activities.

The term “aesthetic experience” corresponds to the inner state of a person exposed to the form and content of artistic objects. Quantifying and interpreting the aesthetic experience of people in different contexts can contribute towards (a) creating context and (b) better understanding people’s affective reactions to different aesthetic stimuli. Focusing on different types of artistic content, such as movies, music, literature, urban art, ancient artwork, and modern interactive technology, the goal of this workshop is to enhance the interdisciplinary collaboration among researchers coming from the following domains: affective computing, aesthetics, human-robot/computer interaction, digital archaeology and art, culture, addictive games.

Emotions and Dialog Acts (DAs) are phenomena in interpersonal communication that are informative of a person’s underlying thoughts and feelings. Thus, affective computing and dialog system researchers are interested in understanding and detecting emotions and DAs in dialogs. Previous studies on emotion recognition and dialog act classification observed that utilizing one feature for classification of the other can improve model performance, which benefits the advancement of affective dialog systems. However, theoretical explanation of such gain remains unclear. In linguistic research, the relationship between an emotion and a DA is often investigated qualitatively. In this work, we conducted both qualitative and quantitative analyses examining the relationships between emotions and DAs. Through statistical analyses, causal discovery methods, and a crowd-sourcing study, using two datasets of emotional dialog we identified emotion-DA causal pairs that provide empirical evidences to existing linguistic theories, while supporting addressing these two features together in affective dialog systems. Moreover, our work offers an effective methodology for revealing linguistic discoveries with a data-driven approach.

The term “aesthetic experience” corresponds to inner states of individuals exposed to art. Investigating form, content, and aesthetic values of artistic objects, indoor and outdoor spaces, urban areas, and modern interactive technology is essential to improve social behaviour, quality of life, and health of humans in the long term. Quantifying and interpreting the aesthetic experience of art receivers in different contexts can contribute towards (a) creating art and (b) better understanding humans’ affective reactions to aesthetic stimuli. Focusing on different types of artistic content, such as movies, music, urban art, ancient artwork, and modern interactive technology, the goal of the Second International Workshop on Multimodal Affect and Aesthetic Experience is to enhance the interdisciplinary collaboration among researchers from the following domains: affective computing, aesthetics, human-robot interaction, and digital archaeology and art.

A robot’s perceived assertiveness can influence how people assess its credibility and their willingness to comply with its suggestions during human-robot interaction. This study proposes a novel measurement of the perceived assertiveness of a robot using both objective assessment of a person’s compliance to a robot’s suggestions in a math quiz task and subjective assessments of a person’s perception and expectation of a robot in different hypothetical scenarios. The proposed measurement evaluates perceived assertiveness of a robot in three dimensions inspired by social behavioral studies on human-human interaction, namely social assertiveness, directiveness, and independence. We conducted an exploratory study using crowdsourcing to test the efficacy of this proposed measurement of perceived assertiveness. In particular, participants were exposed to robots that differed in terms of their anthropomorphism and their communicative styles. The communicative styles are designed following human studies of behaviors that are commonly associated with high or low assertiveness. Our results demonstrated the validity of the proposed measurement of perceived assertiveness. The observed discrepancy between the objective and subjective measurements highlights the necessity of evaluating human perception through multiple approaches. Moreover, gaps between intended and perceived robot assertiveness indicate a potential gap between human communication theories and their applicability to human-robot interaction.

Recognizing emotional reactions of movie audiences to affective movie content is a challenging task in affective computing. Previous research on induced emotion recognition has mainly focused on using audio-visual movie content. Nevertheless, the relationship between the perceptions of the affective movie content (perceived emotions) and the emotions evoked in the audiences (induced emotions) is unexplored. In this work, we studied the relationship between perceived and induced emotions of movie audiences. Moreover, we investigated multimodal modelling approaches to predict movie induced emotions from movie content based features, as well as physiological and behavioral reactions of movie audiences. To carry out analysis of induced and perceived emotions, we first extended an existing database for movie affect analysis by annotating perceived emotions in a crowd-sourced manner. We find that perceived and induced emotions are not always consistent with each other. In addition, we show that perceived emotions, movie dialogues, and aesthetic highlights are discriminative for movie induced emotion recognition besides spectators' physiological and behavioral reactions. Also, our experiments revealed that induced emotion recognition could benefit from including temporal information and performing multimodal fusion. Moreover, our work deeply investigated the gap between affective content analysis and induced emotion recognition by gaining insight into the relationships between aesthetic highlights, induced emotions, and perceived emotions.

Robotic applications have entered various aspects of our lives, such as health care and educational services. In such Human-robot Interaction (HRI), trust and mutual adaption are established and maintained through a positive social relationship between a user and a robot. This social relationship relies on the perceived competence of a robot on the social-emotional dimension. However, because of technical limitations and user heterogeneity, current HRI is far from error-free, especially when a system leaves controlled lab environments and is applied to in-the-wild conditions. Errors in HRI may either degrade a user’s perception of a robot’s capability in achieving a task (defined as performance errors in this work) or degrade a user’s perception of a robot’s socio-affective competence (defined as social errors in this work). The impact of these errors and effective strategies to handle such an impact remains an open question. We focus on social errors in HRI in this work. In particular, we identify the major attributes of perceived socio-affective competence by reviewing human social interaction studies and HRI error studies. This motivates us to propose a taxonomy of social errors in HRI. We then discuss the impact of social errors situated in three representative HRI scenarios. This article provides foundations for a systematic analysis of the social-emotional dimension of HRI. The proposed taxonomy of social errors encourages the development of user-centered HRI systems, designed to offer positive and adaptive interaction experiences and improved interaction outcomes.

With recent advancements in robotics and artificial intelligence, human-robot collaboration has drawn growing interests. In human collaboration, emotion can serve as an evaluation of events and as a communicative cue for people to express and perceive each other's internal states. Thus, we are motivated to investigate the influence of robots' emotional expressions on human-robot collaboration. In particular, we conducted experiments in which a participant interacted with two Cozmo robots in a collaborative game. We found that when the robots exhibited emotional expressions, participants were more likely to collaborate with them and achieved task success in shorter time. Moreover, participants perceived emotional robots more positively and reported to have a more enjoyable experience interacting with them. Our study provides insights on the benefit of incorporating artificial emotions in robots on human-robot collaboration and interaction.

Being able to detect topics and speaker stances in conversations is a key requirement for developing spoken language understanding systems that are personalized and adaptive. In this work, we explore how topic-oriented speaker stance is expressed in conversational speech. To do this, we present a new set of topic and stance annotations of the CallHome corpus of spontaneous dialogues. Specifically, we focus on six stances-positivity, certainty, surprise, amusement, interest, and comfort-which are useful for characterizing important aspects of a conversation, such as whether a conversation is going well or not. Based on this, we investigate the use of neural network models for automatically detecting speaker stance from speech in multi-turn, multi-speaker contexts. In particular, we examine how performance changes depending on how input feature representations are constructed and how this is related to dialogue structure. Our experiments show that incorporating both lexical and acoustic features is beneficial for stance detection. However, we observe variation in whether using hierarchical models for encoding lexical and acoustic information improves performance, suggesting that some aspects of speaker stance are expressed more locally than others. Overall, our findings highlight the importance of modelling interaction dynamics and non-lexical content for stance detection.

Predicting the emotional response of movie audiences to affective movie content is a challenging task in affective computing. Previous work has focused on using audiovisual movie content to predict movie induced emotions. However, the relationship between the audience's perceptions of the affective movie content (perceived emotions) and the emotions evoked in the audience (induced emotions) remains unexplored. In this work, we address the relationship between perceived and induced emotions in movies, and identify features and modelling approaches effective for predicting movie induced emotions. First, we extend the LIRIS-ACCEDE database by annotating perceived emotions in a crowd-sourced manner, and find that perceived and induced emotions are not always consistent. Second, we show that dialogue events and aesthetic highlights are effective predictors of movie induced emotions. In addition to movie based features, we also study physiological and behavioural measurements of audiences. Our experiments show that induced emotion recognition can benefit from including temporal context and from including multimodal information. Our study bridges the gap between affective content analysis and induced emotion prediction.

Automatic emotion recognition is vital for building natural and engaging human-computer interaction systems. Combining information from multiple modalities typically improves emotion recognition performance. In previous work, features from different modalities have generally been fused at the same level with two types of fusion strategies: Feature-Level fusion, which concatenates feature sets before recognition; and Decision-Level fusion, which makes the final decision based on outputs of the unimodal models. However, different features may describe data at different time scales or have different levels of abstraction. Cognitive Science research also indicates that when perceiving emotions, humans use information from different modalities at different cognitive levels and time steps. Therefore, we propose a Hierarchical fusion strategy for multimodal emotion recognition, which incorporates global or more abstract features at higher levels of its knowledge-inspired structure. We build multimodal emotion recognition models combining state-of-the-art acoustic and lexical features to study the performance of the proposed Hierarchical fusion. Experiments on two emotion databases of spoken dialogue show that this fusion strategy consistently outperforms both Feature-Level and Decision-Level fusion. The multimodal emotion recognition models using the Hierarchical fusion strategy achieved state-of-the-art performance on recognizing emotions in both spontaneous and acted dialogue.

In this work, we compare emotion recognition on two types of speech: spontaneous and acted dialogues. Experiments were conducted on the AVEC2012 database of spontaneous dialogues and the IEMOCAP database of acted dialogues. We studied the performance of two types of acoustic features for emotion recognition: knowledge-inspired disfluency and nonverbal vocalisation (DIS-NV) features, and statistical Low-Level Descriptor (LLD) based features. Both Support Vector Machines (SVM) and Long Short-Term Memory Recurrent Neural Networks (LSTM-RNN) were built using each feature set on each emotional database. Our work aims to identify aspects of the data that constrain the effectiveness of models and features. Our results show that the performance of different types of features and models is influenced by the type of dialogue and the amount of training data. Because DIS-NVs are less frequent in acted dialogues than in spontaneous dialogues, the DIS-NV features perform better than the LLD features when recognizing emotions in spontaneous dialogues, but not in acted dialogues. The LSTM-RNN model gives better performance than the SVM model when there is enough training data, but the complex structure of a LSTM-RNN model may limit its performance when there is less training data available, and may also risk over-fitting. Additionally, we find that long distance contexts may be more useful when performing emotion recognition at the word level than at the utterance level.

Automatic emotion recognition has long been a focus of Affective Computing. We aim at improving the performance of state-of-the-art emotion recognition in dialogues using novel knowledge-inspired features and modality fusion strategies. We propose features based on disfluencies and nonverbal vocalisations (DIS-NVs), and show that they are highly predictive for recognizing emotions in spontaneous dialogues. We also propose the hierarchical fusion strategy as an alternative to current feature-level and decision-level fusion. This fusion strategy combines features from different modalities at different layers in a hierarchical structure. It is expected to overcome limitations of feature-level and decision-level fusion by including knowledge on modality differences, while preserving information of each modality.