The first workshop on human-robot interaction in extreme and challenging environments (exactingHRI: https://sites.google.com/monash.edu/exactinghril) focuses on the forefront of HRI research in applications where robots are working with diverse users in uncertain, unknown, or risky environments to deliver reliable outcomes in repeated sessions. In these scenarios, a robot's autonomous and interactive functions are put to the test, with errors likely to arise. Such HRI systems require design and evaluation in-situ with target users, i.e., “exacting” HRI. The exactingHRI 2025 workshop aims to bring together researchers that investigate the diverse human, robot, task, environment, and interaction factors that are challenging for state-of-the-art HRI systems, as well as innovative designs, theories, models, and methods that equip people and robots with the ability to address these challenges. Workshop presenters will share lessons they have learned from successful or failed attempts in testing their work in such difficult settings, in a bid to encourage and guide the necessary efforts that progress our field to solve real-world problems.

Despite the nuclear industry’s reliance on advanced robots being operated by humans, much of the existing research overlooks the operator’s perspective in the context of nuclear decommissioning. This study aims to address this gap by identifying the specific needs and requirements of robot operators in nuclear environments. Three focus groups of experienced robot operators from the UK Atomic Energy Authority and Sellafield Ltd. were conducted to explore key themes, including the operator’s role, tasks where robots are employed, and the risks associated with robot use. Findings reveal that: (1) robots in critical tasks are typically controlled by a team of operators; (2) for human-robot interfaces safety and reliability are the most important features, before effectiveness, intuitiveness and task focus; (3) due to high task variety operators see a need for various types of robots; and (4) operator error is regarded as the most significant and unpredictable risk. Based on these insights, a comprehensive set of 10 robot-specific requirements and 10 overall user requirements has been formulated. The paper provides recommendations for robot operators and designers, detailing how these identified requirements can inform the development of future teleoperated robots for nuclear decommissioning tasks.

This paper explores the design and development of a folding robot required to survey and characterize nuclear facilities only accessible via 150 mm diameter entry ducts. The enclosed legacy facilities at old nuclear sites like Sellafield in the UK have this sort of limited access. When a site reaches the end of its operational life, it must be decommissioned and the resulting waste material must be safely disposed of. The condition, radioactive characteristics, and accessibility of the enclosed environments are unknown; for decommissioning to occur, these environments must be mapped and characterized. For a robot to carry out this task, one of the key requirements is the ability of the robot to traverse rough terrain and obstacles that could be found inside the facility. To accommodate this, while fitting through the entry duct, the chosen design utilizes morphing whegs (i.e., wheel-legs) for locomotion. These are shape-changing wheels that can open out into a set of legs that rotate around an axle, allowing greater traction, diameter, and object traversal ability than wheels alone. The design and morphology of a folding morphing-wheg robot for nuclear characterization, as well as the manufacture and testing of a prototype, is discussed in this paper. A preliminary evaluation of the robot has shown it is capable of climbing up a maximum step height of 150 mm while having a wheel dimension of 100 mm and being able to fit through a 150 mm duct.

We present the Social Group Interactions (SoGrIn) dataset; a dataset which captures non-verbal signals of groups as they complete socially collaborative and formation-provoking tasks. The dataset comprises precise proxemics (captured motion) and facial features (facial landmarks, gaze direction, facial action units) encompassing a total duration of 60 minutes involving 30 individuals, divided into six groups. Also included are basic demographic information and responses to the Big 5 personality questionnaire. The Social Group Interactions dataset is publicly available at https://doi.org/10.5281/zenodo.7778123.

Thailand, as well as many other countries world-wide, is facing a shortage of medical staff. We purpose a solution to improve medical services in health centres: a robot teleoperation system to allow patients to consult with doctors from public hospitals, and for doctors to examine and make decisions about their required care. To develop such a system, a user-centred design (UCD) process is followed. Here we present an important first step in this process to establish user requirements for such a system. Hence, we have conducted a focus group with Thai medical staff from Banphaeo General Hospital and an online survey with potential patients. An online collaborative board has been setup to facilitate running the focus group virtually and provide an effective tool to gather data. A qualitative data is then analysed using a framework analysis. Based on this work, we present a list of user requirements for doctors, patients and assistants and discuss how the collected requirements can be transferred into technical specifications of the system. Our study found that communication among different user groups is the most important requirement.

Social isolation is an issue that effects many people, especially those from ethic minorities, LGBTQIA+ communities, the elderly, those in long-term healthcare, and those living with life-limiting illnesses. It has become increasingly evident during the pandemic, when mental health issues have soared, and the importance of interacting with loved ones has been highlighted. While telecommunication software helped a great deal in these unprecedented circumstances, it does not allow for navigation in remote environments, and lacks high level interactions found in face-to-face communication. Therefore, this system has been developed to address these issues, and this study was being conducted to test the technical usability of the system when being used by healthy participants. It was found that 720p is the highest resolution that can be applied before the camera delay becomes unusable; though participants suggested that they would like the option to switch to 2k resolution should they be looking close up without moving. In addition, it became apparent that overall the participants were positive about the system, but would prefer a less bulky head-mounted display, and that the choice of which robot to use with the system (Nao or Pepper) was entirely down to individual preference based on the task being completed.

When we approach a group, there is an exchange of a multitude of verbal or non-verbal social signals to indicate that we are looking to interact. We continue to share these signals throughout the interaction to portray our thoughts and motivations. We define an interaction by the signals we send; sending different signals evokes a different response. Giving social robots the knowledge of group social interaction, they will have the ability to more effectively participate in these interactions in the real world. In this paper, we present the results from an online data collection study looking at social group dynamics. We collected a dataset of social behaviours in a group using a socially interactive game played online by 88 participants. We also introduce a novel visual social engagement metric, which is derived from two social signals: proxemics (distance between interaction participants) and mutual gaze. We propose a mathematical formula of both mutual gaze as the product of the mutual distances to the optical axis, and the visual social engagement as mutual gaze divided by distance between participants. Additionally, we investigate the influence of personality traits on the resulting interaction patterns. Using the metric, we create unique interaction profiles which suggest that participants have an interaction ‘style’. No clear correlation between personality and interaction patterns was found.

We explore the effectiveness of a dynamically processed incremental referring description system using under-specified ambiguous descriptions that are then built upon using linguistic repair statements, which we refer to as a dynamic system. We build a dynamically processed incremental referring description generation system that is able to provide contextual navigational statements to describe an object in a potential real-world situation of nuclear waste sorting and maintenance. In a study of 31 participants, we test the dynamic system in a case where a user is remote operating a robot to sort nuclear waste, with the robot assisting them in identifying the correct barrels to be removed. We compare these against a static non-ambiguous description given in the same scenario. As well as looking at efficiency with time and distance measurements, we also look at user preference. Results show that our dynamic system was a much more efficient method—taking only 62% of the time on average—for finding the correct barrel. Participants also favoured our dynamic system.

As autonomous robots move towards ubiquity, the need for robots to make decisions under risk that are trustworthy becomes increasingly significant; both to aid acceptance and to fully utilise their autonomous capabilities. We propose that incorporating a human approach to risk assessment into a robot's decision making process will increase user trust. This work investigates four robotic approaches to risk and, through a user study, explores the levels of trust placed in each. These approaches are: risk averse, risk seeking, risk neutral and a human approach to risk. Risk is artificially stimulated through performance-based compensation, in line with previous studies. The study was conducted in a virtual nuclear environment created using the Unity games engine. Forty participants were asked to complete a robot supervision task, in which they observed a robot making risk based decisions and were able to question the robot, question the robot further and ultimately accept or alter the robot's decision. It is shown that a robot that is risk seeking is significantly less trusted than a risk averse robot, a risk neutral robot and a robot utilising human approach to risk. There was found to be no significant difference between the levels of trust placed in the risk averse, risk neutral and human approach to risk. It is also found that the level to which participants question a robot's decisions does not form an accurate measure of trust. The results suggest that when designing a robot that must make risk based decisions during teleoperation in a hazardous environment, an engineer should avoid a risk seeking robot. However, that same engineer may choose whichever of the remaining risk profiles best suits the implementation, with knowledge that the trust in their system is unlikely to be significantly affected.

This study investigated whether virtual reality could be used as platform for conducting human-robot interaction user studies. It was investigated whether user studies performed in virtual reality elicited realistic responses from participants. To answer this question, a real world study was replicated as closely as possible in virtual reality, where a robot tour guide asked participants to keep a secret. The experiment consisted of a virtual museum tour where the robot acted as the tour guide while displaying either social or non-social behaviour. The measurements taken in this study were the objective measurement whether the participants kept the robot's secret or not. Questionnaires were taken to investigate participants' perception of the robot and its feelings, as well as their experienced level of presence and their tendency to become immersed in the virtual environment. Results show that the participants responded differently in the virtual reality study when compared to the original real world study, where the secret was kept more often for the non-social robot, but less often for the social robot. In both the original and replicated study a strong, positive correlation was found between participants' perception of the robot as a social being and their tendency to keep the robot's secret. These inconclusive findings, some changes that were required for the virtual environment compared to the original study, and different participant demographics indicate that more work is needed to determine whether virtual reality can be used as a tool to conduct human-robot interaction experiments.

We describe a novel method for automatic detection of errors in human-robot interactions. Our approach is to detect errors based on the classification of head and shoulder movements of humans who are interacting with erroneous robots. We conducted a user study in which participants interacted with a robot that we programmed to make two types of errors: social norm violations and technical failures. During the interaction, we recorded the behavior of the participants with a Kinect v1 RGB-D camera. Overall, we recorded a data corpus of 237,998 frames at 25 frames per second; 83.48% frames showed no error situation; 16.52% showed an error situation. Furthermore, we computed six different feature sets to represent the movements of the participants and temporal aspects of their movements. Using this data we trained a rule learner, a Naive Bayes classifier, and a k-nearest neighbor classifier and evaluated the classifiers with 10-fold cross validation and leave-one-out cross validation. The results of this evaluation suggest the following: (1) The detection of an error situation works well, when the robot has seen the human before; (2) Rule learner and k-nearest neighbor classifiers work well for automated error detection when the robot is interacting with a known human; (3) For unknown humans, the Naive Bayes classifier performed the best; (4) The classification of social norm violations does perform the worst; (5) There was no big performance difference between using the original data and normalized feature sets that represent the relative position of the participants.

In this article we discuss movement control of a ReMeDi medical mobile robot from the user perspective. The control is essentially limited to the level of operator actions where the operator is a member of a nursing staff. Two working modes are the base of considerations: long distance (LD) and short distance (SD) movement. In this context two robot control techniques are the subject of study: manual with use of a gamepad and “point and click” on a map that is related to autonomous motion with use of an onboard navigation system. In the SD mode the user manually operates the robot, that is close to a laying down patient on a settee. In the LD mode the mobile base moves autonomously in a space shared with people to the desired position. Two user studies were conducted. The results show that from the perspective of LD mode the autonomous navigation is efficient and reduces the burden of the medical personnel. In the SD case, the results show that the users were able to precisely position the robot. Besides, the users perceived the manual control with the gamepad as intuitive. In all cases the medical personnel consider this technology as safe and useful. Safety is also confirmed by patients.

Humor in robotics is a promising, though not yet significantly researched topic. We performed a user study exploring two different kinds of laughter. In our study, participants observed a robot-robot interaction where an iCat and a NAO robot exhibited different laughing behavior. While NAO laughed at itself (self-irony), the iCat laughed at NAO (Schadenfreude1). Our participants watched four turns of the same robot-robot interaction, with either NAO or the iCat laughing, both robots laughing, or no robot laughing (baseline). After each turn we asked the participants to rate both robots' likability individually. Our results show that the participants liked a robot with a positively attributed form of humor significantly more than its gloating robotic interaction partner. However, likability ratings showed a trend to approach each other when either robot laughed or when both robots laughed together. Both, the higher likability ratings for a robot showing positively attributed humor and the decreasing difference in likability ratings when both robots laugh together, provide proof of the positive effect of humor. While participants' age did not affect likability ratings, there was a significant interaction effect between participants' gender and robot type. Female participants rated the iCat more likable, while male participants liked NAO better. In addition, more neurotic people liked the self-ironic robot more when no robot laughed and more open people like the robot showing Schadenfreude more when both robots laughed.

Augmented reality (AR) can serve as a tool to provide helpful information in a direct way to industrial robot programmers throughout the teaching process. It seems obvious that AR support eases the programming process and increases the programmer's productivity and programming accuracy. However, additional information can also potentially increase the programmer's perceived workload. To explore the impact of augmented reality on robot teaching, as a first step we have chosen a Sphero robot control scenario and conducted a within-subject user study with 19 professional industrial robot programmers, including novices and experts. We focused on the perceived workload of industrial robot programmers and their task completion time when using a tablet-based AR approach with visualization of task-based information for controlling a robot. Each participant had to execute three typical robot programming tasks: tool center point teaching, trajectory teaching, and overlap teaching. We measured the programmers' workload in the dimensions of mental demand, physical demand, temporal demand, frustration, effort, and performance. The study results show that the presentation of task-based information in the tablet-based AR interface decreases the mental demand of the industrial robot programmers during the robot control process. At the same time, however, the programmers' task completion time increases.

We present the design for a three-way medical teleconference system for communication between a doctor, a patient, and an assistant. The system includes individual doctor-patient and doctor-assistant communication channels, as well as the capability of starting and stopping communication channels separately. The initial system design is based on results of a user requirement analysis. To evaluate the design, we conducted two user studies in which doctors, assistants, and patients used our teleconference system in a simulated examination scenario. The study results show that the general usability of our system was rated as good. However, doctors, patients, and assistants reported that they would like to receive better visualisation of the connection status for the communication channels and the system status of the robot in general. Based on these results, we present an updated design for the teleconference system. We also provide best practices which can help designers of medical teleconference systems.

We report the results of a user requirements analysis for a medical robotic system that enables doctors to remotely conduct ultrasonography and physical examination on patients. As there are three different user groups in this scenario - doctors, patients, and assistants - we collected user requirements for all of these groups. This analysis forms a basis for the technical specification of the medical robotic system. To gather the user requirements, we conducted a literature review, observed two examinations of a patient conducted by a doctor, organised four workshops with doctors and patients, and quantified the qualitative data in two online surveys. The most important findings of the requirements analysis are that doctors need accurate kinesthetic, tactile, and audiovisual feedback for a proper diagnosis. They need additional patient data apart from the ultrasonography and physical examination (e.g., olfactory information and skin wetness). Doctors, patients, and assistants all want to have a secure audiovisual communication channel during the whole examination and especially patients have concerns regarding safety of the robot arm and data privacy. We present a list of requirements for doctors, patients, and assistants, and discuss their implications for the technical specifications of the system.

We describe the design of the newly developed Ghost-in-the-Machine paradigm and present initial results of an experiment addressing the initiation of service interactions at a bar. For developing policies for a robotic bartender, we investigated which sensor modalities were most informative to humans, and which actions they selected as a socially appropriate response. The results showed that participants used two nonverbal cues for their initial response to a new customer. Those were the distance to the bar and whether the customers' torso was directed to the bar. For acknowledging a new customer, the participants typically responded nonverbally by looking and smiling at the customers. All results can be directly transferred into robotic decision policies.

The Workshop on Multimodal, Multi-Party, Real-World Human-Robot Interaction will be held in Istanbul on 16 November 2014, co-located with the 16th International Conference on Multimodal Interaction (ICMI 2014). The workshop objective is to address the challenges that robots face when interacting with humans in real-world scenarios. The workshop brings together researchers from intention and activity recognition, person tracking, robust speech recognition and language processing, multimodal fusion, planning and decision making under uncertainty, and service robot design. The programme consists of two invited talks, three long paper talks, and seven late-breaking abstracts. Information on the workshop and pointers to workshop papers and slides can be found at http://www.macs.hw.ac.uk/~mef3/icmi-2014-workshop-hri/.

This paper deals with recognizing human actions in depth video data. Current state-of-the-art action recognition methods use hand-designed features, which are difficult to produce and time-consuming to extend to new modalities. In this paper, we propose a novel, 3.5D representation of a depth video for action recognition. A 3.5D graph of the depth video consists of a set of nodes that are the joints of the human body. Each joint is represented by a set of spatio-temporal features, which are computed by an unsupervised learning approach. However, if occlusions occur, the 3D positions of the joints are noisy which increases the intra-class variations in action classes. To address this problem, we propose the Ensemble Weighted Multi-Instance Learning approach (EnwMi) for the action recognition task. It considers the class imbalance and intra-class variations. We formulate the action recognition task with depth videos as a weighted multi-instance problem. We further integrate an ensemble learning method into the weighted multi-instance learning framework. Our approach is evaluated on Microsoft Research Action3D dataset, and the results show that it outperforms state-of-the-art methods.

In this paper we present results from a user evaluation of a robot bartender system which handles state uncertainty derived from speech input by using belief tracking and generating appropriate clarification questions. We present a combination of state estimation and action selection components in which state uncertainty is tracked and exploited, and compare it to a baseline version that uses standard speech recognition confidence score thresholds instead of belief tracking. The results suggest that users are served fewer incorrect drinks when the uncertainty is retained in the state.

When humans speak they often use gram-matically incorrect sentences, which is a problem for grammar-based language pro-cessing methods, since they expect in-put that is valid for the grammar. We present two methods to transform spoken language into grammatically correct sen-tences. The first is an algorithm for au-tomatic ellipsis detection, which finds el-lipses in spoken sentences and searches in a combinatory categorial grammar for suitable words to fill the ellipses. The sec-ond method is an algorithm that computes the semantic similarity of two words us-ing WordNet, which we use to find alter-natives to words that are unknown to the grammar. In an evaluation, we show that the usage of these two methods leads to an increase of 38.64 % more parseable sen-tences on a test set of spoken sentences that were collected during a human-robot interaction experiment. 1

This paper describes a context-dependent social gaze-control system implemented as part of a humanoid social robot. The system enables the robot to direct its gaze at multiple humans who are interacting with each other and with the robot. The attention mechanism of the gaze-control system is based on features that have been proven to guide human attention: nonverbal and verbal cues, proxemics, the visual field of view, and the habituation effect. Our gaze-control system uses Kinect skeleton tracking together with speech recognition and SHORE-based facial expression recognition to implement the same features. As part of a pilot evaluation, we collected the gaze behavior of 11 participants in an eye-tracking study. We showed participants videos of two-person interactions and tracked their gaze behavior. A comparison of the human gaze behavior with the behavior of our gaze-control system running on the same videos shows that it replicated human gaze behavior 89% of the time.

A robot agent existing in the physical world must be able to understand the social states of the human users it interacts with in order to respond appropriately. We compared two implemented methods for estimating the engagement state of customers for a robot bartender based on low-level sensor data: a rule-based version derived from the analysis of human behaviour in real bars, and a trained version using supervised learning on a labelled multimodal corpus. We first compared the two implementations using cross-validation on real sensor data and found that nearly all classifier types significantly outperformed the rule-based classifier. We also carried out feature selection to see which sensor features were the most informative for the classification task, and found that the position of the head and hands were relevant, but that the torso orientation was not. Finally, we performed a user study comparing the ability of the two classifiers to detect the intended user engagement of actual customers of the robot bartender; this study found that the trained classifier was faster at detecting initial intended user engagement, but that the rule-based classifier was more stable.

We address the question of whether service robots that interact with humans in public spaces must express socially appropriate behaviour. To do so, we implemented a robot bartender which is able to take drink orders from humans and serve drinks to them. By using a high-level automated planner, we explore two different robot interaction styles: in the task only setting, the robot simply fulfils its goal of asking customers for drink orders and serving them drinks; in the socially intelligent setting, the robot additionally acts in a manner socially appropriate to the bartender scenario, based on the behaviour of humans observed in natural bar interactions. The results of a user study show that the interactions with the socially intelligent robot were somewhat more efficient, but the two implemented behaviour settings had only a small influence on the subjective ratings. However, there were objective factors that influenced participant ratings: the overall duration of the interaction had a positive influence on the ratings, while the number of system order requests had a negative influence. We also found a cultural difference: German participants gave the system higher pre-test ratings than participants who interacted in English, although the post-test scores were similar.

Robot task planning is an inherently challenging problem, as it covers both continuous-space geometric reasoning about robot motion and perception, as well as purely symbolic knowledge about actions and objects. This paper presents a novel “knowledge of volumes” framework for solving generic robot tasks in partially known environments. In particular, this approach (abbreviated, KVP) combines the power of symbolic, knowledge-level AI planning with the efficient computation of volumes, which serve as an intermediate representation for both robot action and perception. While we demonstrate the effectiveness of our framework in a bimanual robot bartender scenario, our approach is also more generally applicable to tasks in automation and mobile manipulation, involving arbitrary numbers of manipulators.

We introduce a humanoid robot bartender that is capable of dealing with multiple customers in a dynamic, multi-party social setting. The robot system incorporates state-of-the-art components for computer vision, linguistic processing, state management, high-level reasoning, and robot control. In a user evaluation, 31 participants interacted with the bartender in a range of social situations. Most customers successfully obtained a drink from the bartender in all scenarios, and the factors that had the greatest impact on subjective satisfaction were task success and dialogue efficiency.

Robots that interact with humans in everyday situations, need to be able to interpret the nonverbal social cues of their human interaction partners. We show that humans use body posture and head pose as social signals to initiate and terminate interaction when ordering drinks at a bar. For that, we record and analyze 108 interactions of humans interacting with a human bartender. Based on these findings, we train a Hidden Markov Model (HMM) using automatic body posture and head pose estimation. With this model, the bartender robot of the project JAMES can recognize typical social behaviors of human customers. Evaluation shows a recognition rate of 82.9 % for all implemented social behaviors and in particular a recognition rate of 91.2 % for bartender attention requests, which will allow the robot to interact with multiple humans in a robust and socially appropriate way.

We present MultiML, a markup language for the annotation of multimodal human utterances. MultiML is able to represent input from several modalities, as well as the relationships between these modalities. Since MultiML separates general parts of representation from more context-specific aspects, it can easily be adapted for use in a wide range of contexts. This paper demonstrates how speech and gestures are described with MultiML, showing the principles - including hierarchy and underspecification - that ensure the quality and extensibility of MultiML. As a proof of concept, we show how MultiML is used to annotate a sample human-robot interaction in the domain of a multimodal joint-action scenario.