Bias in algorithmic systems is well documented, but the growing deployment of social robots in workplaces, homes, and public spaces poses a distinct challenge: their physical embodiment and social presence make bias visible in appearance, tangible in interaction, and consequential in everyday inclusion. This paper proposes a framework for understanding robotic bias across three dimensions - stereotypes, constraints, and treatments. The framework clarifies how different forms of bias emerge through embodiment and interaction, and highlights why they call for different strategies of intervention. In doing so, it provides a foundation for more systematic recognition and mitigation of bias in human-robot interaction.

Social robots are gradually being integrated into diverse social spheres, assuming multifaceted roles such as store clerks. Many of the interactive tasks entrusted to these robots are intricate, requiring more than simply responding to requests, and thus are not easily preprogrammed. One such task is sales negotiation. Empowering robots with negotiation skills is challenging due to the nuanced, context-dependent nature of effective tactics, which typically vary across different stores. To tackle this, we explore the potential of reinforcement learning without demonstrations for robots to grasp complex social interactions and adapt to specific environments, using sales negotiation as a test case. Drawing insights from interviews with experienced sales clerks and observations of negotiation role-plays, we identified essential verbal and nonverbal features. Coupled with a fast-learning reinforcement method, we evaluated the system through interactions with human participants in two distinct store settings. Results show that the robot was able to acquire negotiation strategies tailored to each environment.

Robots are often designed to help, but help is not always helpful. In everyday situations, it is a socially delicate act: the right offer of help at the wrong moment can be intrusive, unnecessary, or even undermining. In this article, we challenge the prevailing assumption that robots should always offer help, prompting an essential discussion of how robots can discern when to offer help. We introduce a theoretical framework that enables robots to assess the appropriateness of offering help by considering factors such as the relative skill levels of the robot and human user, as well as the social value and cost of assistance. To validate this framework, we conducted a large-scale online study in which participants rated the appropriateness of robot assistance across diverse task scenarios. Their responses supported our core predictions and highlighted additional contextual factors. Building on these results, we discuss potential extensions of the simplified model for real-world settings, including uncertainty management, perception of ability, autonomy preferences, and social presence. We present these directions as opportunities for future research.

In public spaces, robots often need to stop suddenly while navigating among pedestrians, such as when they need to make a quick turn or investigate something detected by their sensors. Poorly signaled stops can cause disruptions, forcing pedestrians to make an abrupt avoidance or even lead to risky collisions. To address this issue, we propose a human-inspired stopping model based on the observation of real people's stops. Interestingly, we found that people also often fail to stop without bothering those behind them, causing disruptions. In contrast, we found that stops that do not bother people following closely involve natural signals such as gradual deceleration, head motion, and body orientation changes. These cues help others anticipate upcoming changes, like an upcoming stop. Based on these observations, we implemented our model on a humanoid robot and conducted a field study in a shopping mall. The results show that incorporating these human-like signals leads to smoother interactions with pedestrians following behind, as compared to simple stopping using in standard robot navigation.

This study investigated the integration of smell detection, storage, and reproduction capabilities in robot avatars, examining how these functions impact interactions between a remote operator and a co-present partner. We developed a robot avatar system equipped with olfactory technology, using a combination of actual hardware and Wizard-of-Oz techniques. In an experimental setting, participants engaged in a task that involved spontaneous discussions about smells. Our findings show that the inclusion of olfactory functionality not only affected the dynamics of interaction but also enhanced the perceived quality of the experience for both the operator and the partner.

Recent progress in robot autonomy and safety has significantly improved human-robot interactions, enabling robots to work alongside humans on various tasks. However, complex assembly tasks still present significant challenges due to inherent task variability and the need for precise operations. This work explores deploying robots in an assistive role for such tasks, where the robot assists by fetching parts while the skilled worker provides high-level guidance and performs the assembly. We introduce GEAR, a gaze-enabled system designed to enhance human-robot collaboration by allowing robots to respond to the user’s gaze. We evaluate GEAR against a touch-based interface where users interact with the robot through a touchscreen. The experimental study involved 30 participants working on two distinct assembly scenarios of varying complexity. Results demonstrated that GEAR enabled participants to accomplish the assembly with reduced physical demand and effort compared to the touchscreen interface, especially for complex tasks, maintaining great performance, and receiving objects effectively. Participants also reported enhanced user experience while performing assembly tasks. Project page:sites.google.com/view/gear-hri

Robots are increasingly present in our everyday environments, offering services and products. But can they influence our choices through movement alone? This paper investigates whether a robot manipulator can nudge user decisions using only its arm movements, without speech, facial expressions, or physical contact. We first identified plausible nudging motions through a bodystorming session, then designed and implemented three composite nudges (positive, neutral, and negative) using a UR5 robot arm. In a video-based online study (N=35), participants more often chose positively nudged items and avoided negatively nudged ones. A small in-person study (N=9) confirmed the effect. These results demonstrate that movement-only nudges can influence decision-making and highlight the potential of subtle physical behaviors for nonverbal persuasion.

Many schools have built de-escalation and sensory rooms to support students who experience heightened emotional states, sensory overload, or difficulty self-regulating in traditional classroom settings. Yet, effective implementation remains challenging due to diverse student needs and resource constraints. Hence, we developed RESET (Robot-Enhanced Social-Emotional Therapy), a robot for facilitating students’ self-regulation in their school’s existing de-escalation space. We present our co-design process, iterative development, and final system components. Following a fully autonomous, month-long deployment in an elementary school, we assessed the robot’s usability and impacts. Results indicate RESET integrated well into the school environment, promoting more efficient deescalation, smoother transitions back to classroom learning, and lasting impacts beyond its deployment period.

Service robots in public spaces can sometimes become victims of ‘abuse’, manifesting as persistent blocking of the robot’s path, physical violence such as pushing or pulling, or abusive language. This kind of abuse can be a serious obstacle to the deployment of robots. We studied the possibility of mitigating obstructive behaviour towards the robot, which is known to happen in the early stages of robot abuse, by having the robot express anger-like emotions. We identified and implemented three distinct anger behaviours: furious, quiet and scolding anger. Through an online video survey, we confirmed that all three designed robot behaviours were perceived as expressions of anger. Finally, we ran an in-lab experiment with child participants to study the influence of the robot’s expressions of anger on children’s obstructive behaviours. The results show that ‘furious anger’ was effective in reducing the frequency of obstructions, whereas the other two anger types were not.

Avatar robots allow a teleoperator to interact with the people and environment of a remote place. Malicious operators can use this technology to perpetrate malicious or low-moral actions. In this study, we used hazard identification workshops to identify low-moral actions that are possible through the locomotor movement, cameras, and microphones of an avatar robot. We conducted three workshops, each with four potential future users of avatars, to brainstorm possible low-moral actions. As avatars are not yet widespread, we gave participants experience with this technology by having them control both a simulated avatar and a real avatar as a malicious anonymous operator in a variety of situations. They also experienced sharing space with an avatar controlled by a malicious anonymous operator. We categorized the ideas generated from the workshops using affinity diagram analysis and identified four major categories: violate privacy and security, inhibit, annoy, and destroy or hurt. We also identified subcategories for each. In the second half of this study, we discuss all low-moral action subcategories in terms of their detection, mitigation, and prevention by studying literature from autonomous, social, teleoperated, and telepresence robots as well as other fields where relevant.

We studied how robots could stop people from repeatedly obstructing them by using reactions that people commonly use. From 35 hours of observation of people in a shopping mall, we identified one commonly used reaction, which we named suggestive avoidance. It consists of making a quick movement to the side while rotating the body and gaze toward the obstructing person, in a way that seems to imply that they were bothered by the obstruction. We modeled the human suggestive avoidance behavior, implemented it on a robot, and tested it both in a lab experiment and a field study. The results from the lab study confirmed that people perceive a robot using suggestive avoidance as being more bothered, as well as more human-like. The field study showed that when a robot uses suggestive avoidance people are less likely to bother it again.

We developed an autonomous hatshop robot for encouraging customers to try on hats by providing comments that appropriately fit their actions, and in such a way also indirectly exerting social pressure. To enable it to offer such a service smoothly in a real shop, we developed a large system (around 150k lines of code with 23 ROS packages) integrated with various technologies, like people tracking, shopping activity recognition and navigation. The robot needed to move in narrow corridors, detect customers, and recognise their shopping activities. We employed an iterative development process, repeating trial-and-error integration with the robot in the actual shop, while also collecting real-world data during field-testing. This process enabled us to improve our shopping activity recognition system by collecting real-world data, and to adapt our software modules to the target shop environment. We report the lessons learnt during our system development process. The results of our 11-day field trial show that our robot was able to provide its services reasonably well. Many customers expressed a positive impression of the robot and its services.

Avatar robots are telepresence robots that allow people to project themselves physically in distant places. Teleoperators use such avatars to have remote social interactions. Mobility adds to their interactive capabilities. However, it also presents challenges in social spaces; for instance, teleoperators can mistakenly move their avatars in a socially inappropriate manner. Since these robots have yet to become ubiquitous, there is little real-world data to objectively analyze the presence and frequency of such undesirable movement. Consequently, the different ways such movements manifest are unknown, and their occurrence cannot be detected. To address this problem, we conducted a laboratory experiment with hired participants in a replica artwork exhibition so that we could gather realistic human-avatar interaction data using sensors. Our data analysis helped us identify three types of inadvertently inappropriate movement behavior. We present a definition of such behavior based on these three inappropriate movement types and a labeled dataset containing more than 400 interactions, 18% of which were inappropriate. We also present detection models for each inappropriate movement type trained using our dataset.

As robots are increasingly integrated into various aspects of everyday life, it becomes essential to develop intelligent systems capable of providing assistance while maintaining social appropriateness. In this paper, we challenge the prevailing assumption that robots should always offer help, prompting an essential discussion of when robots should offer help. We present a systematic way of considering socially appropriate assistance in human-robot interaction and introduce a theoretical framework that enables robots to discern whether or not to offer help to a human user. We examine the factors that influence the social appropriateness of help, including the relative skill levels between the robot and user and measures for assessing the social value and cost of help. Through a series of illustrative examples, we demonstrate the feasibility of our framework in providing socially appropriate assistance.

We considered the robot’s use of smells as a nonverbal way of interacting with people. In particular, we were interested in active smell delivery, which involves delivering scents to specific locations and at precise moments. Since smell is still largely unexplored as an interaction modality for human-robot interactions, many aspects are unknown. To address this gap, through exploratory studies we identified the important factors of mounting positions as well as timing and movement coordination, which need to be considered for the successful integration of active smell delivery on robots.

There has been a recent trend to test robots and intelligent virtual agents as social interaction partners in public domains. Commercial solutions such as Pepper or Cruz are increasingly being tested in scenarios outside the lab. Though at the same, time customer value and business models for social robots in public spaces are scarce, and with the recently halted production of the Pepper, it seems evident that there is no killer application for social robots in public spaces yet. This work wants to break the boundaries between academia and business and give both sides a venue to exchange lessons learned and develop a roadmap on the technical, legal, ethical, and business challenges for deploying social robots.

This article presents a cross-cultural replication of recent work on productively violating gender norms; specifically demonstrating that breaking norms can boost robot credibility while avoiding harmful stereotypes. In this work we demonstrate via a 3 (country) x 3 (robot behaviour) between-subject experiment that these findings replicate cross-culturally across the US, Sweden, and Japan, finding evidence that breaking gender norms boosts robot credibility regardless of gender or cultural context, and regardless of pretest gender biases. Our findings further motivate a call for feminist robots that subvert the existing gender norms of robot design.

Can robots influence the moral behavior of humans by simply doing their job? Robots have been considered as replacement for humans in repetitive jobs in public spaces, but what could that mean for the behavior of the surrounding people is not known. In this work we were interested to see how people change their behavior when they observe either a robot or a person do a morally-laden task. In particular, we studied the influence of seeing a robot or a human picking up and discarding garbage on the observer's willingness to litter or to pick up garbage. The study was done as a video-based survey. Results show that while observing a person clean up does make people less keen to litter, this effect is not present when people watch a robot doing the same action. Moreover, people appear to feel less guilty about littering if they observed a robot doing the cleaning up than in the case when they watched a human cleaner.

Data-driven imitation learning enables service robots to learn social interaction behaviors, but these systems cannot adapt after training to changes in the environment, such as changing products in a store. To solve this, a novel learning system that uses neural attention and approximate string matching to copy information from a product information database to its output is proposed. A camera shop interaction dataset was simulated for training/testing. The proposed system was found to outperform a baseline and a previous state of the art in an offline, human-judged evaluation.

We envision a future where service robots autonomously learn how to interact with humans directly from human-human interaction data, without any manual intervention. In this paper, we present a data-driven pipeline that: (1) takes in low-level data of a human shopkeeper interacting with multiple customers (28 hours of collected data); (2) autonomously extracts high-level actions from that data; and (3) learns -- without manual intervention -- how a robotic shopkeeper should respond to customers' actions online. Our proposed system for learning the interaction logic uses neural networks to first learn which customer actions are important to respond to and then learn how the shopkeeper should respond to those important customer actions. We present a novel technique for learning which customer actions are important by first learning the hidden causal relationship between customer and shopkeeper actions. In an offline evaluation, we show that our proposed technique significantly outperforms state-of-the-art baselines, in both which customer actions are important and how to respond to them.

Social robots often have expressive faces. However, it is not always clear how to design expressions that show a certain emotion. We present a method for a social robot to learn the emotional meaning of its own facial expressions, based on which it can automatically generate faces for any emotion. The robot collects data from an imitation game where humans are asked to mimic the robot's facial expression. The interacting person does not need to explicitly input the meaning of the robot's face so the interaction is natural. We show that humans can successfully recognise the emotions from the learned facial expressions.

We are facilitating software development for mobile social robots that operate `in the wild': that is, in real daily environments such as shopping malls. One fundamental difficulty in development is testing unfinished programs on a real robot and in the presence of people, because this process can take a significantly time-consuming. To ease the software development, we developed a simulator that allows the simulation of interactions among people, and interactions between people and the robot. With a user study, we analyzed how people's working process changed and evaluated the amount of time they saved in software development with the simulator.

In this work, we were interested in creating a robot service for offering help to people who appear to be in need for guidance. In order to achieve that, we first developed a technique to detect pedestrians who walk in an atypical way (e.g., people who do not know their way). In our approach, a motion model of typical pedestrians developed in our previous work was used, and a novel predictability feature was defined that quantifies how well can a person's future position be predicted using that model. The classification method based on this feature gave accurate results and outperformed alternative methods. Using this detection method, we created a robot service for offering guidance to people who were classified as atypical. Experiments done in a shopping mall have shown that the robot was successful in choosing the people to approach, and the reactions from users in the interviews were very positive.

Social robots working in public space often stimulate children's curiosity. However, sometimes children also show abusive behavior toward robots. In our case studies, we observed in many cases that children persistently obstruct the robot's activity. Some actually abused the robot by saying bad things, and at times even kicking or punching the robot. We developed a statistical model of occurrence of children's abuse. Using this model together with a simulator of pedestrian behavior, we enabled the robot to predict the possibility of an abuse situation and escape before it happens. We demonstrated that with the model the robot successfully lowered the occurrence of abuse in a real shopping mall.

The use of 3D range sensors for human position tracking has grown in recent years, especially for augmenting robotic sensing for human-robot interaction. However, extrinsic calibration of the relative positions of 3D range sensors is difficult, due to their limited range, narrow field of view, and distortion at large distances. 2D laser range finders have also been used for pedestrian tracking, providing greater accuracy and coverage at the cost of being more expensive and susceptible to occlusion. In this work, we present two novel techniques for calibrating the positions of 3D range sensors based on shared observations of pedestrians. The first technique uses 3D range sensors alone, and the second technique uses 2D and 3D range sensors together, using the high precision and long range of the 2D sensors to complement the short-range but richer sensing of 3D range sensors. We evaluate the accuracy of both automatic calibration techniques, and we furthermore show that the combination of 2D and 3D sensors gives more robust and accurate calibration than when using 3D sensors alone.

Knowledge about space usage from variables such as density and walking speed could support a variety of service applications. However, there is not much knowledge on how the usage of space changes during extended periods of time and what affects the changes. We have installed a person tracking system in a large area of a shopping center and collected pedestrian data over a year. In this paper, we analyze the collected data to find the changes in pedestrian density and speed, percentage of children, and pedestrian trajectories. The changes from day to day, as well as during the day are examined, and a number of factors that affect them are identified. This is in turn used in the prediction of the state of the space using a Gaussian process model.

Social robots working among pedestrians can attract crowds of people around them and consequently become bothersome entities causing congestion in narrow spaces. This in turn can affect the comfort of pedestrians who wish to pass through. To address this problem, our idea is to endow the robot with three capabilities: anticipating pedestrian crowding around the robot, understanding pedestrians' walking comfort, and planning to avoid congestions in advance. Combining several elementary pedestrian behavior models, the robot is able to simulate hypothetical situations where it navigates between pedestrians and anticipate the degree to which this would affect the pedestrians' walking comfort. During planning, the robot determines the best next navigation step based on the results of simulations. We tested the developed system in a real shopping mall and confirmed that it successfully reduces the robot's influence on pedestrian walking comfort due to congestions.

A method for tracking the position, orientation, and height of persons in large public environments is presented. Such a piece of information is known to be useful both for understanding their actions, as well as for applications such as human-robot interaction. We use multiple 3-D range sensors, which are mounted above human height to have less occlusion between persons. A computationally simple-tracking method is proposed that works on single sensor data and combines multiple sensors so that large areas can be covered with a minimum number of sensors. Moreover, it can work with different sensor types and is robust to the imperfect sensor measurements; therefore, it is possible to combine currently available 3-D range sensor solutions to achieve tracking in wide public spaces. The method was implemented in a shopping center environment, and it was shown that good tracking performance can be achieved.

The overall performance of a robotic system is commonly expressed by a single scenario-specific metric which is supposed to be optimized. However, the metric describing the performance of a single subtask within a scenario may be different. Nevertheless, the scenario performance is most likely dependent on the subtask performances but a mutual transformation is not straightforward in general, especially in complex robotic systems. This leads to what we call the common pricing problem, i.e. the problem to determine the functional relationship among a set of different performance criteria and then account for this relationship in the various optimizations throughout all system layers. In this paper we present an approach to first learn a probabilistic model of the metric interdependencies, and thereafter utilize this model for performance estimation and optimal task parameterization during planning and execution respectively. The proposed method is validated in a simulation.

In this paper we present a method for estimating the position of mobile robots using a combination of both robot’s onboard sensors and sensors at fixed locations in the environment, where we use laser range finders as sensors. This is a situation which arises in so-called Intelligent Spaces, where there are both static sensors and mobile robots present. The method we present extends the robot localization methods based on occupancy grids, however here occupancy grids are used to represent not only the geometry of the environment but also that of the robot. For tracking the robot we employ a particle filter. The details of the method are given and experimental results are shown to illustrate the method.

This paper deals with the problem of object tracking and environment mapping inside a space with distributed sensors - Intelligent Space. In a conventional approach the distributed sensors are used for these tasks, however since the sensors are static this has several disadvantages. In this paper in addition to static sensors we introduce the use of a mobile robot as mobile sensor to gather additional information and improve the estimation performance. We discuss the characteristics of such a tracking system, mainly concentrating on a system that uses laser range finders as both mobile and static sensors. Estimation methods based on Kalman Filter and Covariance Intersection are presented and analyzed. Finally, the presented methods are experimentally tested.