Field studies are irreplaceable but costly, time-consuming, and error-prone, which need careful preparation. Inspired by rapid-prototyping in manufacturing, we propose a fast, low-cost evaluation method using Vision-Language Model (VLM) personas to simulate outcomes comparable to field results. While LLMs show human-like reasoning and language capabilities, autonomous vehicle (AV)-pedestrian interaction requires spatial awareness, emotional empathy, and behavioral generation. This raises our research question: To what extent can VLM personas mimic human responses in field studies? We conducted parallel studies: 1) one real-world study with 20 participants, and 2) one video-study using 20 VLM personas, both on a street-crossing task. We compared their responses and interviewed five HCI researchers on potential applications. Results show that VLM personas mimic human response patterns (e.g., average crossing times of 5.25 s vs. 5.07 s) lack the behavioral variability and depth. They show promise for formative studies, field study preparation, and human data augmentation.

We introduce a novel immersive social teleoperation interface to perform social interaction intuitively and semi-automatic locomotion simultaneously. Teleoperated robots in complex, human-centric social environments present a significant challenge on simultaneously managing intricate navigation while engaging in natural social interaction. This dual task imposes a high cognitive load, hindering the fluidity and quality of social interaction. As existing systems typically prioritize either task-oriented teleoperation control or non-moving social interaction, failing to integrate dynamic locomotion with social engagement effectively. Our system combines a head-mounted display with a wide-angle, 270-degree video feed to support extensive situation awareness and a strong sense of presence to overcome these limitations by fostering an immersive and socially aware experience. Our interface performs low-level navigation of the robot by pointing at a place to go and selecting a person to follow. The operator can focus on high-level goals (social interactions). We evaluated our interface through a rigorous field experiment, using a testbed (remote guide) scenario developed through iterative pilot studies in a real-world shopping mall. Our findings demonstrate that the operator's task performance improves statistically significantly. We report other findings and discuss limitations and future improvements. In short, our novel interface, integrating immersive visualization with autonomous navigation, enables operators to achieve a more intuitive and engaging social interaction in dynamic remote social environments.

We propose a method to automatically assess group openness by observing interactions within socially occupied spaces. Openness refers to a group’s predisposition to allow external parties to join a social encounter. Robots need the capability of assessing openness to ensure social comfort and improve first impressions, saving time and resources by avoiding unwelcomed interactions. We extract social signals from gestures, posture, space and constellations using skeleton data from depth sensors. We collected 88 unique around 5-minute group interactions involving 82 Japanese participants across two scenarios: one with an external focus of attention (a poster illustration on the wall, Poster scenario) and another without one (standing conversation, No Poster scenario). We trained a linear classifier optimised with a Stochastic Gradient Descent on raw data, as well as aggregated bodily motion and spatial cues with a full and a half temporal overlap by implementing a sliding window temporal analysis across observation windows, demonstrating its capability to assess group openness accurately. Our approach was evaluated across temporal analysis, scenario comparison and group sizes, reaching 79% peak prediction accuracy and an F1 score of 0.8201 for the No Poster scenario and 78% with a 0.8147 F1 score for the Poster scenario, outperforming a human baseline of 50% and 70%, respectively. Feature importance scores suggest groups display different cues for openness in each scenario. Important cues for the No Poster scenario included tightness, group size, body crunching, right arm extension and mouth covering. For the Poster scenario, relevant cues were arm extension, distance to the wall, gap between members and tightness. These findings contribute to understanding group openness prior to engagement, improving robots’ social intelligence in successfully approaching human groups.

We explored whether and how individuals attribute authority to a robot. Online participants ( N = 362 Japanese adults) received short videos showing a social interaction between a group of three human actors and a lone agent. Crucially, in the videos, the group interacted with a robot or a fourth human actor. The group bowed deeply for the lone agent. After each video, participants were asked to say to what extent the lone agent can be described as a source of authority ( authority attribution ) and to what extent the group would comply with an order issued by the lone agent ( obedience expectation ). We found that participants perceived the robot as in charge and expected the group to comply with its orders, although participants attributed less authority to the robot than to the human actor and expected the group to obey the human actor more than the robot. As such, we conclude that bowing as a social cue for authority attribution applies to interactions between robots and humans. These findings can have important applications for designing authority- or leader-like robot figures.

When verbal communication is limited, robots passing objects to humans without providing additional information (e.g., temperature, weight) can result in potential poor handovers and disappointing user experiences. To address this issue, we introduced a method for conveying object properties through robot manipulation. We began by proposing four criteria for selecting properties from two widely recognized sets: one focusing on the semantic features of objects and the other on tactile sensations. These properties were clustered into eight physical categories: hot, cold, heavy, light, slippery, sticky, fragile, and smelly. Professional actors were then recruited to demonstrate these properties through object manipulation, from which we extracted a set of fundamental yet expressive manipulation behaviors, i.e., key elements, that help people recognize these properties. These elements were implemented on a dual-arm robot, followed by an evaluation of their utility through participant feedback. To generate time-constrained sequences of elements, we developed a property-based motion planner that balances time and utility in conveying object properties. Results from a within-subjects study involving 20 participants showed that individuals could accurately interpret the properties conveyed by robot object manipulation, validating the effectiveness of the proposed approach.

When employing autonomous driving technology in logistics, small autonomous delivery vehicles (aka delivery robots) encounter challenges different from passenger vehicles when interacting with other road users. We conducted an online video survey as a pre-study and found that autonomous delivery vehicles need external human-machine interfaces (eHMIs) to ask for help due to their small size and functional limitations. Inspired by everyday human communication, we chose arms as eHMI to show their request through limb motion and gesture. We held an in-house workshop to identify the arm’s requirements for designing a specific arm with shrink-ability (conspicuous when delivering messages but not affect traffic at other times). We prototyped a small delivery robot with a shrinkable arm and filmed the experiment videos. We conducted two studies (a video-based and a 360-degree-photo VR-based) with 18 participants. We demonstrated that arm-on-delivery robots can increase interaction efficiency by drawing more attention and communicating specific information.

While text-based external human-machine interface (eHMI) is widely accepted, one limitation is the lack of capability to communicate spatial information such as a different person or location. We built a mixed-eHMI using "eye" as a target-specifier when "text" shows the clear intention to their communication partners. We conducted a pre-experimental observation to develop two testbed scenarios, followed by a video-based user study via life-size projection with a real-car prototype mounted a text display and a set of robotic eyes. The results demonstrated that our proposed "text + eye" combination may represent geospatial information by increasing the success pick-up rate.

To personalize the robot guide experience, the robot needs to detect a person's indifference and adjust its explanation toward the person's interest in topics. However, detecting the person's indifference is challenging in a museum, as we cannot use a bulky wearable or facial expression recognition due to unexpected light condition or standing position. We propose to observe people's behaviors and movements on detecting people's indifference. To prove its feasibility, we invited 11 participants to our in-lab museum-like environment. Our robot explains exhibits while videorecording the interaction. Then, we asked participants to watch the recordings and report when they felt bored or indifferent to the explanation. We labelled their movement and matched them to their report so that we know which behaviors and movements hint the person's indifference. We used the decision tree and random forest methods to understand the common pattern when people are indifferent during the explanation in a museum scenario. From our observation experiment, we found that if the listener nods their heads many times or looks at the exhibit for a long time, they are likely interested in the topic, fewer overall movements or looking elsewhere hint that the listener may be indifferent, and if the explanation goes longer than three minutes, the listener would be likely bored.

Robots are increasingly becoming present in workplaces and social scenarios, where people are interacting and completing tasks with them. In this paper, we explore the importance of the physical seating arrangement or placement of the robot in relation to the human. We designed a novel collaborative human-robot task for exploring how human-robot seating arrangement may influence interaction and the person's perceptions of a robot, and present the results from a pilot and two formal experiments (total 72 participants). Drawing from proxemics literature, we compared a person sitting next to a robot to being across a table from it or beside it, with a corner between them. Our results highlight a range of impacts on participant attitudes toward the robot, as well as their behavior and interaction with it. In particular, seating arrangement impacted participant preferences for the robot, and their use of aggression or condescension during disagreements with it. Our results highlight the importance of seating arrangements for collaborative human-robot teams.

Social robots are being designed to use human-like communication techniques, including body language, social signals, and empathy, to work effectively with people. Just as between people, some robots learn about people and adapt to them. In this paper we present one such robot design: we developed Sam, a robot that learns minimal information about a person's background, and adapts to this background. Our in-the-wild study found that people helped Sam for significantly longer when it adapted to match their background. While initially we saw this as a success, in re-considering our study we started seeing a different angle. Our robot effectively deceived people (changed its story and text), based on some knowledge of their background, to get more work from them. There was little direct benefit to the person from this adaptation, yet the robot stood to gain free labor. We would like to pose the question to the community: is this simply good robot design, or, is our robot being manipulative? Where does the ethical line lay between a robot leveraging social techniques to improve interaction, and the more negative framing of a robot or algorithm taking advantage of people? How can we decide what is good here, and what is less desirable?

We designed and evaluated a series of teleoperation interface techniques that aim to draw operator attention while mitigating negative effects of interruption. Monitoring live teleoperation video feeds, for example to search for survivors in search and rescue, can be cognitively taxing, particularly for operators driving multiple robots or monitoring multiple cameras. To reduce workload, emerging computer vision techniques can automatically identify and indicate (cue) salient points of potential interest for the operator. However, it is not clear how to cue such points to a preoccupied operator -- whether cues would be distracting and a hindrance to operators -- and how the design of the cue may impact operator cognitive load, attention drawn, and primary task performance. In this paper, we detail our iterative design process for creating a range of visual attention-grabbing cues that are grounded in psychological literature on human attention, and two formal evaluations that measure attention-grabbing capability and impact on operator performance. Our results show that visually cueing on-screen points of interest does not distract operators, that operators perform poorly without the cues, and detail how particular cue design parameters impact operator cognitive load and task performance. Specifically, full-screen cues can lower cognitive load, but can increase response time; animated cues may improve accuracy, but increase cognitive load. Finally, from this design process we provide tested, and theoretically grounded cues for attention drawing in teleoperation.

Robot teleoperation, such as for search and rescue, uses multiple specialized cameras (e.g., wide environmental and sharp narrow views) to aid in task awareness. Simple display techniques, such as tiling, require ongoing mental mapping between the views; cameras that pan or tilt exacerbate the problem as the inter-view relationship changes. The detail-in-context technique bypasses this mental mapping requirement by providing a single integrated feed showing all cameras, with detail overlaid within the context. However, how this can be adapted to for robot teleoperation with multiple pan-and-tilt cameras has not yet been demonstrated. We present Monocle, an interactive detail-in-context teleoperation interface that integrates a pan-and-tilt narrow-angle first-person view into a wide-angle behind-robot view; operators can move the Monocle around a scene to obtain more resolution when and where needed. Evaluation results demonstrate Monocle's feasibility and show that it can help operators complete search and rescue tasks more effectively in comparison to simple solutions.

We present a set of mini-map alternatives for indicating the relative locations of robot team members in a tele-operation interface, and evaluation results showing that these can perform as well as mini-maps while being less intrusive. Teleoperation operators often work with a team of robots to improve task effectiveness. Maintaining awareness of where robot team members are, relative to oneself, is important for team effectiveness, such as for deciding which robot may help with a task, may be best suited to investigate a point of interest, or to determine where one should move next. We explore the use of established interface techniques from mobile computing for supporting teleoperators in maintaining peripheral awareness of robot team members' relative locations. We evaluate the nontrivial adoption of these techniques to teleoperation, comparing to an overview mini-map base case. Our results indicate that in-feed embedded indicators perform comparatively well to mini-maps, while being less obtrusive, indicating that they are a viable alternative for teleoperation interfaces.

We investigated obedience to an authoritative robot and asked experiment participants to do a task they would rather not do. Obedience questions are increasingly important as robots participate in tasks where they give people directions. We conducted a series of HRI obedience experiments, comparing a robotic authority to other authority instances, including: (i) a human, (ii) a remote-controlled robot, and (iii) robots of variant embodiments. The results suggest that half of participants will continue to perform a tedious task under the direction of a robot, even after expressing desire to stop. Further, we failed to find an effect of robot embodiment and perceived autonomy on obedience. Instead, the robot's perceived authority status may be more strongly correlated to obedience.

This paper presents an original scenario design specifically created for exploring gender-related issues surrounding collaborative human-robot teams on assembly lines. Our methodology is grounded squarely in the need for increased gender work in human-robot interaction. As with most research in social human-robot interaction, investigating and exploring gender issues relies heavily on an evaluation methodology and scenario that aims to maximize ecological validity, so that the lab results can generalize to a real-world social scenario. In this paper, we present our discussion on study elements required for ecological validity in our context, present an original study design that meets these criteria, and present initial pilot results that reflect on our approach and study design.

In designing and evaluating human-robot interactions and interfaces, researchers often use a simulated robot due to the high cost of robots and time required to program them. However, it is important to consider how interaction with a simulated robot differs from a real robot; that is, do simulated robots provide authentic interaction? We contribute to a growing body of work that explores this question and maps out simulated-versus-real differences, by explicitly investigating empathy: how people empathize with a physical or simulated robot when something bad happens to it. Our results suggest that people may empathize more with a physical robot than a simulated one, a finding that has important implications on the generalizability and applicability of simulated HRI work. Empathy is particularly relevant to social HRI and is integral to, for example, companion and care robots. Our contribution additionally includes an original and reproducible HRI experimental design to induce empathy toward robots in laboratory settings, and an experimentally validated empathy-measuring instrument from psychology for use with HRI.

Remote control robots are being found in an increasing number of application domains, including search and rescue, exploration, and reconnaissance. There is a large body of HRI research that investigates interface design for remote navigation, control, and sensor monitoring, while aiming for interface enhancements that benefit the remote operator such as improving ease of use, reducing operator mental load, and maximizing awareness of a robot's state and remote environment. Even though many remote control robots have multi-degree-of-freedom robotic manipulator arms for interacting with the environment, there is only limited research into easy-to-use remote control interfaces for such manipulators, and many commercial robotic products are still using simplistic interface technologies such as keypads or gamepads with arbitrary mappings to arm morphology. In this paper, we present an original interface for the remote control of a multi-degree of freedom robotic arm. We conducted a controlled experiment to compare our interface to an existing commercial keypad interface and detail our results that indicate our interface was easier to use, required less cognitive task load, and enabled people to complete tasks more quickly. In this paper, we present an original interface for the remote control of a multi-degree of freedom robotic arm. We conducted a controlled experiment to compare our interface to an existing commercial keypad interface and detail our results that indicate our interface was easier to use, required less cognitive task load, and enabled people to complete tasks more quickly.