This paper presents a novel learning-based approach to dynamic robot-to-human handover, addressing the challenges of delivering objects to a moving receiver. We hypothesize that dynamic handover, where the robot adjusts to the receiver's movements, results in more efficient and comfortable interaction compared to static handover, where the receiver is assumed to be stationary. To validate this, we developed a nonparametric method for generating continuous handover motion, conditioned on the receiver's movements, and trained the model using a dataset of 1,000 human-to-human handover demonstrations. We integrated preference learning for improved handover effectiveness and applied impedance control to ensure user safety and adaptiveness. The approach was evaluated in both simulation and real-world settings, with user studies demonstrating that dynamic handover significantly reduces handover time and improves user comfort compared to static methods. Videos and demonstrations of our approach are available at https://zerotohero7886.github.io/dyn-r2h-handover/.

Markerless hand-tracking has become increasingly common on commercially available virtual and mixed reality headsets to improve the naturalness of interaction and immersivity of virtual environments. However, there has been limited examination of the performance of markerless hand-tracking on commercial head-mounted displays. Here, we propose an evaluation methodology that leverages a robotic manipulator to measure the positional accuracy, jitter, and latency of such systems and provides a standardized characterization framework of markerless hand-tracking. We apply this methodology to evaluate the hand-tracking performance of two recent mixed reality devices from Meta: the Quest Pro and Quest 3. Results demonstrate the influence of proximity to the headset, rotation of hand, and joint selected as the tracking feature on hand-tracking performance. We found that hand-tracking error and jitter were lowest for both headsets in conditions where the knuckle was the tracking point compared to the fingertip. Regarding positional accuracy, in best-performing conditions, the Quest Pro outperformed the Quest 3 with 1.22 cm of average error compared to 1.73 cm. The opposite result was true concerning jitter, with results of 1.77 cm and 1.11 cm for the Quest Pro and Quest 3, respectively. We found latency highly variable for the Quest Pro (15.8 - 229.2 ms) and Quest 3 (14.4 - 220.5 ms). This work provides a testing framework for highly systematic and repeatable performance measurements of markerless hand-tracking systems embedded in headsets.

In this paper, we propose a SOCratic model for Robots Approaching humans based on TExt System (SOCRATES) focusing on the human search and approach based on free-form textual description; the robot first searches for the target user, then the robot proceeds to approach in a human-friendly manner. We present a Socratic human search model that connects large pre-trained foundation models to solve the downstream task of searching for the target person based on textual descriptions. In particular, textual descriptions used for searching are composed of appearance (e.g., wearing white shirt with black hair) and location clues (e.g., a student that works with robots). Additionally, we propose a hybrid learning-based framework for generating human-friendly robotic motion to approach a person, consisting of a learning-from-demonstration module and a knowledge distillation module utilizing LLMs. We evaluate the search performance of the proposed method in both simulation and real-world environments using the Boston Dynamics Spot robot. Moreover, we evaluate the effectiveness of our proposed framework through analysis involving human participants and investigate the perceived warmth of our system.

In this paper, we focus on the problem of efficiently locating a target object described with free-form text using a mobile robot equipped with vision sensors (e.g., an RGBD camera). Conventional active visual search predefines a set of objects to search for, rendering these techniques restrictive in practice. To provide added flexibility in active visual searching, we propose a system where a user can enter target commands using free-form text; we call this system Zero-shot Active Visual Search (ZAVIS). ZAVIS detects and plans to search for a target object inputted by a user through a semantic grid map represented by static landmarks (e.g., desk or bed). For efficient planning of object search patterns, ZAVIS considers commonsense knowledge-based co-occurrence and predictive uncertainty while deciding which landmarks to visit first. We validate the proposed method with respect to SR (success rate) and SPL (success weighted by path length) in both simulated and real-world environments. The proposed method outperforms previous methods in terms of SPL in simulated scenarios, and we further demonstrate ZAVIS with a Pioneer-3AT robot in real-world studies.

This paper describes the development of a system for lifelike gaze in human-robot interactions using a humanoid Audio-Animatronics®bust. Previous work examining mutual gaze between robots and humans has focused on technical implementation. We present a general architecture that seeks not only to create gaze interactions from a technological standpoint, but also through the lens of character animation where the fidelity and believability of motion is paramount; that is, we seek to create an interaction which demonstrates the illusion of life. A complete system is described that perceives persons in the environment, identifies persons-of-interest based on salient actions, selects an appropriate gaze behavior, and executes high fidelity motions to respond to the stimuli. We use mechanisms that mimic motor and attention behaviors analogous to those observed in biological systems including attention habituation, saccades, and differences in motion bandwidth for actuators. Additionally, a subsumption architecture allows layering of simple motor movements to create increasingly complex behaviors which are able to interactively and realistically react to salient stimuli in the environment through subsuming lower levels of behavior. The result of this system is an interactive human-robot experience capable of human-like gaze behaviors.

We present a system for fast and robust handovers with a robot character, together with a user study investigating the effect of robot speed and reaction time on perceived interaction quality. The system can match and exceed human speeds and confirms that users prefer human-level timing. The system has the appearance of a robot character, with a bear-like head and a soft anthropomorphic hand and uses Bézier curves to achieve smooth minimum-jerk motions. Fast timing is enabled by low latency motion capture and real-time trajectory generation: the robot initially moves towards an expected handover location and the trajectory is updated on-the-fly to converge smoothly to the actual handover location. A hybrid automaton provides robustness to failure and unexpected human actions. In a 3x3 user study, we vary the speed of the robot and add variable sensorimotor delays. We evaluate the social perception of the robot using the Robot Social Attribute Scale (RoSAS). Inclusion of a small delay, mimicking the delay of the human sensorimotor system, leads to an improvement in perceived qualities over both no delay and long delay conditions. Specifically, with no delay the robot is perceived as more discomforting, and with a long delay it is perceived as less warm.

This work explores social perceptions of robots within the domain of human-to-robot handovers. Using the Robotic Social Attributes Scale (RoSAS), we explore how users socially judge robot receivers as three factors are varied: initial position of the robot arm prior to handover, grasp method employed by the robot when receiving a handover object trading off perceived object safety for time efficiency, and retraction speed of the arm following handover. Our results show that over multiple handover interactions with the robot, users gradually perceive the robot receiver as being less discomforting and having more emotional warmth. Additionally, we have found that by varying grasp method and retraction speed, users may hold significantly different judgments of robot competence and discomfort. With these results, we recognize empirically that users are able to develop social perceptions of robots which can change through modification of robot receiving behaviour and through repeated interaction with the robot. More widely, this work suggests that measurement of user social perceptions should play a larger role in the design and evaluation of human-robot interactions and that the RoSAS can serve as a standardized tool in this regard.

This paper presents a future-focused approach for robot programming based on augmented trajectories. Using a mixed reality head-mounted display (Microsoft Hololens) and a 7-DOF robot arm, we designed an augmented reality (AR) robotic interface with four interactive functions to ease the robot programming task: 1) Trajectory specification. 2) Virtual previews of robot motion. 3) Visualization of robot parameters. 4) Online reprogramming during simulation and execution. We validate our AR-robot teaching interface by comparing it with a kinesthetic teaching interface in two different scenarios as part of a pilot study: creation of contact surface path and free space path. Furthermore, we present an industrial case study that illustrates our AR manufacturing paradigm by interacting with a 7-DOF robot arm to reduce wrinkles during the pleating step of the carbon-fiber-reinforcement-polymer vacuum bagging process in a simulated scenario.

We present a system enabling users to accurately catch a real ball while immersed in a virtual reality environment. We examine three visualizations: rendering a matching virtual ball, the predicted trajectory of the ball, and a target catching point lying on the predicted trajectory. In our demonstration system, we track the projectile motion of a ball as it is being tossed between users. Using Unscented Kalman Filtering, we generate predictive estimates of the ball's motion as it approaches the catcher. The predictive assistance visualizations effectively increases the user's senses but can also alter the user's strategy in catching.

To enable robots to learn handover orientations from observing natural handovers, we conduct a user study to measure and compare natural handover orientations with giver-centered and receiver-centered handover orientations for twenty common objects. We use a distance minimization approach to compute mean handover orientations. We posit that, computed means of receiver-centered orientations could be used by robot givers to achieve more efficient and socially acceptable handovers. Furthermore, we introduce the notion of affordance axes for comparing handover orientations, and offer a definition for computing them. Observable patterns were found in receiver-centered handover orientations. Comparisons show that depending on the object, natural handover orientations may not be receiver-centered; thus, robots may need to distinguish between good and bad handover orientations when learning from natural handovers.

In this paper we provide empirical evidence that using humanlike gaze cues during human-robot handovers can improve the timing and perceived quality of the handover event. Handovers serve as the foundation of many human-robot tasks. Fluent, legible handover interactions require appropriate nonverbal cues to signal handover intent, location and timing. Inspired by observations of human-human handovers, we implemented gaze behaviors on a PR2 humanoid robot. The robot handed over water bottles to a total of 102 naïve subjects while varying its gaze behaviour: no gaze, gaze designed to elicit shared attention at the handover location, and the shared attention gaze complemented with a turn-taking cue. We compared subject perception of and reaction time to the robot-initiated handovers across the three gaze conditions. Results indicate that subjects reach for the offered object significantly earlier when a robot provides a shared attention gaze cue during a handover. We also observed a statistical trend of subjects preferring handovers with turn-taking gaze cues over the other conditions. Our work demonstrates that gaze can play a key role in improving user experience of human-robot handovers, and help make handovers fast and fluent.

This work explores a novel interaction paradigm driven by implicit, low-attention user control, accomplished by monitoring a user's physiological state. We have designed and prototyped this interaction for a first use case of bookmarking an audio stream, to holistically explore the implicit interaction concept. Here, a user's galvanic skin conductance (GSR) is monitored for orienting responses (ORs) to external interruptions; our prototype automatically bookmarks the media such that the user can attend to the interruption, then resume listening from the point he/she is interrupted. To test this approach's viability, we addressed questions such as: does GSR exhibit a detectable response to interruptions, and how should the interaction utilize this information? In evaluating this system in a controlled environment, we found an OR detection accuracy of 84%; users provided subjective feedback on its accuracy and utility.