Learning from Demonstration (LfD) empowers robots to acquire new skills through human demonstrations, making it feasible for everyday users to teach robots. However, the success of learning and generalization heavily depends on the quality of these demonstrations. Consistency is often used to indicate quality in LfD, yet the factors that define this consistency remain underexplored. In this article, we evaluate a comprehensive set of motion data characteristics to determine which consistency measures best predict learning performance. By ensuring demonstration consistency prior to training, we enhance models’ predictive accuracy and generalization to novel scenarios. We validate our approach with two user studies involving participants with diverse levels of robotics expertise. In the first study ( N = 24), users taught a PR2 robot to perform a button-pressing task in a constrained environment, while in the second study ( N = 30), participants trained an UR5 robot on a pick-and-place task. Results show that demonstration consistency significantly impacts success rates in both learning and generalization, with 70% and 89% of task success rates in the two studies predicted using our consistency metrics. Moreover, our metrics estimate generalized performance success rates with 76% and 91% accuracy. These findings suggest that our proposed measures provide an intuitive, practical way to assess demonstration data quality before training, without requiring expert data or algorithm-specific modifications. Our approach offers a systematic way to evaluate demonstration quality, addressing a critical gap in LfD by formalizing consistency metrics that enhance the reliability of robot learning from human demonstrations.

Learning from Demonstration (LfD) is a framework that allows lay users to easily program robots. However, the efficiency of robot learning and the robot’s ability to generalize to task variations hinge upon the quality and quantity of the provided demonstrations. Our objective is to guide human teachers to provide more effective demonstrations, thus facilitating efficient robot learning. To achieve this, we propose to use a measure of uncertainty, namely task-related information entropy , as a criterion for suggesting informative demonstration examples to human teachers to improve their teaching skills. This approach seeks to minimize the requisite number of demonstrations by enhancing their distribution throughout the workspace. In a conducted experiment \((N = 24)\) , an augmented reality (AR)-based guidance system was employed to train novice users to produce additional demonstrations from areas with the highest entropy within the workspace. These novice users were trained for a few trials to teach the robot a generalizable task using a limited number of demonstrations. Subsequently, the users’ performance after training was assessed first on the same task (retention) and then on a new task (transfer) without guidance. The results indicate a substantial improvement in robot learning efficiency from the teacher’s demonstrations, with an improvement of up to 198% observed on the novel task. Furthermore, the proposed approach was compared to a state-of-the-art heuristic rule and found to improve robot learning efficiency by 210% compared to the heuristic rule. The scripts used in this article are available on GitHub .

We provide an experimental evaluation of a wearable augmented reality (AR) system we have developed for human-robot teams working on tasks requiring collaboration in shared physical workspace. Recent advances in AR technology have facilitated the development of more intuitive user interfaces for many human-robot interaction applications. While it has been anticipated that AR can provide a more intuitive interface to robot assistants helping human workers in various manufacturing scenarios, existing studies in robotics have been largely limited to teleoperation and programming. Industry 5.0 envisions cooperation between human and robot working in teams. Indeed, there exist many industrial tasks that can benefit from human-robot collaboration. A prime example is high-value composite manufacturing. Working with our industry partner towards this example application, we evaluated our AR interface design for shared physical workspace collaboration in human-robot teams. We conducted a multi-dimensional analysis of our interface using established metrics. Results from our user study (n = 26) show that, subjectively, the AR interface feels more novel and a standard joystick interface feels more dependable to users. However, the AR interface was found to reduce physical demand and task completion time, while increasing robot utilization. Furthermore, user’s freedom of choice to collaborate with the robot may also affect the perceived usability of the system.

Mobile robots are increasingly being deployed in public spaces such as shopping malls, airports, and urban sidewalks. Most of these robots are designed with human-aware motion planning capabilities but are not designed to communicate with pedestrians. Pedestrians encounter these robots without prior understanding of the robots’ behaviour, which can cause discomfort, confusion, and delayed social acceptance. In this research, we explore the common human-robot interaction at a doorway or bottleneck in a structured environment. We designed and evaluated communication cues used by a robot when yielding to a pedestrian in this scenario. We conducted an online user study with 102 participants using videos of a set of robot-to-human yielding cues. Results show that a Robot Retreating cue was the most socially acceptable cue. Repeated measures and Friedman’s ANOVAs on components of social acceptability were statistically significant (p = .01) and had small and medium effect sizes (ηp2= .04, ηp2= .08). The results of this work help guide the development of mobile robots for public spaces.

Pushrim-activated power-assisted wheels (PAPAWs) are assistive technologies that provide on-demand assistance to wheelchair users. PAPAWs operate based on a collaborative control scheme and require an accurate interpretation of the user’s intent to provide effective propulsion assistance. This paper investigates a user-specific intention estimation framework for wheelchair users. We used Gaussian Mixture models (GMM) to identify implicit intentions from user-pushrim interactions (i.e., input torque to the pushrims). Six clusters emerged that were associated with different phases of a stroke pattern and the intention about the desired direction of motion. GMM predictions were used as "ground truth" labels for further intention estimation analysis. Next, Random Forest (RF) classifiers were trained to predict user intentions. The best optimal classifier had an overall prediction accuracy of 94.7%. Finally, a Bayesian filtering (BF) algorithm was used to extract sequential dependencies of the user-pushrim measurements. The BF algorithm improved sequences of intention predictions for some wheelchair maneuvers compared to the GMM and RF predictions. The proposed intention estimation pipeline is computationally efficient and was successfully tested and used for real-time prediction of wheelchair user’s intentions. This framework provides the foundation for the development of user-specific and adaptive PAPAW controllers.

When the question of who should get access to a communal resource first is uncertain, people often negotiate via nonverbal communication to resolve the conflict. What should a robot be programmed to do when such conflicts arise in Human-Robot Interaction? The answer to this question varies depending on the context of the situation. Learning from how humans use hesitation gestures to negotiate a solution in such conflict situations, we present a human-inspired design of nonverbal hesitation gestures that can be used for Human-Robot Negotiation. We extracted characteristic features of such negotiative hesitations humans use, and subsequently designed a trajectory generator (Negotiative Hesitation Generator) that can re-create the features in robot responses to conflicts. Our human-subjects experiment demonstrates the efficacy of the designed robot behaviour against non-negotiative stopping behaviour of a robot. With positive results from our human-robot interaction experiment, we provide a validated trajectory generator with which one can explore the dynamics of human-robot nonverbal negotiation of resource conflicts.

This paper investigate potential use of augmented reality (AR) for physical human-robot collaboration in large-scale, labour-intensive manufacturing tasks. While it has been shown that use of AR can help increase task efficiency in teleoperative and robot programming tasks involving smaller-scale robots, its use for physical human-robot collaboration in shared workspaces and large-scale manufacturing tasks have not been well-studied. With the eventual goal of applying our AR system to collaborative aircraft body manufacturing, we compare in a user study the use of an AR interface we developed with a standard joystick for human robot collaboration in an experiment task simulating industrial carbon-fibre-reinforced-polymer manufacturing procedure. Results show that use of AR yields reduced task time and physical demand, with increased robot utilization.

Programming and executing robot trajectories is a routine manufacturing procedure. However, current interfaces (i.e., teach pendants) are bulky, unintuitive, and interrupts task flow. Recently, augmented reality (AR) has been used to create alternative solutions. However, input modalities of such systems tend to be limited. By introducing the use of electromyography (EMG), we have created a novel multimodal wearable interface for online trajectory programming and execution. Through the use of EMG, our system aims to bridge the user's force activation to the robot arm force profile. Our proposed system provides two interaction methods for trajectory execution and force control using 1) arm EMG and 2) arm orientation. We compared these methods with a standard joystick in a user study to test their usability. Results show that proposed methods have increased physical demands but yield equivalent task performance, demonstrating the potential of our proposed interface to provide a wearable alternative solution.

Pushrim-activated power-assisted wheels (PAPAWs) are assistive technologies that provide on-demand assistance to wheelchair users. PAPAWs operate based on a collaborative control scheme. Therefore, they rely on accurate interpretation of the user's intent to provide effective propulsion assistance. This paper presents a learning-based approach to predict wheelchair users' intention when performing a variety of wheelchair activities. We obtained kinematic and kinetic data from manual wheelchair users when performing standard wheelchair activities such as turns and ascents. Our measurements revealed variability in physical capabilities and propulsion habits of different users, therefore, highlighting the need for the development of personalized intention inference models. We used Gaussian Mixture models to label different phases of user-pushrim interactions based on individual user's wheeling behaviour. Supervised classifiers were trained with each user's data and these models were used to predict the user's intentions during different propulsion activities. We found random forest classifiers had high accuracy (>92%) in predicting different states of individual-specific wheelchair propulsion and user intent for 2 participants. This proposed framework is computationally efficient and can be used for real-time prediction of wheelchair users' intention. The outcome of this clustering-classification pipeline provides relevant information for designing user-specific and adaptive PAPAW controllers.

Robot Learning from Demonstration (LfD) allows robots to implement autonomous manipulation by observing the movements executed by a demonstrator. As such, LfD has been established as a key element for useful user interactions in everyday environments. Kinesthetic teaching, a teaching technique within LfD, entails physically guiding the robot to achieve a task. When demonstrating complex actions on a multi-DoF manipulator, novice users typically encounter difficulties with trajectory continuity and joint orientation, necessitating training by an expert. A comparison between different training approaches is conducted in a study of nine novice users. These approaches are kinesthetic, observational and discovery-learning. The kinesthetic method utilizes record and playback functions implemented on a 7-DoF Barrett Technology WAM robot. A novice user passively holds the arm while an expert's trajectory is replayed. A visual demonstration by the expert is used for the observational training group. The discovery-learning group does not receive an expert demonstration; they use trial-and-error to produce the trajectory on their own. Task-space performance is evaluated pre- and post-training for each user to determine the relative and absolute performance improvements of the groups across the three training approaches. Absolute performance improvements are compared to the performance of an expert and a minimum-jerk trajectory to gauge how skillful the participant becomes with respect to the expert. The kinesthetic approach shows superior indicators of performance in trajectory similarity to the minimum-jerk trajectory with 39% and 13% improvement over the observational and discovery methods, respectively. Observational training shows greater improvement in terms of the smoothness of the velocity profile with 32.7% compared to 29.5% and 21.9% for both discovery and kinesthetic training, respectively.

In this paper, we propose a novel navigation system for mobile robots in pedestrian-rich sidewalk environments. Sidewalks are unique in that the pedestrian-shared space has characteristics of both roads and indoor spaces. Like vehicles on roads, pedestrian movement often manifests as linear flows in opposing directions. On the other hand, pedestrians also form crowds and can exhibit much more random movements than vehicles. Classical algorithms are insufficient for safe navigation around pedestrians and remaining on the sidewalk space. Thus, our approach takes advantage of natural human motion to allow a robot to adapt to sidewalk navigation in a safe and socially-compliant manner. We developed a group surfing method which aims to imitate the optimal pedestrian group for bringing the robot closer to its goal. For pedestrian-sparse environments, we propose a sidewalk edge detection and following method. Underlying these two navigation methods, the collision avoidance scheme is human-aware. The integrated navigation stack is evaluated and demonstrated in simulation. A hardware demonstration is also presented.

This paper presents a future-focused approach for robot programming based on augmented trajectories. Using a mixed reality head-mounted display (Microsoft Hololens) and a 7-DOF robot arm, we designed an augmented reality (AR) robotic interface with four interactive functions to ease the robot programming task: 1) Trajectory specification. 2) Virtual previews of robot motion. 3) Visualization of robot parameters. 4) Online reprogramming during simulation and execution. We validate our AR-robot teaching interface by comparing it with a kinesthetic teaching interface in two different scenarios as part of a pilot study: creation of contact surface path and free space path. Furthermore, we present an industrial case study that illustrates our AR manufacturing paradigm by interacting with a 7-DOF robot arm to reduce wrinkles during the pleating step of the carbon-fiber-reinforcement-polymer vacuum bagging process in a simulated scenario.

With advances in sensor and actuator design, intelligent computing techniques and personal care robotics, today's robots hold promise as fully interactive, therapeutic human companions. To achieve this ambitious goal, key interaction components must be identified and then systematically designed and evaluated. Based on successes of human-animal therapy, we propose affective touch as one such component. Delivering this adjunct in a controllable robot form allows us to examine its efficacy for therapeutic applications such as anxiety management. With an approach grounded in social cognitive theories for human-animal relations, we deployed a social robot, the Haptic Creature, in an interaction designed to be calming: participants held the robot on their laps and stroked it as it was breathing. As a result, their heart and respiration rates significantly decreased relative to stroking a non-breathing robot. They also reported themselves as calmer and happier.

In collaborative tasks, people often communicate using nonverbal gestures to coordinate actions. When two people reach for the same object at the same time, they often respond to an imminent potential collision with jerky halting hand motions that we term hesitation gestures. Successful implementation of such communicative conflict response behaviour onto robots can be useful. In a myriad of human-robot interaction contexts involving shared spaces and objects, this behaviour can provide a fast and effective means for robots to express awareness of conflict and cede right-of-way during collaborative work with users. Our previous work suggests that when a six-degree-of-freedom (6-DOF) robot traces a simplified trajectory of recorded human hesitation gestures, these robot motions are also perceived by humans as hesitation gestures. In this work, we present a characteristic motion profile derived from the recorded human hesitation motions, called the Acceleration-based Hesitation Profile (AHP). We test its efficacy to generate communicative hesitation responses by a robot in a fast-paced human-robot interaction experiment.

In this study, we investigate and characterize haptic interaction in human-to-human handovers and identify key features that facilitate safe and efficient object transfer. Eighteen participants worked in pairs and transferred weighted objects to each other while we measured their grip forces and load forces. Our data show that during object transfer, both the giver and receiver employ a similar strategy for controlling their grip forces in response to changes in load forces. In addition, an implicit social contract appears to exist in which the giver is responsible for ensuring object safety in the handover and the receiver is responsible for maintaining the efficiency of the handover. Compared with prior studies, our analysis of experimental data show that there are important differences between the strategies used by humans for both picking up/placing objects on table and that used for handing over objects, indicating the need for specific robot handover strategies as well. The results of this study will be used to develop a controller for enabling robots to perform object handovers with humans safely, efficiently, and intuitively.

Nonverbal communication cues play an important role in human-human interaction and are expected to take a similar role in human-robot collaboration. In current industrial practice, human-robot turn-taking is explicitly human controlled, via a command channel such as switch or button. However, such a master-slave approach does not permit collaborative interaction, and requires the human to focus on both controlling the robot's behavior and on the task, thereby affecting overall performance. In this paper, implicit, nonverbal communication cues are examined as a non-explicit communication channel during a turn-taking task context. The aim of this study is to characterize the types and frequencies of nonverbal cues important to regulating turn taking during an assembly-task-type collaboration. This analysis will guide the selection of cues that can be expressed by the robot as implicit user inputs while human and robot complete a shared task.

Unwanted conflicts are inevitable between collaborating agents that share spaces and resources. Motivated by the use of nonverbal communications as a conflict resolution mechanism by humans, this study investigates the communicative capabilities reflected in the trajectory characteristics of hesitation gestures during human-robot collaboration. Hesitation gestures and non-hesitation human arm motions were recorded from a series of reach-and-retract tasks and embodied on a 6-DOF robot arm. A total of 86 survey respondents watched and scored recordings of these motions according to whether they recognized hesitation gestures as exhibited by both the human and the robot. Using the survey's statistical evidence indicating that hesitation trajectories embodied in an articulated robot arm can be recognized by human observers, we identified trajectory characteristics of hesitation gestures. The contribution of our work is an empirically grounded robot trajectory specification that provides communicative cues for conflict resolution during collaborative reaching scenarios.

We present the system design for a novel robotic balance simulator that enables the investigation of the balance mechanisms involved in natural human standing. Our system allows for complete control of task dynamics to mimic normal standing while avoiding the pitfalls associated with applying external perturbations. The system enables subjects to balance themselves according to a programmable physical model of an inverted pendulum. Subjects were able to balance the system, and results show that the load stiffness curves approximate those of normal human standing to within 20.1 ± 9.7% (S.D.). Differences were within the range expected from control loop delay, reduced ankle motion, and approximations inherent to the inverted pendulum model.

status: Published

In order to develop a friendly and safe interaction between humans and robots, it is essential for the robot to evaluate user's affective states and respond accordingly. However, affective states are typically assessed using offline questionnaires and user reports. In this paper we investigate the use of an online-device for collecting real-time user reports of affective state during interaction with a robot. These reports are compared to both previous survey reports taken after the interaction, and the affective states estimated by an inference system. The aim is to evaluate and characterize the physiological signal-based inference system and determine which factors significantly influence its performance. This analysis will be used in future work, to fine tune the affective estimations by identifying what kind of variations in physiological signals precede or accompany the variations in reported affective states.

The most critical challenge for Personal Robotics is to manage the issue of human safety and yet provide the physical capability to perform useful work. This paper describes a novel concept for a mobile, 2-armed, 25-degree-of- freedom system with backdrivable joints, low mechanical impedance, and a 5 kg payload per arm. System identification, design safety calculations and performance evaluation studies of the first prototype are included, as well as plans for a future development.

In this paper, we propose a non-invasive and nonrestrictive human vital-signs sensory system using an electrically-conductive fiber. To maintain health, it is important to monitor our physical condition daily. Even during sleep, a person's condition is not constant and sometimes it can change suddenly. Therefore, we are developing a human vital-signs sensory system for monitoring purposes. Our proposed system consists of a sensing part, data collection part and signal processing part. The sensing part is composed of electrically-conductive fibers incorporated in a conventional bed sheet. Since the sensors are arranged in a grid structure, we can detect the pressure distribution on the sheet. We utilize this sheet to acquire human vital signs such as respiration and heart rate continuously. With a preliminary prototype, we show the basic properties of the proposed sensory system.

This paper presents a novel computer-controlled bicycle ergometer, the TiltCycle, for use in human biomechanics studies of pedaling. The TiltCycle has a tilting (reclining) seat and backboard, a split crank to isolate the left and right loads to the feet of the cyclist, and two belt-driven, computer-controller motors to provide both assistance and resistance loads. Sensors measure the kinematics and force production of the pedaling work performed, as well as goniometer and electromyography signals from the lower limbs. The technical description includes the mechanical design, low-level software and control algorithms designed for studies in human lower-limb biomechanics and bilateral coordination, and concludes with validation testing and system identification results.

The need for effective and low-cost ways of facilitating upper limb therapy for individuals recovering from stroke has led to the Driver's Simulation Environment for Arm Therapy (Driver's SEAT). This assistive device is an upper limb one-degree-of-freedom robotic therapy device that incorporates a modified PC-based driving simulator. This paper describes the novel use of a driving simulator to implement an upper limb bimanual exercise task and presents control algorithms that use torque feedback as a method of encouraging the preferential use of one limb over another. The key feature is a split steering wheel, which allows us to measure the tangential forces applied to the wheel by the subjects' impaired and unimpaired upper limbs independently. We present experimental results to show Driver's SEAT's ability to measure bilateral forces and quantify tracking performance.

This paper describes the implementation of a robot control architecture designed to combine a manipulation task design environment with a motion controller that uses the operational space formulation to define and implement arm trajectories and object manipulation. The ProVAR desktop manipulation system is an assistive robot for individuals with a severe physical disability, such as quadriplegia as a result of a high-level spinal cord injury. ProVAR allows non-technical operators access to the robot's capabilities through a direct-manipulation simulation/preview user interface. The novel interface concept is based on two built-in characters to play the roles of helpful consultant and down-to-earth robot arm. This team-based interface concept was chosen to maximize user performance and comfort in controlling the inherently complex mechatronic technology. This paper describes our design decisions and rationale.

The authors develop an interface system for service robots. In order to assure adaptability to lightly structured environments and easy programming of new tasks, we propose an automatic adaptive motion generation methodology and introduce it in the context of an interface system for an assistive rehabilitation robot. The implementation of the robot interface system using VRML and Java shows the effectiveness of the proposed method.