Assistive robots offer agency to humans with severe motor impairments. Often, these users control high-DoF robots through low-dimensional interfaces—such as using a 1-D sip/puff interface to operate a 6-DoF robotic arm. This mismatch results in having access to only a subset of control dimensions at a given time, imposing unintended and artificial constraints on robot motion. As a result, interface-limited demonstrations embed suboptimal motions that reflect interface restrictions rather than user intent. To address this, we present a trajectory reconstruction algorithm that reasons about task, environment, and interface constraints to lift demonstrations into the robot’s full control space. We evaluate our approach using real-world demonstrations of ADL-inspired tasks performed via a 2-D joystick and 1-D sip/puff control interface, teleoperating two distinct 7-DoF robotic arms. Analyses of the reconstructed demonstrations and derived control policies show that lifted trajectories are faster and more efficient than their interface-constrained counterparts while respecting user preferences.

When individuals are paralyzed from injury or damage to the brain, upper body movement and function can be compromised. While the use of body motions to interface with machines has shown to be an effective noninvasive strategy to provide movement assistance and to promote physical rehabilitation, learning to use such interfaces to control complex machines is not well understood. In a five session study, we demonstrate that a subset of an uninjured population is able to learn and improve their ability to use a high-dimensional Body-Machine Interface (BoMI), to control a robotic arm. We use a sensor net of four inertial measurement units, placed bilaterally on the upper body, and a BoMI with the capacity to directly control a robot in six dimensions. We consider whether the way in which the robot control space is mapped from human inputs has any impact on learning. Our results suggest that the space of robot control does play a role in the evolution of human learning: specifically, though robot control in joint space appears to be more intuitive initially, control in task space is found to have a greater capacity for longer-term improvement and learning. Our results further suggest that there is an inverse relationship between control dimension couplings and task performance.

We present an assistance system that reasons about a human’s intended actions during robot teleoperation in order to provide appropriate modifications on unintended behavior. Existing methods typically treat the human and control interface as a black box and assume the measured user input is noise-free, and use this signal to infer task-level human intent. We recognize that the signal measured through the interface is masked by the physical limitations of the user and the interface they are required to use. With this key insight, we model the human’s physical interaction with a control interface during robot teleoperation, and distinguish between interface-level intended and measured physical actions explicitly. By reasoning over the unobserved intentions using model-based inference techniques, our assistive system provides customized modifications on a user’s issued commands. We validate our algorithm both in simulation and with a 10-person human subject study in which we evaluate the performance of the proposed assistance paradigms. Our results show that the assistance paradigms helped to significantly reduce task completion time, number of mode switches, cognitive workload, and user frustration, and improve overall user satisfaction.

A dynamic autonomy allocation framework automatically shifts how much control lies with the human versus the robotics autonomy, for example based on factors such as environmental safety or user preference. To investigate the question of which factors should drive dynamic autonomy allocation, we perform a human subject study to collect ground truth data that shifts between levels of autonomy during shared-control robot operation. Information streams from the human, the interaction between the human and the robot, and the environment are analyzed. Machine learning methods—both classical and deep learning—are trained on this data. An analysis of information streams from the human-robot team suggests features which capture the interaction between the human and the robotics autonomy are the most informative in predicting when to shift autonomy levels. Even the addition of data from the environment does little to improve upon this predictive power. The features learned by deep networks, in comparison to the hand-engineered features, prove variable in their ability to represent shift-relevant information. This work demonstrates the classification power of human-only and human-robot interaction information streams for use in the design of shared-control frameworks, and provides insights into the comparative utility of various data streams and methods to extract shift-relevant information from those data.

Effective human-robot collaboration in shared autonomy requires reasoning about the intentions of the human partner. To provide meaningful assistance, the autonomy has to first correctly predict, or infer, the intended goal of the human collaborator. In this work, we present a mathematical formulation for intent inference during assistive teleoperation under shared autonomy. Our recursive Bayesian filtering approach models and fuses multiple non-verbal observations to probabilistically reason about the intended goal of the user without explicit communication. In addition to contextual observations, we model and incorporate the human agent's behavior as goal-directed actions with adjustable rationality to inform intent recognition. Furthermore, we introduce a user-customized optimization of this adjustable rationality to achieve user personalization. We validate our approach with a human subjects study that evaluates intent inference performance under a variety of goal scenarios and tasks. Importantly, the studies are performed using multiple control interfaces that are typically available to users in the assistive domain, which differ in the continuity and dimensionality of the issued control signals. The implications of the control interface limitations on intent inference are analyzed. The study results show that our approach in many scenarios outperforms existing solutions for intent inference in assistive teleoperation, and otherwise performs comparably. Our findings demonstrate the benefit of probabilistic modeling and the incorporation of human agent behavior as goal-directed actions where the adjustable rationality model is user customized. Results further show that the underlying intent inference approach directly affects shared autonomy performance, as do control interface limitations.

Humans and robots team together to perform tasks in various domains. Some tasks are easier to perform than others, but little work focuses on discovering the underlying mechanisms that affect perceived difficulty and task performance. To fill this gap, we propose a formalized approach to task characterization for human-robot teams using Taguchi design of experiments and conjoint analysis. With this, we conduct a 20 person study where participants operate a 6 degree of freedom robotic arm to perform manipulations defined by 6 kinematic features. We find that rotational features of a task contribute significantly more to decreased performance and increased difficulty than translational features. The participants also perform the activities with autonomy assistance. The data shows a reduction in the effect of these features on performance and difficulty when assistance is active. Furthermore, we examine when to trigger assistance based on thresholds set from outlier detection. The analysis indicates that rotational features and features leading to kinematic singularities are useful for triggering assistance.

Shared control for human-robot teams - where both the human and the robot's autonomy provide commands to the hardware - offers advantages over fully teleoperated or fully autonomous systems by utilizing the unique skill sets of both the human and robot's autonomy simultaneously. However, the mechanism by which control is shared is often static and many teams could benefit from adjusting this mechanism, such that the human or autonomy alternatively receive more control authority in different scenarios. The question then is: how do we know when these scenarios occur? In this paper, we present a method to estimate the performance of human-robot teams using a novel metric called Discrete N-Dimensional Entropy of Behavior (DNDEB). DNDEB utilizes knowledge of a high-performing human-robot team to build a model of how the team should operate. The model is used to predict the human's command. The error between the prediction and actual command is tracked and after a certain number of samples, entropy is estimated. A higher level of entropy corresponds to deviations from the high-performance model, which can be interpreted as poor performance by the human-robot team (e.g., long task time or a collision). Our formulation offers several advantages: it (1) accepts discrete inputs of any size, (2) does not require additional sensors, and (3) is tunable to the specific application. To validate this, we conduct a 15person study where subjects operated a powered wheelchair under three different shared-control paradigms. We find that entropy is higher for cases with longer task durations and cases where there is a collision. Moreover, we use DNDEB thresholds as a mechanism to predict the performance of the human-robot team online and find an average accuracy of 91% with a prescience rate of 72%.

Effective human-robot collaboration in shared control requires reasoning about the intentions of the human user. In this work, we present a mathematical formulation for human intent recognition during assistive teleoperation under shared autonomy. Our recursive Bayesian filtering approach models and fuses multiple non-verbal observations to probabilistically reason about the intended goal of the user. In addition to contextual observations, we model and incorporate the human agent's behavior as goal-directed actions with adjustable rationality to inform the underlying intent. We examine human inference on robot motion and furthermore validate our approach with a human subjects study that evaluates autonomy intent inference performance under a variety of goal scenarios and tasks, by novice subjects. Results show that our approach outperforms existing solutions and demonstrates that the probabilistic fusion of multiple observations improves intent inference and performance for shared-control operation.

In this paper, we present a novel grasp detection algorithm targeted towards assistive robotic manipulation systems. We consider the problem of detecting robotic grasps using only the raw point cloud depth data of a scene containing unknown objects, and apply a geometric approach that categorizes objects into geometric shape primitives based on an analysis of local surface properties. Grasps are detected without a priori models, and the approach can generalize to any number of novel objects that fall within the shape primitive categories. Our approach generates multiple candidate object grasps, which moreover are semantically meaningful and similar to what a human would generate when teleoperating the robot-and thus should be suitable manipulation goals for assistive robotic systems. An evaluation of our algorithm on 30 household objects includes a pilot user study, confirms the robustness of the detected grasps and was conducted in real-world experiments using an assistive robotic arm.

This work presents an algorithm for automated real-time ramp detection using 3D point cloud data in the context of shared-control powered wheelchairs. Limitations in the interfaces available to those with severe motor impairments can make basic maneuvering tasks with powered wheelchairs difficult. Although a significant amount of work has been done on obstacle detection and avoidance, much less attention has been given to algorithms for the safe and reliable detection of ramps and inclines; even though navigating these structures is an important part of urban life. We provide an algorithmic solution for accurately detecting traversable inclines for applications with powered wheelchairs using the Point Cloud Library (PCL) within the Robotics Operating System (ROS) framework. All algorithms are implemented first in simulation and later evaluated on data obtained from indoor and outdoor urban environments. We measure the performance of our algorithm with systematic testing on several different ramp structures, observed from varied viewpoints. Results show that our algorithm is successful in detecting the orientation, slope, and width of traversable ramps with up to 100% accuracy and an average detection accuracy of 88%.

Intelligent behavior in robots is implemented through algorithms. Historically, much of algorithmic robotics research strives to compute outputs that achieve mathematically rigid conditions, such as minimizing path length. But today's robots are increasingly being used to empower the daily lives of people, and experience shows that traditional algorithmic approaches are poorly suited for the unpredictable, idiosyncratic, and adaptive nature of human-robot interaction. This raises a need for entirely new computational, mathematical, and technical approaches for robots to better understand and react to humans. The human-friendly robots of the future will need new algorithms, informed from the ground up by HRI research, to generate interpretable, ethical, socially-acceptable behavior, ensure safety around humans, and execute tasks of value to society.

For many users of myoelectric prostheses there is a set of functionality which remains out of reach with current technology. In this work, we provide a first assessment of an extension to classical myoelectric prostheses control approaches that introduces simple automation that is shapable, using EMG signals. The idea is not to replace classical techniques, but to introduce automation for tasks, like those which require the coordination of multiple degrees of freedom, for which automation is well-suited. A prototype system is developed in simulation and an exploratory user study is performed to provide a first assessment, that evaluates our proposed approach and provides guidance for future development. A comparison is made between different formulations for the shaping controls, as well as to a classical control paradigm. Results from the user study are promising: showing significant performance improvements when using the automated controllers, and also unanimous preference for the use of automated controllers on this task. Additionally, some questions about the optimal user interaction with the system are revealed. All of these results support the case for continued development of the proposed approach, including more extensive user studies.

There are basic manuvering tasks with a powered wheelchair, like docking under a table and passage through a doorway or narrow hallway, which can be difficult for users with severe motor impairments - not only because of limitations in their own motor control, but also because of limitations in the control interfaces available to them. Robot automation can help transfer some of this control burden from the user to the machine. This work presents an algorithm for the automated detection of safe docking locations at rectangular and circular docking structures (tables, desks) with proper alignment information using 3D point cloud data. The safe docking locations can then be provided as goals to an autonomous path planner, within the context of providing adaptive driving assistance for powered wheelchair users. We evaluate the performance of our algorithm with systematic testing on several docking structures, observed from varied viewpoints.

This work presents an algorithm for rapid, automated detection of open doorways using 3D point cloud data. The algorithm has been developed in the context of shared-control powered wheelchairs in which adaptive assistance is provided to individuals who otherwise might not possess the fine motor control necessary to handle potentially challenging activities, such as doorway traversal. In this context it is important to go beyond the 2D laser scanner for open doorway detection, for both safety reasons as well as opportunities for improved shared-control behavior development. We evaluate the doorway detection by systematically testing the performance on several doors and door configurations from varied view points, using point clouds generated by a Microsoft Kinect.

Behavior adaptation with execution experience is a practical feature for any policy learning system. Our work provides performance feedback to a robot learner in the form of tactile corrections from a human teacher, for the purpose of policy refinement as well as policy reuse. Multiple variants of our general approach have been validated on the iCub robot, as building blocks towards a high-DoF humanoid system that integrates tactile sensing on the hands and arms into complex behaviors and sophisticated learning routines.

Nowadays, programming by demonstration (PbD) has become an important paradigm for policy learning in roboticsm [3]. The idea of having robots capable of learning from humans through natural communication means is indeed fascinating. As an extension of the traditional PbD learning scheme, where robots only learn by observing a human teacher, our work follows the recently suggested principle of policy refinement and reuse through interactive corrective feedback [1].However, to be responsive to such feedback, robots must be capable of sensing the world, especially human contact. Our work focuses on the sense of touch. Its integration in robotic applications has many advantages such as: a) safer and more natural interactions with objects and humans, b) improvement and simplification of the control mechanisms for human-robot interaction and object manipulation [2].Our video reports on two experimental studies conducted with the iCub, a 53 degree of freedom humanoid robot endowed with tactile sensing on its forearms and fingertips. a) In a hand-positioning task, the robot is shown how to bring its hand to the location where an object should be grasped. A wrong placement or a wrong approach to the target is corrected by the teacher though a tactile interface [1]. b) In a reactive grasping task, the robot is taught how to use its fingertip sensors to adapt and maintain its grasp in the face of external perturbations on the grasped object.The results of both our experiments show how tactile sensing can be utilized effectively to learn robust control policies through human coaching, by enabling a) online policy refinement and reuse, and b) rapid adaptation to external perturbations.

Traditional approaches to programming robots are generally inaccessible to non-robotics-experts. A promising exception is the learning from demonstration paradigm. Here a policy mapping world observations to action selection is learned, by generalizing from task demonstrations by a teacher. Most learning from demonstration work to date considers data from a single teacher. In this paper, we consider the incorporation of demonstrations from multiple teachers. In particular, we contribute an algorithm that handles multiple data sources, and additionally reasons about reliability differences between them. For example, multiple teachers could be inequally proficient at performing the demonstrated task. We introduce Demonstration Weight Learning (DWL) as a learning from demonstration algorithm that explicitly represents multiple data sources and learns to select between them, based on their observed reliability and according to an adaptive expert learning inspired approach. We present a first implementation of DWL within a simulated robot domain. Data sources are shown to differ in reliability, and weighting is found impact task execution success. Furthermore, DWL is shown to produce appropriate data source weights that improve policy performance.

As robots become more commonplace within society, the need for tools to enable non-robotics-experts to develop control algorithms, or policies, will increase. Learning from demonstration (LfD) offers one promising approach, where the robot learns a policy from teacher task executions. Our interests lie with robot motion control policies which map world observations to continuous low-level actions. In this work, we introduce advice-operator policy improvement (A-OPI) as a novel approach for improving policies within LfD. Two distinguishing characteristics of the A-OPI algorithm are data source and continuous state-action space. Within LfD, more example data can improve a policy. In A-OPI, new data is synthesized from a student execution and teacher advice. By contrast, typical demonstration approaches provide the learner with exclusively teacher executions. A-OPI is effective within continuous state-action spaces because high level human advice is translated into continuous-valued corrections on the student execution. This work presents a first implementation of the A-OPI algorithm, validated on a Segway RMP robot performing a spatial positioning task. A-OPI is found to improve task performance, both in success and accuracy. Furthermore, performance is shown to be similar or superior to the typical exclusively teacher demonstrations approach.

Learning by demonstration can be a powerful and natural tool for developing robot control policies. That is, instead of tedious hand-coding, a robot may learn a control policy by interacting with a teacher. In this work we present an algorithm for learning by demonstration in which the teacher operates in two phases. The teacher first demonstrates the task to the learner. The teacher next critiques learner performance of the task. This critique is used by the learner to update its control policy. In our implementation we utilize a 1-Nearest Neighbor technique which incorporates both training dataset and teacher critique. Since the teacher critiques performance only, they do not need to guess at an effective critique for the underlying algorithm. We argue that this method is particularly well-suited to human teachers, who are generally better at assigning credit to performances than to algorithms. We have applied this algorithm to the simulated task of a robot intercepting a ball. Our results demonstrate improved performance with teacher critiquing, where performance is measured by both execution success and efficiency.

Hierarchical state machines have proven to be a powerful tool for controlling autonomous robots due to their flexibility and modularity. For most real robot implementations, however, it is often the case that the control hierarchy is hand-coded. As a result, the development process is often time intensive and error prone. In this paper, we explore the use of an experts learning approach, based on Auer and colleagues' Exp3 (1995), to help overcome some of these limitations. In particular, we develop a modified learning algorithm, which we call rExp3, that exploits the structure provided by a control hierarchy by treating each state machine as an 'expert'. Our experiments validate the performance of rExp3 on a real robot performing a task, and demonstrate that rExp3 is able to quickly learn to select the best state machine expert to execute. Through our investigations in these environments, we identify a need for faster learning recovery when the relative performances of experts reorder, such as in response to a discrete environment change. We introduce a modified learning rule to improve the recovery rate in these situations and demonstrate through simulation experiments that rExp3 performs as well or better than Exp3 under such conditions.

In this paper, we focus on human-robot interaction in a team task where we identify the need for peer-to-peer (P2P) teamwork, with no fixed hierarchy for decision making between robots and humans. Instead, all team members are equal participants and decision making is truly distributed. We have fully developed a P2P team within Segway Soccer, a research domain, built upon Robocup robot soccer, that we have introduced to explore the challenge of P2P coordination in human-robot teams with dynamic, adversarial tasks. We recently participated in the first Segway Soccer games between two competing teams at the 2005 RoboCup US Open. We believe these games are the first ever between two human-robot P2P teams. Based on the competition, we realized two different approaches to P2P teams. We present our robot-centric approach to P2P team coordination and contrast it to the human-centric approach of the opponent team.

As we progress towards a world where robots play an integral role in society, a critical problem that remains to be solved is the pickup team challenge; that is, dynamically formed heterogeneous robot teams executing coordinated tasks where little information is known a priori about the tasks, the robots, and the environments in which they would operate. Successful solutions to forming pickup teams would enable researchers to experiment with larger numbers of robots and enable industry to efficiently and cost-effectively integrate new robot technology with existing legacy teams. In this paper, we define the challenge of pickup teams and propose the treasure hunt domain for evaluating the performance of pickup teams. Additionally, we describe a basic implementation of a pickup team that can search and discover treasure in a previously unknown environment. We build on prior approaches in market-based task allocation and plays for synchronized task execution, to allocate roles amongst robots in the pickup team, and to execute synchronized team actions to accomplish the treasure hunt task