Effective and fluent close-proximity human-robot interaction requires understanding how humans get habituated to robots and how robot motion affects human comfort. While prior work has identified humans' preferences over robot motion characteristics and studied their influence on comfort, we are yet to understand how novice first-time robot users get habituated to robots and how robot motion impacts the dynamics of comfort over repeated interactions. To take the first step towards such understanding, we carry out a user study to investigate the connections between robot motion and user comfort and habituation. Specifically, we study the influence of workspace overlap, end-effector speed, and robot motion legibility on overall comfort and its evolution over repeated interactions. Our analyses reveal that workspace overlap, in contrast to speed and legibility, has a significant impact on users' perceived comfort and habituation. In particular, lower workspace overlap leads to users reporting significantly higher overall comfort, lower variations in comfort, and fewer fluctuations in comfort levels during habituation.

Effective coordination of heterogeneous multi-robot teams requires optimizing allocations, schedules, and motion plans in order to satisfy complex multi-dimensional task requirements. This challenge is exacerbated by the fact that real-world applications inevitably introduce uncertainties into robot capabilities and task requirements. In this paper, we extend our previous work on trait-based time-extended task allocation to account for such uncertainties. Specifically, we leverage the Sequential Probability Ratio Test to develop an algorithm that can guarantee that the probability of failing to satisfy task requirements is below a user-specified threshold. We also improve upon our prior approach by accounting for temporal deadlines in addition to synchronization and precedence constraints in a Mixed-Integer Linear Programming model. We evaluate our approach by benchmarking it against three baselines in a simulated battle domain in a city environment and compare its performance against a state-of-the-art framework in a pandemic-inspired multi-robot service coordination problem. Results demonstrate the effectiveness and advantages of our approach, which leverages redundancies to manage risk while simultaneously minimizing makespan.

Close-proximity human-robot interactions can be improved through the optimization of task-centric factors or by prioritizing the user experience. Prior work has often explored these factors individually. In this paper, we conducted a within-subject study with 18 participants that compared a multi-objective robot motion adaptation method (CoMOTO) against methods that optimize distance from the user (UserAvoidant) or task performance (ShortestPath) in a close-proximity human-robot interaction task. In the task, the robot and participants worked on different tasks in an overlapping workspace. We show that while CoMOTO trajectories took a longer time, they caused significantly fewer interruptions compared to ShortestPath and generated shorter trajectories than UserAvoidant. CoMOTO was also perceived as significantly more intelligent, more trustworthy, and preferred by an overwhelming majority of the participants.

Mixed human-robot teams (HRTs) have the potential to perform complex tasks by leveraging diverse and complementary capabilities within the team. However, assigning humans to operator roles in HRTs is challenging due to the significant variation in user capabilities. While much of prior work in role assignment treats humans as interchangeable (either generally or within a category), we investigate the utility of personalized models of operator capabilities based in relevant human factors in an effort to improve overall team performance. We call this approach individualized role assignment (IRA) and provide a formal definition. A key challenge for IRA is associated with the fact that factors that affect human performance are not static (e.g., one’s ability to track multiple objects can change during or between tasks). Instead of relying on time-consuming and highly-intrusive measurements taken during the execution of tasks, we propose the use of short cognitive tests, taken before engaging in human-robot tasks, and predictive models of individual performance to perform IRA. Results from a comprehensive user study conclusively demonstrate that IRA leads to significantly better team performance than a baseline method that assumes human operators are interchangeable, even when we control for the influence of the robots’ performance. Further, our results point to the possibility that such relative benefits of IRA will increase as the number of operators (i.e., choices) increase for a fixed number of tasks.

To realize effective heterogeneous multi-robot teams, researchers must leverage individual robots’ relative strengths and coordinate their individual behaviors. Specifically, heterogeneous multi-robot systems must answer three important questions: who (task allocation), when (scheduling), and how (motion planning). While specific variants of each of these problems are known to be NP-Hard, their interdependence only exacerbates the challenges involved in solving them together. In this paper, we present a novel framework that interleaves task allocation, scheduling, and motion planning. We introduce a search-based approach for trait-based time-extended task allocation named Incremental Task Allocation Graph Search (ITAGS). In contrast to approaches that solve the three problems in sequence, ITAGS’s interleaved approach enables efficient search for allocations while simultaneously satisfying scheduling constraints and accounting for the time taken to execute motion plans. To enable effective interleaving, we develop a convex combination of two search heuristics that optimizes the satisfaction of task requirements as well as the makespan of the associated schedule. We demonstrate the efficacy of ITAGS using detailed ablation studies and comparisons against two state-of-the-art algorithms in a simulated emergency response domain.

Multi-robot task allocation (MRTA) problems involve optimizing the allocation of robots to tasks. MRTA problems are known to be challenging when tasks require multiple robots and the team is composed of heterogeneous robots. These challenges are further exacerbated when we need to account for uncertainties encountered in the real-world. In this work, we address coalition formation in heterogeneous multi-robot teams with uncertain capabilities. We specifically focus on tasks that require coalitions to collectively satisfy certain minimum requirements. Existing approaches to uncertainty-aware task allocation either maximize expected pay-off (risk-neutral approaches) or improve worst-case or near-worst-case outcomes (risk-averse approaches). Within the context of our problem, we demonstrate the inherent limitations of unilaterally ignoring or avoiding risk and show that these approaches can in fact reduce the probability of satisfying task requirements. Inspired by models that explain foraging behaviors in animals, we develop a risk-adaptive approach to task allocation. Our approach adaptively switches between risk-averse and risk-seeking behavior in order to maximize the probability of satisfying task requirements. Comprehensive numerical experiments conclusively demonstrate that our risk-adaptive approach outperforms risk-neutral and risk-averse approaches. We also demonstrate the effectiveness of our approach using a simulated multi-robot emergency response scenario.

Coordinating human-robot teams requires careful planning and allocation of tasks to the most appropriate agents. This challenge is exacerbated by the fact that, unlike their robot teammates, humans exhibit significant variation in their abilities. Existing work largely ignores this variation in favor of simpler aggregate models, failing to leverage specialized capabilities of different individuals. In this work, we introduce simple cognitive tests for measuring inherent variations in human capabilities related to human-robot teaming, specifically, the ability to maintain situational awareness and to mentally model latent network structures. We then demonstrate that user study participant performance on these cognitive tests is correlated with, and thus is a predictor for, their performance on human-robot teaming tasks. These findings have the potential to improve human-robot teaming algorithms (e.g., task allocation) by providing a mechanism to better leverage individual differences in human agents.

We address the problem of adapting robot trajectories to improve safety, comfort, and efficiency in humanrobot collaborative tasks. To this end, we propose CoMOTO, a trajectory optimization framework that utilizes stochastic motion prediction to anticipate the human's motion and adapt the robot's joint trajectory accordingly. We design a multiobjective cost function that simultaneously optimizes for i) separation distance, ii) visibility of the end-effector, iii) legibility, iv) efficiency, and v) smoothness. We evaluate CoMOTO against three existing methods for robot trajectory generation when in close proximity to humans. Our experimental results indicate that our approach consistently outperforms existing methods over a combined set of safety, comfort, and efficiency metrics.

To realize effective heterogeneous multi-agent teams, we must be able to leverage individual agents' relative strengths. Recent work has addressed this challenge by introducing trait-based task assignment approaches that exploit the agents' relative advantages. These approaches, however, assume that the agents' traits remain static. Indeed, in real-world scenarios, traits are likely to vary as agents execute tasks. In this paper, we present a transformation-based modeling framework to bridge the gap between state-of-the-art task assignment algorithms and the reality of dynamic traits. We define a transformation as a function that approximates dynamic traits with static traits based on a specific statistical measure. We define different candidate transformations, investigate their effects on different dynamic trait models, and the resulting task performance. Further, we propose a variance-based transformation as a general solution that approximates a variety of dynamic models, eliminating the need for hand specification. Finally, we demonstrate the benefits of reasoning about dynamic traits both in simulation and in a physical experiment involving the game of capture-the-flag.

Large teams of heterogeneous agents have the potential to solve complex multi-task problems that are intractable for a single agent working independently. However, solving complex multi-task problems requires leveraging the relative strengths of the different kinds of agents in the team. We present Stochastic TRAit-based Task Assignment (STRATA), a unified framework that models large teams of heterogeneous agents and performs effective task assignments. Specifically, given information on which traits (capabilities) are required for various tasks, STRATA computes the assignments of agents to tasks such that the trait requirements are achieved. Inspired by prior work in robot swarms and biodiversity, we categorize agents into different species (groups) based on their traits. We model each trait as a continuous variable and differentiate between traits that can and cannot be aggregated from different agents. STRATA is capable of reasoning about both species-level and agent-level variability in traits. Further, we define measures of diversity for any given team based on the team’s continuous-space trait model. We illustrate the necessity and effectiveness of STRATA using detailed experiments based in simulation and in a capture-the-flag game environment.

We propose a learning framework, named Multi-Coordinate Cost Balancing (MCCB), to address the problem of acquiring point-to-point movement skills from demonstrations. MCCB encodes demonstrations simultaneously in multiple differential coordinates that specify local geometric properties. MCCB generates reproductions by solving a convex optimization problem with a multi-coordinate cost function and linear constraints on the reproductions, such as initial, target, and via points. Further, since the relative importance of each coordinate system in the cost function might be unknown for a given skill, MCCB learns optimal weighting factors that balance the cost function. We demonstrate the effectiveness of MCCB via detailed experiments conducted on one handwriting dataset and three complex skill datasets.

An algorithm called adaptive-neural-intention estimator (ANIE) is presented to infer the intent of a human operator's arm movements based on the observations from a 3-D camera sensor (Microsoft Kinect). Intentions are modeled as the goal locations of reaching motions in 3-D space. Human arm's nonlinear motion dynamics are modeled using an unknown nonlinear function with intentions represented as parameters. The unknown model is learned by using a neural network. Based on the learned model, an approximate expectation-maximization algorithm is developed to infer human intentions. Furthermore, an identifier-based online model learning algorithm is developed to adapt to the variations in the arm motion dynamics, the motion trajectory, the goal locations, and the initial conditions of different human subjects. The results of experiments conducted on data obtained from different users performing a variety of reaching motions are presented. The ANIE algorithm is compared with an unsupervised Gaussian mixture model algorithm and an Euclidean distance-based approach by using Cornell's CAD-120 data set and data collected in the Robotics and Controls Laboratoy at UConn. The ANIE algorithm is compared with the inverse LQR and ATCRF algorithms using a labeling task carried out on the CAD-120 data set.

In this paper, we present an algorithm to infer the intent of a human operator's arm movements based on the observations from a Microsoft Kinect sensor. Intentions are modeled as goal locations in 3-dimensional (3D) space where the human is intending to reach. Human intention inference is a critical step towards realizing safe human-robot collaboration. This work models the human arm's nonlinear motion dynamics using an unknown nonlinear function with intentions modeled as parameters. The unknown model is learned using a neural network (NN). Based on the learned model, an approximate expectation-maximization (E-M) algorithm is developed to infer human intentions. Furthermore, an identifier-based online model learning algorithm is developed to adapt to variations in the arm motion dynamics, trajectory of motion, goal locations, and initial conditions of different human subjects. We show the results of our algorithm using two sets of experiments conducted on data obtained from different users.

In this paper, a template tracking algorithm is presented that uses mutual information (MI) criteria for template matching and gyroscope information to predict rotation between two camera images. The tracking algorithm can also take an user input for template selection and update. The tracking algorithm uses Hu moments that are invariant to 2D rotation, translation and scaling to validate the tracker. Homography is used to represent template warping parameters. The algorithm is aided with gyroscope measurements to estimate the camera motion information which helps to improve the initial guess of the warping condition. Template selection using the user's input is based on properties of the target, such as its location in the frame. The user driven strategy makes the tracker capable of tracking different objects of interest and might reduce the computational burden for template localization. The tracking algorithm presented in this paper shows significant improvements over recently developed gyro-aided Kanade-Lucas-Tomasi (KLT) tracker and the MI-only tracker in the case of multi-modal images and rapid camera motion.