Motivated by the vision of integrating mobile robots closer to humans in warehouses, hospitals, manufacturing plants, and the home, we focus on robot navigation in dynamic and spatially constrained environments. Ensuring human safety, comfort, and efficiency in such settings requires that robots are endowed with a model of how humans move around them. Human motion prediction around robots is especially challenging due to the stochasticity of human behavior, differences in user preferences, and data scarcity. In this work, we perform a methodical investigation of the effects of human motion prediction quality on robot navigation performance, as well as human productivity and impressions. We design a scenario involving robot navigation among two human subjects in a constrained workspace and instantiate it in a user study (N=80) involving two different robot platforms, conducted across two sites from different world regions. Key findings include evidence that: 1) the widely adopted average displacement error is not a reliable predictor of robot navigation performance and human impressions; 2) the common assumption of human cooperation breaks down in constrained environments, with users often not reciprocating robot cooperation, and causing performance degradations; 3) more efficient robot navigation often comes at the expense of human efficiency and comfort.

We focus on human-robot collaborative transport, in which a robot and a user collaboratively move an object to a goal pose. In the absence of explicit communication, this problem is challenging because it demands tight implicit coordination between two heterogeneous agents, who have very different sensing, actuation, and reasoning capabilities. Our key insight is that the two agents can coordinate fluently by encoding subtle, communicative signals into actions that affect the state of the transported object. To this end, we design an inference mechanism that probabilistically maps observations of joint actions executed by the two agents to a set of joint strategies of workspace traversal. Based on this mechanism, we define a cost representing the human's uncertainty over the unfolding traversal strategy and introduce it into a model predictive controller that balances between uncertainty minimization and efficiency maximization. We deploy our framework on a mobile manipulator (Hello Robot Stretch) and evaluate it in a within-subjects lab study$(N=24)$. We show that our framework enables greater team performance and empowers the robot to be perceived as a significantly more fluent and competent partner compared to baselines lacking a communicative mechanism.

We focus on push-based multi-object rearrangement planning using a nonholonomically constrained mobile robot. The simultaneous geometric, kinematic, and physics constraints make this problem especially challenging. Prior work on rearrangement planning often relaxes some of these constraints by assuming dexterous hardware, prehensile manipulation, or sparsely occupied workspaces. Our key insight is that by capturing these constraints into a unified representation, we could empower a constrained robot to tackle difficult problem instances by modifying the environment in its favor. To this end, we introduce a push-traversability graph, whose vertices represent poses that the robot can push objects from, and edges represent optimal, kinematically feasible, and stable transitions between them. Based on this graph, we develop ReloPush, a graph-based planning framework that takes as input a complex multi-object rearrangement task and breaks it down into a sequence of single-object pushing tasks. We evaluate ReloPush across a series of challenging scenarios, involving the rearrangement of densely cluttered workspaces with up to nine objects, using a 1/10-scale robot racecar. ReloPush exhibits orders of magnitude faster runtimes and significantly more robust execution in the real world, evidenced in lower execution times and fewer losses of object contact, compared to two baselines lacking our proposed graph structure.

The integration of advances from machine learning and computer vision with the classical autonomy stack has brought successful robot deployments in fulfilment, manufacturing, and transportation. However, unstructured and dynamic environments such as pedestrian spaces and streets, workplaces, and homes pose additional challenges such as modeling human behavior, understanding user perceptions, and ensuring human safety and comfort. My work addresses such challenges to enable robots to fluently work with and around people to increase productivity and assist users.

Cooking recipes are challenging to translate to robot plans as they feature rich linguistic complexity, temporally-extended interconnected tasks, and an almost infinite space of possible actions. Our key insight is that combining a source of cooking domain knowledge with a formalism that captures the temporal richness of cooking recipes could enable the extraction of unambiguous, robot-executable plans. In this work, we use Linear Temporal Logic (LTL) as a formal language expressive enough to model the temporal nature of cooking recipes. Leveraging a pretrained Large Language Model (LLM), we present Cook2LTL, a system that translates instruction steps from an arbitrary cooking recipe found on the internet to a set of LTL formulae, grounding high-level cooking actions to a set of primitive actions that are executable by a manipulator in a kitchen environment. Cook2LTL makes use of a caching scheme that dynamically builds a queryable action library at runtime. We instantiate Cook2LTL in a realistic simulation environment (AI2-THOR), and evaluate its performance across a series of cooking recipes. We demonstrate that our system significantly decreases LLM API calls (−51%), latency (−59%), and cost (−42%) compared to a baseline that queries the LLM for every newly encountered action at runtime.

We focus on robot navigation in crowded environments. To navigate safely and efficiently within crowds, robots need models for crowd motion prediction. Building such models is hard due to the high dimensionality of multiagent domains and the challenge of collecting or simulating interaction-rich crowd-robot demonstrations. While there has been important progress on models for offline pedestrian motion forecasting, transferring their performance on real robots is nontrivial due to close interaction settings and novelty effects on users. In this paper, we investigate the utility of a recent state-of-the-art motion prediction model (S-GAN) for crowd navigation tasks. We incorporate this model into a model predictive controller (MPC) and deploy it on a self-balancing robot which we subject to a diverse range of crowd behaviors in the lab. We demonstrate that while S-GAN motion prediction accuracy transfers to the real world, its value is not reflected on navigation performance, measured with respect to safety and efficiency; in fact, the MPC performs indistinguishably even when using a simple constant-velocity prediction model, suggesting that substantial model improvements might be needed to yield significant gains for crowd navigation tasks. Footage from our experiments can be found at https://youtu.be/mzFiXgSKsZ0.

We focus on the problem of rearranging a set of objects with a team of car-like robot pushers built using off-the-shelf components. Maintaining control of pushed objects while avoiding collisions in a tight space demands highly coordinated motion that is challenging to execute on constrained hardware. Centralized replanning approaches become intractable even for small-sized problems whereas decentralized approaches often get stuck in deadlocks. Our key insight is that by carefully assigning pushing tasks to robots, we could reduce the complexity of the rearrangement task, enabling robust performance via scalable decentralized control. Based on this insight, we built PuSHR, a system that optimally assigns pushing tasks and trajectories to robots offline, and performs trajectory tracking via decentralized control online. Through an ablation study in simulation, we demonstrate that PuSHR dominates baselines ranging from purely centralized to fully decentralized in terms of success rate and time efficiency across challenging tasks with up to 4 robots. Hardware experiments demonstrate the transfer of our system to the real world and highlight its robustness to model inaccuracies. Our code can be found at https://github.com/prl-mushr/pushr, and videos from our experiments at https://youtu.be/nyUn9mHoR8Y.

Robot navigation in crowded public spaces is a complex task that requires addressing a variety of engineering and human factors challenges. These challenges have motivated a great amount of research resulting in important developments for the fields of robotics and human-robot interaction over the past three decades. Despite the significant progress and the massive recent interest, we observe a number of significant remaining challenges that prohibit the seamless deployment of autonomous robots in crowded environments. In this survey article, we organize existing challenges into a set of categories related to broader open problems in robot planning, behavior design, and evaluation methodologies. Within these categories, we review past work and offer directions for future research. Our work builds upon and extends earlier survey efforts by (a) taking a critical perspective and diagnosing fundamental limitations of adopted practices in the field and (b) offering constructive feedback and ideas that could inspire research in the field over the coming decade.

One of the fundamental modes of learning in children is through curiosity. Children (and adults) interact with new people, learn about novel objects, activities and other stimuli through curiosity and other intrinsic motivations. Creating autonomous robots that learn continually through intrinsic curiosity may result in breakthroughs in artificial intelligence. Such robots could continue to learn about themselves and the world around them through curiosity, thus improving their abilities over their ‘lifetime’. Although recent works on curiosity in different fields have produced significant results, most of these works have focused on constrained simulated environments which do not involve human interaction. However, in real-world applications such as healthcare, home-assistance etc., robots generally have to interact with humans on a regular basis. In these scenarios, it is imperative that curiosity is directed towards seeking out and learning important information from the humans when needed rather than simply learning in an unsupervised manner. Further, there is limited work on how humans perceive such curious robots and whether humans prefer curious robots that adapt over time to other robots that simply perform their assigned tasks. In this workshop, our goal is to bring together researchers and practitioners in different multidisciplinary fields to discuss the role of robot curiosity in real-world applications and its implications in human-robot interaction (HRI).

It is difficult to run long-term in-the-wild studies with mobile robots. This is partly because the robots we, as human-robot interaction (HRI) researchers, are interested in deploying prioritize expressivity over navigational capabilities, and making those robots autonomous is often not the focus of our research. One way to address these difficulties is with the Wizard of Oz (WoZ) methodology, where a researcher teleop-erates the robot during its deployment. However, the constant attention required for teleoperation limits the duration of WoZ deployments, which in-turn reduces the amount of in-the-wild data we are able to collect. Our key insight is that several types of in-the-wild mobile robot studies can be run without autonomous navigation, using wandering instead. In this paper we present and share code for our wandering robot system, which enabled Kuri, an expressive robot with limited sensor and computational capabilities, to traverse the hallways of a$28,000 \text{ ft}^{2}$floor for four days. Our system relies on informed direction selection to avoid obstacles and traverse the space, and periodic human help to charge. After presenting the outcomes from the four-day deployment, we then discuss the benefits of deploying a wandering robot, explore the types of in-the-wild studies that can be run with wandering robots, and share pointers for enabling other robots to wander. Our goal is to add wandering to the toolbox of navigation approaches HRI researchers use, particularly to run in-the-wild deployments with mobile robots.

We focus on the problem of analyzing multiagent interactions in traffic domains. Understanding the space of behavior of real-world traffic may offer significant advantages for algorithmic design, data-driven methodologies, and bench-marking. However, the high dimensionality of the space and the stochasticity of human behavior may hinder the identification of important interaction patterns. Our key insight is that traffic environments feature significant geometric and temporal structure, leading to highly organized collective behaviors, often drawn from a small set of dominant modes. In this work, we propose a representation based on the formalism of topological braids that can summarize arbitrarily complex multiagent behavior into a compact object of dual geometric and symbolic nature, capturing critical events of interaction. This representation allows us to formally enumerate the space of outcomes in a traffic scene and characterize their complexity. We illustrate the value of the proposed representation in summarizing critical aspects of real-world traffic behavior through a case study on recent driving datasets. We show that despite the density of real-world traffic, observed behavior tends to follow highly organized patterns of low interaction. Our framework may be a valuable tool for evaluating the richness of driving datasets, but also for synthetically designing balanced training datasets or benchmarks.

During in-hand manipulation, robots must be able to continuously estimate the pose of the object in order to generate appropriate control actions. The performance of algorithms for pose estimation hinges on the robot's sensors being able to detect discriminative geometric object features, but previous sensing modalities are unable to make such measurements robustly. The robot's fingers can occlude the view of environment- or robot-mounted image sensors, and tactile sensors can only measure at the local areas of contact. Motivated by fingertip-embedded proximity sensors' robustness to occlusion and ability to measure beyond the local areas of contact, we present the first evaluation of proximity sensor based pose estimation for in-hand manipulation. We develop a novel two-fingered hand with fingertip-embedded optical time-of-flight proximity sensors as a testbed for pose estimation during planar in-hand manipulation. Here, the in-hand manipulation task consists of the robot moving a cylindrical object from one end of its workspace to the other. We demonstrate, with statistical significance, that proximity-sensor based pose estimation via particle filtering during in-hand manipulation: a) exhibits 50% lower average pose error than a tactile-sensor based baseline; b) empowers a model predictive controller to achieve 30% lower final positioning error compared to when using tactile-sensor based pose estimates.

Mobile robots struggle to integrate seamlessly in crowded environments such as pedestrian scenes, often disrupting human activity. One obstacle preventing their smooth integration is our limited understanding of how humans may perceive and react to robot motion. Motivated by recent studies highlighting the benefits of intent-expressive motion for robots operating close to humans, we describe Social Momentum (SM), a planning framework for legible robot motion generation in multiagent domains. We investigate the properties of motion generated by SM via two large-scale user studies: an online, video-based study ( N = 180) focusing on the legibility of motion produced by SM and a lab study ( N = 105) focusing on the perceptions of users navigating next to a robot running SM in a crowded space. Through statistical and thematic analyses of collected data, we present evidence suggesting that (a) motion generated by SM enables quick inference of the robot’s navigation strategy; (b) humans navigating close to a robot running SM follow comfortable, low-acceleration paths; and (c) robot motion generated by SM is positively perceived and indistinguishable from a teleoperated baseline. Through the discussion of experimental insights and lessons learned, this article aspires to inform future algorithmic and experimental design for social robot navigation.

Chopsticks constitute a simple yet versatile tool that humans have used for thousands of years to perform a variety of challenging tasks ranging from food manipulation to surgery. Applying such a simple tool in a diverse repertoire of scenarios requires significant adaptability. Towards developing autonomous manipulators with comparable adaptability to humans, we study chopsticks-based manipulation to gain insights into human manipulation strategies. We conduct a within-subjects user study with 25 participants, evaluating three different data-collection methods: normal chopsticks, motion-captured chopsticks, and a novel chopstick telemanipulation interface. We analyze factors governing human performance across a variety of challenging chopstick-based grasping tasks. Although participants rated teleoperation as the least comfortable and most difficult-to-use method, teleoperation enabled users to achieve the highest success rates on three out of five objects considered. Further, we notice that subjects quickly learned and adapted to the teleoperation interface. Finally, while motion-captured chopsticks could provide a better reflection of how humans use chopsticks, the teleoperation interface can produce quality on-hardware demonstrations from which the robot can directly learn.

State-of-the-art social robot navigation algorithms often lack a thorough experimental validation in human environments: simulated evaluations are often conducted under unrealistically strong assumptions that prohibit deployment in real world environments; experimental demonstrations that are limited in sample size do not provide adequate evidence regarding the user experience and the robot behavior; field studies may suffer from the noise imposed by uncontrollable factors from the environment; controlled lab experiments often fail to properly enforce challenging interaction settings. This paper contributes a first step towards addressing the outlined gaps in the literature. We present an original experiment, designed to test the implicit interaction between a mobile robot and a group of navigating human participants, under challenging settings in a controlled lab environment. We conducted a large-scale, within-subjects design study with 105 participants, exposed to three different conditions, corresponding to three distinct navigation strategies, executed by a telepresence robot (two autonomous, one teleoperated). We analyzed observed human and robot trajectories, under close interaction settings and participants' impressions regarding the robot's behavior. Key findings, extracted from a comparative statistical analysis include: (1) evidence that human acceleration is lower when navigating around an autonomous robot compared to a teleoperated one; (2) the lack of evidence to support the conventional expectation that teleoperation would be humans' preferred strategy. To the best of our knowledge, our study is unique in terms of goals, settings, thoroughness of evaluation and sample size.

Intent-expressive robot motion has been shown to result in increased efficiency and reduced planning efforts for copresent humans. Existing frameworks for generating intent-expressive robot behaviors have typically focused on applications in static or structured environments. Under such settings, emphasis is placed towards communicating the robot»s intended final configuration to other agents. However, in dynamic, unstructured and multi-agent domains, such as pedestrian environments, knowledge of the robot»s final configuration is not sufficiently informative as it completely ignores the complex dynamics of interaction among agents. To address this problem, we design a planning framework that aims at generating motion that clearly communicates an agent»s intended collision avoidance strategy rather than its destination. Our framework estimates the most likely intended avoidance protocols of others based on their past behaviors, superimposes them, and generates an expressive and socially compliant robot action that reinforces the expectations of others regarding these avoidance protocols. This action facilitates inference and decision making for everyone, as illustrated in the simplified topological pattern of agents» trajectories. Extensive simulations demonstrate that our framework consistently achieves significantly lower topological complexity, compared against common benchmark approaches in multi-agent collision avoidance. The significance of this result for real world applications is demonstrated by a user study that reveals statistical evidence suggesting that multi-agent trajectories of lower topological complexity tend to facilitate inference for observers.

Robots must be cognizant of how their actions will be interpreted in context. Actions performed in the context of a joint activity comprise two aspects: functional and communicative. The functional component achieves the goal of the action, whereas its communicative component, when present, expresses some information to the actor's partners in the joint activity. The interpretation of such communication requires leveraging information that is public to all participants, known as common ground. Much of human communication is performed through this implicit mechanism, and humans cannot help but infer some meaning -- whether or not it was intended by the actor -- from most actions. We present a framework for robots to utilize this communicative channel on top of normal functional actions to work more effectively with human partners. We consider the role of the actor and the observer, both individually and jointly, in implicit communication, as well as the effects of timing. We also show how the framework maps onto various modes of action, including natural language and motion. We consider these modes of action in various human-robot interaction domains, including social navigation and collaborative assembly.

We present a novel, data-driven framework for planning socially competent robot behaviors in crowded environments. The core of our approach is a topological model of collective navigation behaviors, based on braid groups. This model constitutes the basis for the design of a human-inspired probabilistic inference mechanism that predicts the topology of multiple agents' future trajectories, given observations of the context. We derive an approximation of this mechanism by employing a neural network learning architecture on synthetic data of collective navigation behaviors. Our planner makes use of this mechanism as a tool for interpreting the context and understanding what future behaviors are in compliance with it. The planning agent makes use of this understanding to determine a personal action that contributes to the context in the most clear way possible, while ensuring progress to its destination. Our simulations provide evidence that our planning framework results in socially competent navigation behaviors not only for the planning agent, but also for interacting naive agents. Performance benefits include (1) early conflict resolutions and (2) faster uncertainty decrease for the other agents in the scene.

In this paper we present an open-source design for the development of low-complexity, anthropomorphic, underactuated robot hands with a selectively lockable differential mechanism. The differential mechanism used is a variation of the whiffletree (or seesaw) mechanism, which introduces a set of locking buttons that can block the motion of each finger. The proposed design is unique since with a single motor and the proposed differential mechanism the user is able to control each finger independently and switch between different grasping postures in an intuitive manner. Anthropomorphism of robot structure and motion is achieved by employing in the design process an index of anthropomorphism. The proposed robot hands can be easily fabricated using low-cost, off-the-shelf materials and rapid prototyping techniques. The efficacy of the proposed design is validated through different experimental paradigms involving grasping of everyday life objects and execution of daily life activities. The proposed hands can be used as affordable prostheses, helping amputees regain their lost dexterity.

In this paper we introduce an index for the quantification of anthropomorphism of robot arms. The index is defined as a weighted sum of specific metrics which evaluate the similarities between the human and robot arm workspaces, providing a normalized score between 0 (non-anthropomorphic artifacts) and 1 (human-identical artifacts). The human arm workspaces were extracted using data reported in anthropometry studies. The formulation is general enough to allow utilization in various applications, by adjusting the weighting factors according to the specifications of each study. The proposed methodology can be used for assessing the human-likeness of existing robot arms as well as to provide specifications for the design of new anthropomorphic robots and prosthetic devices. To assess the efficiency of the proposed methods a comparative analysis between five kinematically different robot arm models is conducted and simulated paradigms are presented.

In this paper, we propose an optimization scheme for deriving task-specific force closure grasps for underactuated robot hands. Motivated by recent neuroscientific studies on the human grasping behavior, a novel grasp strategy is built upon past analysis regarding the task-specificity of human grasps, that also complies with the recent soft synergy model of underactuated hands. Our scheme determines an efficient force closure grasp (i.e., configuration and contact points/forces) with a posture compatible with the desired task, taking into consideration the mechanical and geometric limitations imposed by the design of the hand and the object shape. The efficiency of the algorithm is verified through simulated paradigms on a hypothetical underactuated hand with the kinematic model of the DLR/HIT II five fingered robot hand.

In this paper we present a series of design directions for the development of affordable, modular, light-weight, intrinsically-compliant, underactuated robot hands, that can be easily reproduced using off-the-shelf materials. The proposed robot hands, efficiently grasp a series of everyday life objects and are considered to be general purpose, as they can be used for various applications. The efficiency of the proposed robot hands has been experimentally validated through a series of experimental paradigms, involving: grasping of multiple everyday life objects with different geometries, myoelectric (EMG) control of the robot hands in grasping tasks, preliminary results on a grasping capable quadrotor and autonomous grasp planning under object position and shape uncertainties.

In this work, we present a novel concept in the area of optimal grasp synthesis, confronting both geometric and mechanical constraints. Initializing from a locally optimal force distribution on some predefined feasible contact points, our method improves gradually the grasp quality avoiding simultaneously singularities and mechanical limitations. The proposed scheme implements sequential perturbations on the contact points and the wrist's position/orientation incorporating a post-optimality method in an iterative process to derive the consecutive optimal states. The main novelty of this work lies in the fact that only local information of the object's surface is required, which can be provided for instance by an appropriate tactile sensor suite. Finally, a simulation study on the DLR/HIT Hand II clarifies and verifies the efficiency of the approach.