We present a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration (PARTNR) designed to study human-robot coordination in household activities. PARTNR tasks exhibit characteristics of everyday tasks, such as spatial, temporal, and heterogeneous agent capability constraints. We employ a semi-automated task generation pipeline using Large Language Models (LLMs), incorporating simulation-in-the-loop for the grounding and verification. PARTNR stands as the largest benchmark of its kind, comprising 100,000 natural language tasks, spanning 60 houses and 5,819 unique objects. We analyze state-of-the-art LLMs on PARTNR tasks, across the axes of planning, perception and skill execution. The analysis reveals significant limitations in SoTA models, such as poor coordination and failures in task tracking and recovery from errors. When LLMs are paired with 'real' humans, they require 1.5x as many steps as two humans collaborating and 1.1x more steps than a single human, underscoring the potential for improvement in these models. We further show that fine-tuning smaller LLMs with planning data can achieve performance on par with models 9 times larger, while being 8.6x faster at inference. Overall, PARTNR highlights significant challenges facing collaborative embodied agents and aims to drive research in this direction.

As robots and other autonomous agents are increasingly incorporated into complex domains, characterizing interaction within heterogeneous teams that include both humans and machines becomes more necessary. Previous literature has addressed the task of characterizing human-robot interaction from different perspectives and in multiple contexts. However, the numerous factors behind interaction work in conjunction, and the insights gained from one perspective can inadvertently affect another, creating a need for unification of these taxonomies and frameworks within an overarching taxonomy that systematically defines these relationships. In this paper we review existing taxonomies related to human-robot interaction, the behavioral sciences, and social and algorithmic taxonomies, and propose an overarching ontology for the factors from these works. We identify three main components characterizing the structure of an interaction (environment, task, and team), and structure them over two levels: contextual factors and factors driven by local dynamics. Finally, we present an analysis of how these factors affect decisions about levels of robot automation and level of information abstraction in an interaction, and discuss curent gaps in the literature that can motivate future research.

We present a method to learn context-dependent outcomes of behaviors in unstructured indoor environments. The idea is that certain features in the environment may be predictive of differences in outcomes, such as how long a mobile robot takes to traverse a corridor. Doing so enables the robot to plan more effectively, and also be able to interact with people more effectively by more accurately predicting when its plans may take longer to execute or may be likely to fail. We use a node-and-edge based map of the environment and treat the traversal time of the robot for each edge as a random variable to be characterized. The first step is to determine whether the distribution of the random variable is multimodal and, if so, we learn to classify the modes using a hierarchy of plan-time features (e.g., time of the day, day of the week) and run-time features (observations of recent traversal times through other corridors). We utilize a cascading regression system that first estimates which mode of the traversal distribution we expect the robot to observe, and then predict the actual traversal time through a corridor. On average, our method produces a mean residual error of less than 2.7 seconds.