Robot programming languages that represent tasks as graph structures are both popular and accessible among programming novices and experts. However, these languages are largely decoupled from robots' automated task planning capabilities, rendering developers unable to explicitly leverage their robot's ability to plan its own actions. We thereby created the Interaction Specification Language (ISL), which enables de-velopers to import and apply elements from a robot planning do-main in a graph-based programming paradigm. For developers, ISL provides flexibility in the reliance on automated planning. For researchers, the release of our open-source ISL lexer and parser is intended to promote standardization and test-driven development. We additionally provide a metric by which ISL programs can be evaluated.

Robot end users increasingly require accessible means of specifying tasks for robots to perform. Two common end-user programming paradigms include drag-and-drop interfaces and natural language programming. Although natural language interfaces harness an intuitive form of human communication, drag-and-drop interfaces enable users to meticulously and precisely dictate the key actions of the robot’s task. In this paper, we investigate the degree to which both approaches can be combined. Specifically, we construct a large language model (LLM)-based pipeline that accepts natural language as input and produces human-like action sequences as output, specified at a level of granularity that a human would produce. We then compare these generated action sequences to another dataset of hand-specified action sequences. Although our results reveal that larger models tend to outperform smaller ones in the production of human-like action sequences, smaller models nonetheless achieve satisfactory performance.

Assistive robots will be more effective if they can accurately reason about the intentions and beliefs of the user (i.e., have Theory of Mind (ToM)). ToM benchmarks allow us to examine how well an artificial agent (e.g., robot) is able to do ToM reasoning in a given scenario. However, there is a need for ToM benchmarks that are more representative of the challenges faced in assistive robotics. Existing benchmarks from AI and HRI make simplifying assumptions, such as simply defined goals, plans that are indicative of goals, and no user errors. To address the challenges from relaxing these assumptions, we propose the Theory of Mind of Children Assembling Tangrams (ToMCAT) dataset. The data is derived from videos of children building tangram puzzles while being assisted by a social robot. As a baseline benchmark, we evaluated two approaches for how well they can recognize which puzzle this child is building based on a single observation. Analogical reasoning correctly recognized the puzzle more than 75% of the time and had perfect accuracy for puzzle states that were close to complete. However, an out-of-the-box commercial LLM correctly recognized the puzzle only 60% of the time and was accurate on less than 80% of the completed puzzles. Our results suggest that the ToMCAT dataset offers challenges for recognizing the intended puzzle of a child. Furthermore, the dataset provides opportunities to examine additional ToM reasoning capabilities. Overall, the ToMCAT dataset provides a useful benchmark to facilitate the advancement of ToM reasoning for assistive robotics.

End-user programming (EUP) tools must balance user control with the robot's ability to plan and act autonomously. Many existing task-oriented EUP tools enforce a specific level of control, e.g., by requiring that users hand-craft detailed sequences of actions, rather than offering users the flexibility to choose the level of task detail they wish to express. We thereby created a novel EUP system, Polaris, that in contrast to most existing EUP tools, uses goal predicates as the fundamental building block of programs. Users can thereby express high-level robot objectives or lower-level checkpoints at their choosing, while an off-the-shelf task planner fills in any remaining program detail. To ensure that goal-specified programs adhere to user expectations of robot behavior, Polaris is equipped with a Plan Visualizer that exposes the planner's output to the user before runtime. In what follows, we describe our design of Polaris and its evaluation with 32 human participants. Our results support the Plan Visualizer's ability to help users craft higher-quality programs. Furthermore, there are strong associations between user perception of the robot and Plan Visualizer usage, and evidence that robot familiarity has a key role in shaping user experience.

When robots have multiple tasks to perform, they must determine the order in which to complete them. Interleaving tasks is efficient for the robot trying to finish its to-do list, but it may be less satisfying for a human whose request was delayed in favor of schedule efficiency. Following online research that examined delays with various motivations, we created two in-person studies in which participants’ tasks were impacted by the robot’s other tasks. In the first, participants either requested a task for the robot to complete on their behalf or watched the robot performing tasks for other people. We measured how their opinions changed depending on whether their task’s completion was delayed due to another participant’s task or they were observing without a task of their own. In the second, participants had a robot walk them to an office and became delayed as the robot detoured to another location. We measured how opinions of the robot changed depending on who requested the detour task and the length of the detour. Overall, participants positively viewed task interleaving as long as the delay and inconvenience imposed by someone else’s task were small and the task was well-justified. Also, observers often had lower opinions of the robot than participants who requested tasks, highlighting a concern for online research.

Designing novel application development environments (ADEs) is a growing area of systems research within the human-robot interaction (HRI) community. This research involves the design of a novel system, the ADE, to afford end users and application designers the ability to develop robot applications. Researchers then usually validate their ADEs in the form of user studies or a series of case studies. In this paper, we highlight a problem with the typical approach to conducting ADE research within HRI—there is currently little standardization in how these systems are designed, developed, and validated, leading to difficulty in sharing resources between different research groups and the inability to compare similar ADEs to each other. We argue that a standardized formal representation embedded within an Interaction Specification Language (ISL) can lead to more streamlined development and validation of ADEs for HRI. Furthermore, we discuss several desired characteristics that an ISL should embody.

Recent work has suggested that a robot that in-terrupts assigned tasks for the sake of curiosity is perceived as less competent, but that communicating acknowledgment of the curious behavior can mitigate some of those feelings [1]. In real-world situations, there are many reasons why a robot's task could be interrupted in favor of another. For example, a robot handling requests for tasks from people in different locations could navigate more efficiently if it interleaves those tasks, but it ideally would not do so at the expense of the users' perceptions of the robot. In order to understand the impact of different task interleaving patterns on human perceptions of a robot's behavior, we performed a study in which a robot performed a delivery task and an investigative task, interleaving them in various ways. The participants were told either that the investigative task was motivated by a request from another person, motivated by curiosity, or they received no information about why the robot performed the action. While participants acknowledged that interleaving tasks should be allowed, they rated the robot as more competent when its tasks were not interleaved. They were most receptive to interleaving when they knew the investigative task was for another person and less receptive to long task detours away from the delivery route, especially when the inspection task was motivated by curiosity.

In this paper, we present a novel shared-control telemanipulation method that is designed to incrementally improve a user»s motor ability. Our method initially corrects for the user»s suboptimal control trajectories, gradually giving the user more direct control over a series of training trials as he/she naturally gets more accustomed to the task. Our shared-control method, calledShared Dynamic Curves, blends suboptimal user translation and rotation control inputs with known translation and rotation paths needed to complete a task. Shared Dynamic Curves provide a translation and rotation path in space along which the user can easily guide the robot, and this curve can bend and flex in real-time as a dynamical system to pull the user»s motion gracefully toward a goal. We show through a user study that Shared Dynamic Curves affords effective motor learning on certain tasks compared to alternative training methods. We discuss our findings in the context of shared control and speculate on how this method could be applied in real-world scenarios such as job training or stroke rehabilitation.

In Navy human firefighting teams, touch is used extensively to communicate among teammates. In noisy, chaotic, and visually challenging environments, such as among fires on Navy ships, this is the only reliable means of communication. The overarching goal of this work is to augment Navy firefighting teams with an autonomous robot serving as a nozzle operator; to accomplish this, the robot must understand the tactile gestures of its human teammates. Preliminary results recognizing touch gestures have indicated the potential of such an autonomous system to serve as a nozzle operator in human-centric firefighting scenarios.

We present ACT-R/E (Adaptive Character of Thought-Rational / Embodied), a cognitive architecture for human-robot interaction. Our reason for using ACT-R/E is two-fold. First, ACT-R/E enables researchers to build good embodied models of people to understand how and why people think the way they do. Then, we leverage that knowledge of people by using it to predict what a person will do in different situations; e.g., that a person may forget something and may need to be reminded or that a person cannot see everything the robot sees. We also discuss methods of how to evaluate a cognitive architecture and show numerous empirically validated examples of ACT-R/E models.

The variability of human behavior during plan execution poses a difficult challenge for human-robot teams. In this paper, we use the concepts of theory of mind to enable robots to account for two sources of human variability during team operation. When faced with an unexpected action by a human teammate, a robot uses a simulation analysis of different hypothetical cognitive models of the human to identify the most likely cause for the human's behavior. This allows the cognitive robot to account for variances due to both different knowledge and beliefs about the world, as well as different possible paths the human could take with a given set of knowledge and beliefs. An experiment showed that cognitive robots equipped with this functionality are viewed as both more natural and intelligent teammates, compared to both robots who either say nothing when presented with human variability, and robots who simply point out any discrepancies between the human's expected, and actual, behavior. Overall, this analysis leads to an effective, general approach for determining what thought process is leading to a human's actions.

The capability to assemble structures is fundamental to the use of robotics in precursor missions in orbit and on planetary surfaces. We have performed autonomous assembly in neutral buoyancy of elements of a space truss whose mating components require positioning tolerances of the same order of magnitude as the noise in the sensor systems used for the docking. Numerous trade-offs, design decisions, and innovations were made during the development of the assembly system in order to both reduce and compensate for the sensor noise. By using relative positioning, decoupling sensing and manipulation, caching high-quality position estimates, and developing a new waypoint-completion metric, we were able to reduce sensor noise to the sub-millimeter level and autonomously assemble components with millimeter tolerances. In this paper, we discuss our approaches to the problem and report the results of a series of autonomous assembly operations.

In many robotic domains, efficiency is an important component of task execution. One way to improve task efficiency is to lessen the overhead of beginning a task by making sure the necessary agents are near the task site when execution begins, minimizing travel time delays - in other words, pre-positioning agents for their future tasks. In static, certain domains, this can easily be done in advance and incorporated into the initial plan. In dynamic domains such as search and rescue, however, there is not enough certainty about task execution to plan for this ahead of time. To address this, we present here a planner that adds pre-positioning to a plan during execution. The planner strategically positions groups of idle robots whose future task assignments are uncertain in order to minimize travel time by the group as a whole once its members are allocated tasks. Because this planner must run in real time, we present five versions of the planning algorithm, addressing the trade-off of computation time and solution quality that results. We then show that by adding in this type of planning, the overhead of beginning a task can be reduced by up to 90%.

In order for humans and robots to work effectively together, they need to be able to converse about abilities, goals and achievements. Thus, we are developing an interaction infrastructure called the "Human-Robot Interaction Operating System" (HRI/OS). The HRI/OS provides a structured software framework for building human-robot teams, supports a variety of user interfaces, enables humans and robots to engage in task-oriented dialogue, and facilitates integration of robots through an extensible API.

This paper presents a robot search task (social tag) that uses social interaction, in the form of asking for help, as an integral component of task completion. We define socially distributed perception as a robot's ability to augment its limited sensory capacities through social interaction.

We are interested in the problem of a human operator who has to respond to requests for help from an autonomous robotic construction team. A difficult aspect of this problem is gaining an awareness of the requesting robot’s situation quickly enough to avoid slowing the whole team down. One approach to speeding the initial acquisition of situational awareness is to maintain a buffer of data, and play it back for the human when their help is needed. The paper reports on an experiment to determine the proper composition and length of this buffer for our domain of multi-robot construction. The experiments show that 5- 10 seconds of one raw video feed in combination with a processed display led to the fastest operator attainment of situational awareness. We draw several conclusions from this experiment, which may generalize to other scenarios.

An important mode of human-robot interaction is teleoperation, in which a human operator directly controls a robot via hardware such as a joystick or mouse. Such control is not always easy, however, as the viewpoint of the human, the alignment of the input device, and the local coordinate frame of the robot are rarely all aligned. These discrepancies force the user to reconcile the involved coordinate frames during teleoperation. Therefore, the choice of coordinate frames is critical since an unintuitive coordinate frame mapping will likely lead to higher mental workload and reduced efficiency. We discuss this concern, describe the various difficulties involved with natural remote teleoperation of a robot, and report experiments that demonstrate the effects of using different frames of reference on task performance and user mental workload

The Peer-To-Peer Human-Robot Interaction (P2P-HRI) project is developing techniques to improve task coordination and collaboration between human and robot partners. Our work is motivated by the need to develop effective human-robot teams for space mission operations. A central element of our approach is creating dialogue and interaction tools that enable humans and robots to flexibly support one another. In order to understand how this approach can influence task performance, we recently conducted a series of tests simulating a lunar construction task with a human-robot team. In this paper, we describe the tests performed, discuss our initial results, and analyze the effect of intervention on task performance.

Recent research in human-robot interaction has investigated the concept of Sliding, or Adjustable, Autonomy, a mode of operation bridging the gap between explicit teleoperation and complete robot autonomy. This work has largely been in single-agent domains-involving only one human and one robot-and has not examined the issues that arise in multiagent domains. We discuss the issues involved in adapting Sliding Autonomy concepts to coordinated multiagent teams. In our approach, remote human operators have the ability to join, or leave, the team at will to assist the autonomous agents with their tasks (or aspects of their tasks) while not disrupting the team's coordination. Agents model their own and the human operator's performance on subtasks to enable them to determine when to request help from the operator. To validate our approach, we present the results of two experiments. The first evaluates the human/multirobot team's performance under four different collaboration strategies including complete teleoperation, pure autonomy, and two distinct versions of Sliding Autonomy. The second experiment compares a variety of user interface configurations to investigate how quickly a human operator can attain situational awareness when asked to help. The results of these studies support our belief that by incorporating a remote human operator into multiagent teams, the team as a whole becomes more robust and efficient