Fluid 6DOF teleoperation of robot manipulators enables telemanipulation where autonomy is not possible, facilitates the collection of demonstration data, and aids routine robotics development. Amongst 6DOF input devices, 3D mice stand apart for their ergonomic design and low cost, but their sensitivity and users' relative inexperience with them require special design considerations. We contribute a web software package that makes integrating 3D mice in robot manipulation interfaces easy. The package consists of configurable input signal processing schemes that can make the device more forgiving by, for instance, rejecting small inputs or emphasizing a dominant axis, and an interactive visual representation of the device's 6DOF twist input, which helps with operator familiarization and provides a visual aide during teleoperation. We provide a demonstration interface illustrating a typical integration with a ROS/ROS2 robot system and give usage advice based on our research experience.

The performance of prediction-based assistance for robot teleoperation degrades in unseen or goal-rich environments due to incorrect or quickly-changing intent inferences. Poor predictions can confuse operators or cause them to change their control input to implicitly signal their goal. We present a new assistance interface for robotic manipulation where an operator can explicitly communicate a manipulation goal by pointing the end-effector. The pointing target specifies a region for local pose generation and optimization, providing interactive control over grasp and placement pose candidates. We evaluate this explicit pointing interface against an implicit inference-based assistance scheme and an unassisted control condition in a within-subjects user study (N=20), where participants teleoperate a simulated robot to complete a multi-step singulation and stacking task in cluttered environments. We find that operators prefer the explicit interface, experience fewer pick failures and report lower cognitive workload. Our code is available at: github.com/NVlabs/fast-explicit-teleop.

It is difficult to run long-term in-the-wild studies with mobile robots. This is partly because the robots we, as human-robot interaction (HRI) researchers, are interested in deploying prioritize expressivity over navigational capabilities, and making those robots autonomous is often not the focus of our research. One way to address these difficulties is with the Wizard of Oz (WoZ) methodology, where a researcher teleop-erates the robot during its deployment. However, the constant attention required for teleoperation limits the duration of WoZ deployments, which in-turn reduces the amount of in-the-wild data we are able to collect. Our key insight is that several types of in-the-wild mobile robot studies can be run without autonomous navigation, using wandering instead. In this paper we present and share code for our wandering robot system, which enabled Kuri, an expressive robot with limited sensor and computational capabilities, to traverse the hallways of a$28,000 \text{ ft}^{2}$floor for four days. Our system relies on informed direction selection to avoid obstacles and traverse the space, and periodic human help to charge. After presenting the outcomes from the four-day deployment, we then discuss the benefits of deploying a wandering robot, explore the types of in-the-wild studies that can be run with wandering robots, and share pointers for enabling other robots to wander. Our goal is to add wandering to the toolbox of navigation approaches HRI researchers use, particularly to run in-the-wild deployments with mobile robots.

Researchers have proposed models of curiosity as a means to drive robots to learn and adapt to their environments. While these models balance goal- and exploration-oriented actions in a mathematically principled manor, it is not understood how users perceive a robot that pursues off-task actions. Motivated by a model of curiosity based on intrinsic rewards, we conducted three online video-surveys with a total of 264 participants, evaluating a variety of curious behaviors. Our results indicate that a robot's off-task actions are perceived as expressions of curiosity, but that these actions lead to a negative impact on perceptions of the robot's competence. When the robot explains or acknowledges its deviation from the primary task, this can partially mitigate the negative effects of off-task actions.

In this work, we present methods for using human-robot dialog to improve language understanding for a mobile robot agent. The agent parses natural language to underlying semantic meanings and uses robotic sensors to create multi-modal models of perceptual concepts like red and heavy. The agent can be used for showing navigation routes, delivering objects to people, and relocating objects from one location to another. We use dialog clari_cation questions both to understand commands and to generate additional parsing training data. The agent employs opportunistic active learning to select questions about how words relate to objects, improving its understanding of perceptual concepts. We evaluated this agent on Amazon Mechanical Turk. After training on data induced from conversations, the agent reduced the number of dialog questions it asked while receiving higher usability ratings. Additionally, we demonstrated the agent on a robotic platform, where it learned new perceptual concepts on the y while completing a real-world task.

Natural language understanding for robotics can require substantial domain- and platform-specific engineering. For example, for mobile robots to pick-and-place objects in an environment to satisfy human commands, we can specify the language humans use to issue such commands, and connect concept words like red can to physical object properties. One way to alleviate this engineering for a new domain is to enable robots in human environments to adapt dynamically-continually learning new language constructions and perceptual concepts. In this work, we present an end-to-end pipeline for translating natural language commands to discrete robot actions, and use clarification dialogs to jointly improve language parsing and concept grounding. We train and evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we transfer the learned agent to a physical robot platform to demonstrate it in the real world.

Many robotics applications involve navigating to positions specified in terms of their semantic significance. A robot operating in a hotel may need to deliver room service to a named room. In a hospital, it may need to deliver medication to a patient's room. The Building-Wide Intelligence Project at UT Austin has been developing a fleet of autonomous mobile robots, called BWIBots, which perform tasks in the computer science department. Tasks include guiding a person, delivering a message, or bringing an object to a location such as an office, lecture hall, or classroom. The process of constructing a map that a robot can use for navigation has been simplified by modern SLAM algorithms. The attachment of semantics to map data, however, remains a tedious manual process of labeling locations in otherwise automatically generated maps. This paper introduces a system called PRISM to automate a step in this process by enabling a robot to localize door signs - a semantic markup intended to aid the human occupants of a building - and to annotate these locations in its map.

In recent years, research has shown that transfer learning methods can be leveraged to construct curricula that sequence a series of simpler tasks such that performance on a final target task is improved. A major limitation of existing approaches is that such curricula are handcrafted by humans that are typically domain experts. To address this limitation, we introduce a method to generate a curriculum based on task descriptors and a novel metric of transfer potential. Our method automatically generates a curriculum as a directed acyclic graph (as opposed to a linear sequence as done in existing work). Experiments in both discrete and continuous domains show that our method produces curricula that improve the agent's learning performance when compared to the baseline condition of learning on the target task from scratch.