Despite the availability of increasingly more robust machine perception and control algorithms, human operators are still needed to directly control robotic systems in extreme environments where there is a high cost of failure. This paper reports a novel architecture for semi-autonomous teleoperation over high-latency telemetry and the results of a 15-subject user study used to evaluate that architecture. The intent-recognition-based traded control (IRTC) architecture recognizes operator intent based on remote scene and task modeling, and automatically trades control to an autonomous reactive subsystem. Each subject participated in one two-hour session during which they controlled a single simulated redundant robotic manipulator with 4000ms of closed-loop latency. Subjects continuously assemble and disassemble a 3D lattice structure with or without IRTC-based assistance. The user study showed that the IRTC system improved operator performance in the simulated multi-object assembly task. All source code for conducting the experiment including the simulation and data processing is available online under a permissive open-source license.

We report the development and a preliminary multi-user evaluation of an assisted teleoperation architecture for assembly tasks over high-latency networks. While such tasks still require human insight, there are often elements of these tasks which can be executed autonomously. Our architecture contextualizes a user's intended actions in a remote scene. It assists the user by performing the intended action more precisely based on the latest local scene model. We report the results of a multi-user study of nine participants to evaluate performance of this approach using a full-3D dynamic robotic simulation in a laboratory environment. The study required users to assemble part of a structure with 200 or 4000 milliseconds of time delay, with or without assistance from the reported system. For each condition, we evaluated performance based both on partial and final task completion times, and we estimated each user's workload with the NASA Task Load Index. The study showed that our architecture has the potential to increase feasibility of the task both with and without large telemetry delay and to enable users to complete elements of the task more rapidly. We also found that the assistance dramatically reduced the users' workload. We provide open-source implementations for all components of our system, which can be adapted to other robotic platforms.

Densely cluttered scenes are composed of multiple objects which are in close contact and heavily occlude each other. Few existing 3D object recognition systems are capable of accurately predicting object poses in such scenarios. This is mainly due to the presence of objects with textureless surfaces, similar appearances and the difficulty of object instance segmentation. In this paper, we present a hierarchical semantic segmentation algorithm which partitions a densely cluttered scene into different object regions. A RANSAC-based registration method is subsequently applied to estimate 6-DoF object poses within each object class. Part of this algorithm includes a generalized pooling scheme used to construct robust and discriminative object representations from a convolutional architecture with multiple pooling domains. We also provide a new RGB-D dataset which serves as a benchmark for object pose estimation in densely cluttered scenes. This dataset contains five thousand scene frames and over twenty thousand labeled poses of ten common hand tools. We show that our method demonstrates improved performance of pose estimation on this new dataset compared with other state-of-the-art methods.

Flexible interaction and instruction is a key enabling technology for expanding robotics into small to medium scale manufacturing, in-home assistance for physically disabled individuals, and robotic surgery. In these cases, performing a task manually is neither practical nor scalable, yet complete automation is cost-prohibitive or impossible. Thus, our interest is in collaborative systems that can be easily trained to work with an operator. This collaborative robotic system should be instructable in a generalizable way for a wide range of tasks, and should generalize to new tasks gracefully with minimal retraining. At the same time, for a given task, the system should take advantage of user interaction modalities needed to accomplish the task, subject to the constraints of the available interfaces. These ideas motivate the Adjutant framework. Adjutant supports human-robot collaborative operations for ranges of user roles and robot capability. Adjutant models human-robot systems via sets of robot capabilities, composable high-level functions that can be specialized to specific tasks, and collaborative behaviors which relate these capabilities to specific user interfaces or interaction paradigms. Adjutant also incorporates several methods encapsulating reusable task information into capabilities, thus specializing them, including tool affordances, perceptual grounding templates, and tool movement primitives. We have implemented Adjutant as a software framework in ROS and, in this paper, explore the utility of Adjutant for performing several real-world collaborative manufacturing tasks on an industrial robot test-bed.

In this paper we present an approach to extending the capabilities of telemanipulation systems by intelligently augmenting a human operator's motion commands based on quantitative three-dimensional scene perception at the remote telemanipulation site. This framework is the first prototype of the Augmented Shared-Control for Efficient, Natural Telemanipulation (ASCENT) System. ASCENT aims to enable new robotic applications in environments where task complexity precludes autonomous execution or where low-bandwidth and/or high-latency communication channels exist between the nearest human operator and the application site. These constraints can constrain the domain of telemanipulation to simple or static environments, reduce the effectiveness of telemanipulation, and even preclude remote intervention entirely. ASCENT is a semi-autonomous framework that increases the speed and accuracy of a human operator's actions via seamless transitions between one-to-one teleoperation and autonomous interventions. We report the promising results of a pilot study validating ASCENT in a transatlantic telemanipulation experiment between The Johns Hopkins University in Baltimore, MD, USA and the German Aerospace Center (DLR) in Oberpfaffenhofen, Germany. In these experiments, we observed average telemetry delays of 200ms, and average video delays of 2s with peaks of up to 6s for all data. We also observed 75% frame loss for video streams due to bandwidth limits, giving 4fps video.

This paper reports the design requirements, practical challenges, and a preliminary design for a magnetic resonance imaging (MRI) guided, three degree-of-freedom (DOF) transrectal prostate intervention robot. We show the operational space constraints imposed by patient anatomy when performing transrectal prostate procedures in a magnetic resonance (MR) scanner bore, as determined by analyzing data from 12 patient procedures with a device. We also describe practical challenges arising in designing a compact actuated MR compatible needle placement robot for MRI-guided transrectal needle intervention in the prostate. We present a preliminary design which aims to improve upon previous un-actuated and partially-actuated devices with the addition of an actuated needle insertion module. Such an enhancement enables needle driving to take place inside the MR scanner bore and thereby may reduce the overall procedure time - thus improving patient comfort and reducing likelyhood of needle targeting errors resulting from patient motion. We show that it is feasible to add such actuation while reducing the footprint of the device in accordance with the anatomical and MR scanner constraints and practical design requirements.

As autonomous personal robots come of age, we expect certain applications to be executed with a high degree of repeatability and robustness. In order to explore these applications and their challenges, we need tools and strategies that allow us to develop them rapidly. Serving drinks (i.e., locating, fetching, and delivering), is one such application with well-defined environments for operation, requirements for human interfacing, and metrics for successful completion. In this paper we present our experiences and results while building an autonomous robotic assistant using the PR21platform and ROS2. The system integrates several new components that are built on top of the PR2's current capabilities. Perception components include dynamic obstacle identification, mechanisms for identifying the refrigerator, types of drinks, and human faces. Planning components include navigation, arm motion planning with goal and path constraints, and grasping modules. One of the main contributions of this paper is a new task-level executive system, SMACH, based on hierarchical concurrent state machines, which controls the overall behavior of the system. We provide in-depth discussions on the solutions that we found in accomplishing our goal, and the implementation strategies that let us achieve them.