We describe a mobile manipulation hardware and software system capable of autonomously performing complex human-level tasks in real homes, after being taught the task with a single demonstration from a person in virtual reality. This is enabled by a highly capable mobile manipulation robot, whole-body task space hybrid position/force control, teaching of parameterized primitives linked to a robust learned dense visual embeddings representation of the scene, and a task graph of the taught behaviors. We demonstrate the robustness of the approach by presenting results for performing a variety of tasks, under different environmental conditions, in multiple real homes. Our approach achieves 85% overall success rate on three tasks that consist of an average of 45 behaviors each. The video is available at: https://youtu.be/HSyAGMGikLk.

The use of the cognitive capabilties of humans to help guide the autonomy of robotics platforms in what is typically called “supervised-autonomy” is becoming more commonplace in robotics research. The work discussed in this paper presents an approach to a human-in-the-loop mode of robot operation that integrates high level human cognition and commanding with the intelligence and processing power of autonomous systems. Our framework for a “Supervised Remote Robot with Guided Autonomy and Teleoperation” (SURROGATE) is demonstrated on a robotic platform consisting of a pan-tilt perception head, two 7-DOF arms connected by a single 7-DOF torso, mounted on a tracked-wheel base. We present an architecture that allows high-level supervisory commands and intents to be specified by a user that are then interpreted by the robotic system to perform whole body manipulation tasks autonomously. We use a concept of “behaviors” to chain together sequences of “actions” for the robot to perform which is then executed real time.

This paper describes the development of hard- ware, software, and algorithms for a prototype limbed robot capable of surface mobility on small bodies (asteroids and comets). It also describes the development of a laboratory testbed capable of simulating the micro-gravity and terrain of small bodies. A path following algorithm that uses visual odometry, robot body kinematics, and a variety of specialized gaits was used to demonstrate micro-gravity mobility with as few as 12 actuators. A mapping algorithm is also demonstrated that will enable path planning, limb trajectory planning, mobile grasping, and foot placement in future work. The results of this paper demonstrate that robust, stable, and precise small body mobility is feasible with a limbed robot.

This paper addresses the open challenge of planning quasi-static walking motions for robots with kinematically redundant limbs. Focusing on RoboSimian, a quadrupedal robot developed by the Jet Propulsion Laboratory (JPL), we develop a practical method for generating statically stable walking motions by pre-computing a reduced dimensional inverse kinematics (IK) lookup table with certain uniqueness and smoothness properties. We then use that lookup table to generate IK solutions at the beginning and end of walking phases (e.g., swing, body shift, etc), and connect these waypoints using the Rapidly exploring Random Tree Connect (RRT-Connect) algorithm [1]. Thus, we avoid arbitrarily choosing an IK solution at the goal (that may turn out to be difficult to reach from the start) by setting this choice through design and use of a task-specific lookup table, which can be analyzed offline. Our approach also introduces a complementary formulation of the RRT-Connect configuration space that addresses contact and closure constraints by using the forward kinematics of one stance leg to determine the body pose while treating additional stance limbs as dependent on the body pose and solving their inverse kinematics with IK table lookup. We demonstrate an implementation of some of this framework on RoboSimian and discuss generalizations and extensions.

This paper describes the stereo vision near-field terrain mapping system used by the Legged Squad Support System (LS3) quadruped vehicle to automatically adjust its gait in complex natural terrain. The mapping system achieves high robustness with a combination of stereo model-based outlier rejection and spatial and temporal filtering, enabled by a unique hybrid 2D/3D data structure. Classification of sparse structures allows the vehicle to traverse through vegetation. Inference of negative obstacles allows the vehicle to avoid steep drop-offs. A custom designed near-infrared illumination system enables operation at night. The mapping system has been tested extensively with controlled experiments and 72km of field testing in a wide variety of terrains and conditions.

This paper develops an estimation framework for sensor-guided manipulation of a rigid object via a robot arm. Using an unscented Kalman Filter (UKF), the method combines dense range information (from stereo cameras and 3D ranging sensors) as well as visual appearance features and silhouettes of the object and manipulator to track both an object-fixed frame location as well as a manipulator tool or palm frame location. If available, tactile data is also incorporated. By using these different imaging sensors and different imaging properties, we can leverage the advantages of each sensor and each feature type to realize more accurate and robust object and reference frame tracking. The method is demonstrated using the DARPA ARM-S system, consisting of a Barrett™WAM manipulator.

This paper presents a model based approach to autonomous dexterous manipulation, developed as part of the DARPA Autonomous Robotic Manipulation (ARM) program. The developed autonomy system uses robot, object, and environment models to identify and localize objects, and well as plan and execute required manipulation tasks. Deliberate interaction with objects and the environment increases system knowledge about the combined robot and environmental state, enabling high precision tasks such as key insertion to be performed in a consistent framework. This approach has been demonstrated across a wide range of manipulation tasks, and in independent DARPA testing archived the most successfully completed tasks with the fastest average task execution of any evaluated team.

We present a real-time system that enables a highly capable dynamic quadruped robot to maintain an accurate 6-DOF pose estimate (better than 0.5m over every 50m traveled) over long distances traversed through complex, dynamic outdoor terrain, during day and night, in the presence of camera occlusion and saturation, and occasional large external disturbances, such as slips or falls. The system fuses a stereo-camera sensor, inertial measurement units (IMU), and leg odometry with an Extended Kalman Filter (EKF) to ensure robust, low-latency performance. Extensive experimental results obtained from multiple field tests are presented to illustrate the performance and robustness of the system over hours of continuous runs over hundreds of meters of distance traveled in a wide variety of terrains and conditions.

This paper describes a method for learning the terrain classification of long-range appearance data from short- range, stereo-based geometry, along with a map representation for utilizing this data to improve autonomous off-road navigation. The continuous, online learning method allows the system to constantly adapt to changing terrain and environmental conditions, while the polar-perspective map representation allows the system to effectively plan with stereo data at long ranges. Various evaluations of the long-range classification and improvements in system performance are described, including results from an independent third-party testing team.

We introduce a new method for stereo visual SLAM (simultaneous localization and mapping) that works in unstructured, outdoor environments. Unlike other grid-based SLAM algorithms, which use occupancy grid maps, our algorithm uses a new mapping technique that maintains a posterior distribution over the height variance in each cell. This idea was motivated by our experience with outdoor navigation tasks, which has shown height variance to be a useful measure of traversability. To obtain a joint posterior over poses and maps, we use a Rao-Blackwellized particle filter: the pose distribution is estimated using a particle filter, and each particle has its own map that is obtained through exact filtering conditioned on the particle's pose. Visual odometry provides good proposal distributions for the particle pose. In the analytical (exact) filter for the map, we update the sufficient statistics of a gamma distribution over the precision (inverse variance) of heights in each grid cell. We verify the algorithm's accuracy on two outdoor courses by comparing with ground truth data obtained using electronic surveying equipment. In addition, we solve for the optimal transformation from the SLAM map to georeferenced coordinates, based on a noisy GPS signal. We derive an online version of this alignment process, which can be used to maintain a running estimate of the robot's global position that is much more accurate than the GPS readings.

A system that enables continuous slip compensation for a Mars rover has been designed, implemented, and field-tested. This system is composed of several components that allow the rover to accurately and continuously follow a designated path, compensate for slippage, and reach intended goals in high-slip environments. These components include: visual odometry, vehicle kinematics, a Kalman filter pose estimator, and a slip compensation/path follower. Visual odometry tracks distinctive scene features in stereo imagery to estimate rover motion between successively acquired stereo image pairs. The vehicle kinematics for a rocker-bogie suspension system estimates motion by measuring wheel rates, and rocker, bogie, and steering angles. The Kalman filter merges data from an inertial measurement unit (IMU) and visual odometry. This merged estimate is then compared to the kinematic estimate to determine how much slippage has occurred, taking into account estimate uncertainties. If slippage has occurred then a slip vector is calculated by differencing the current Kalman filter estimate from the kinematic estimate. This slip vector is then used to determine the necessary wheel velocities and steering angles to compensate for slip and follow the desired path.

We present an overview of the Coupled Layered Architecture for Robotic Autonomy. CLARAty develops a framework for generic and reusable robotic components that can be adapted to a number of heterogeneous robot platforms. It also provides a framework that will simplify the integration of new technologies and enable the comparison of various elements. CLARAty consists of two distinct layers: a functional layer and a decision layer. The functional layer defines the various abstractions of the system and adapts the abstract components to real or simulated devices. It provides a framework and the algorithms for low- and mid-level autonomy. The decision layer provides the system's high-level autonomy, which reasons about global resources and mission constraints. The decision layer accesses information from the functional layer at multiple levels of granularity. We also present some of the challenges in developing interoperable software for various rover platforms.