We describe a mobile manipulation hardware and software system capable of autonomously performing complex human-level tasks in real homes, after being taught the task with a single demonstration from a person in virtual reality. This is enabled by a highly capable mobile manipulation robot, whole-body task space hybrid position/force control, teaching of parameterized primitives linked to a robust learned dense visual embeddings representation of the scene, and a task graph of the taught behaviors. We demonstrate the robustness of the approach by presenting results for performing a variety of tasks, under different environmental conditions, in multiple real homes. Our approach achieves 85% overall success rate on three tasks that consist of an average of 45 behaviors each. The video is available at: https://youtu.be/HSyAGMGikLk.

Sampling-based motion planners (SBMP) are commonly used to generate motion plans by incrementally constructing a search tree through a robot's configuration space. For high degree-of-freedom systems, sampling is often done in a lower-dimensional space, with a steering function responsible for local planning in the higher-dimensional configuration space. However, for highly-redundant systems with complex kinematics, this approach is problematic due to the high computational cost of evaluating the steering function, especially in cluttered environments. Therefore, having an efficient, informed sampler becomes critical to online robot operation. In this study, we develop a learning-based approach with policy improvement to compute an optimal sampling distribution for use in SBMPs. Motivated by the challenge of whole-body planning for a 31 degree-of-freedom mobile robot built by the Toyota Research Institute, we combine our learning-based approach with classical graph-search to obtain a constrained sampling distribution. Over multiple learning iterations, the algorithm learns a probability distribution weighting areas of low-cost and high probability of success, which a graph search algorithm then uses to obtain an optimal sampling distribution for the robot. On challenging motion planning tasks for the robot, we observe significant computational speed-up, fewer edge evaluations, and more efficient paths with minimal computational overhead. We show the efficacy of our approach with a number of experiments in whole-body motion planning.

The use of the cognitive capabilties of humans to help guide the autonomy of robotics platforms in what is typically called “supervised-autonomy” is becoming more commonplace in robotics research. The work discussed in this paper presents an approach to a human-in-the-loop mode of robot operation that integrates high level human cognition and commanding with the intelligence and processing power of autonomous systems. Our framework for a “Supervised Remote Robot with Guided Autonomy and Teleoperation” (SURROGATE) is demonstrated on a robotic platform consisting of a pan-tilt perception head, two 7-DOF arms connected by a single 7-DOF torso, mounted on a tracked-wheel base. We present an architecture that allows high-level supervisory commands and intents to be specified by a user that are then interpreted by the robotic system to perform whole body manipulation tasks autonomously. We use a concept of “behaviors” to chain together sequences of “actions” for the robot to perform which is then executed real time.

This paper considers how a multi-limbed robot can carry out manipulation tasks involving simultaneous and compatible end-effector velocity and force goals, while also maintaining quasi-static stance stability. The formulation marries a local optimization process with an assumption of a compliant model of the environment. For purposes of illustration, we first develop the formulation for a single fixed based manipulator arm. Some of the basic kinematic variables we previously introduced for multi-limbed robot mechanism analysis in [1] are extended to accomodate this new formulation. Using these extensions, we provide a novel definition for static equilibrium of multi-limbed robot with actuator limits, and provide general conditions that guarantee the ability to apply arbitrary end-effector forces. Using these extended definitions, we present the local optimization problem and its solution for combined manipulation and stance. We also develop, using the theory of strong alternatives, a new definition and a computable test for quasi-static stance feasibility in the presence of manipulation forces. Simulations illustrate the concepts and method.

This paper provides kinematic analysis and local motion planning methods for multi-limbed robots. In particular, we consider combined stance and reach tasks for robotic mechanisms whose limbs can be used either as legs or manipulator arms. An example of such a system is the RoboSimian robot participating in the DARPA Robotics Challenge (Figure 1). We develop relationships which model the key quasi-statics and kinematics of these mechanisms: the stance map, the stance Jacobian, and the reach Jacobian, as well as the stance constrained center-of-mass Jacobian. We also introduce characterizations of multi-limbed mechanism configurations in terms of the properties of these maps: local dexterity and limberness. This paper also introduces local planning methods which seek to balance the motion of legs, body, and arms of such mechanisms so as to realize manipulation goals while also maintaining awareness of stance stability issues. Examples with a simple planar model illustrate the methods.

This paper considers motion planning and control problems that are motivated by the design of tethered, extreme terrain robots. We abstract the mobility structure of these systems using a tethered differential drive robot with rimless wheels. We analyze several important issues related to this geometry. First it is shown that this vehicle cannot be modeled deterministically unless an additional degree of freedom relative to the standard differential drive vehicle is provided. The simplest kinematically consistent model is one that allows for slight prismatic motion of the axle, approximating the effects of wheel slip. We show that under mild assumptions, such a vehicle's reachable set is dense in SE(2), implying local maneuverability. Next we study some of the constraints which the tether places on the vehicle's motions and derive scaling laws relating wheel and vehicle speeds. Using these results, we provide simple planning and approximate path-following methods that allow tether management. In particular, we consider trajectories produced by solving an optimal control problem to minimize the integral of absolute tether-reeling rate.

Affordable test articles for descent and landing are crucial for developing commercial lunar landing capability. To ensure successful lunar landing, flight software must be tested over mission-length durations on hardware exhibiting dynamics analogous to those of true flight articles. Energetic and structural constraints typically preclude affordable long-duration lander tests.