We describe a mobile manipulation hardware and software system capable of autonomously performing complex human-level tasks in real homes, after being taught the task with a single demonstration from a person in virtual reality. This is enabled by a highly capable mobile manipulation robot, whole-body task space hybrid position/force control, teaching of parameterized primitives linked to a robust learned dense visual embeddings representation of the scene, and a task graph of the taught behaviors. We demonstrate the robustness of the approach by presenting results for performing a variety of tasks, under different environmental conditions, in multiple real homes. Our approach achieves 85% overall success rate on three tasks that consist of an average of 45 behaviors each. The video is available at: https://youtu.be/HSyAGMGikLk.

We present an approach to navigating a biped robot safely and efficiently through a complicated environment of previously unknown obstacles and terrain using only on-board sensing and odometry. Sensing of the environment is performed by a pivoting laser scanner, which continues to update the terrain representation as the robot walks. Safe stepping motions are planned from this data to follow the user's command, given in the form of an end goal, a rough path, or a joystick input. Results are demonstrated on a prototype robot in several environments.

This paper describes an online 3D reconstruction system from stereo image sequences to obtain a dense local world model for robot navigation. The proposed method consists of three components: 1) stereo depth map calculation, 2) correspondence calculation in time sequential images by tracking raw image features, 3) 6DOF camera motion estimation by RANSAC and integrate depth map into 3D reconstructed model. We examined and evaluated our method in a motion capture environment for comparison. Finally experimental results of a humanoid robot H7 are denoted.

This paper describes an online system for footstep planning using a 3D map reconstructed by visual odometry. This system consists of two key components: 3D reconstruction via visual odometry from a stereo image sequence to obtain a dense local world model, and a footstep planner for biped robots using the reconstructed 3D map. Visual odometry is a method to connect 3D image sequences to obtain 6DOF camera motion and dense 3D environment information. The method described in this paper consists of three components: stereo depth map calculation, 3D flow calculation from tracking raw image features, and 6DOF camera motion estimation from RANSAC. Using the resulting 3D data, an optimal sequence of footstep locations is planned. The footstep planner is provided a height map of the terrain and a discrete set of possible footstep motions. The planner then evaluates footstep locations for viability using a collection of heuristic metrics designed to encode the relative safety, effort required, and overall motion complexity. Finally, we implemented this system on the humanoid robot H7. A local 3D map is reconstructed using visual odometry at about 10 Hz and the footstep planner replans at intervals of four steps. The robot walked across a floor, avoiding obstacles and reaching the goal.