We present a novel approach to robotic grasp planning using both a learned grasp proposal network and a learned 3D shape reconstruction network. Our system generates 6-DOF grasps from a single RGB-D image of the target object, which is provided as input to both networks. By using the geometric reconstruction to refine the candidate grasp produced by the grasp proposal network, our system is able to accurately grasp both known and unknown objects, even when the grasp location on the object is not visible in the input image.This paper presents the network architectures, training procedures, and grasp refinement method that comprise our system. Experiments demonstrate the efficacy of our system at grasping both known and unknown objects (91% success rate in a physical robot environment, 84% success rate in a simulated environment). We additionally perform ablation studies that show the benefits of combining a learned grasp proposal with geometric reconstruction for grasping, and also show that our system outperforms several baselines in a grasping task.

We present a novel method enabling robots to quickly learn to manipulate objects by leveraging a motion planner to generate “expert” training trajectories from a small amount of human-labeled data. In contrast to the traditional sense-plan-act cycle, we propose a deep learning architecture and training regimen called PtPNet that can estimate effective end-effector trajectories for manipulation directly from a single RGB-D image of an object. Additionally, we present a data collection and augmentation pipeline that enables the automatic generation of large numbers (millions) of training image and trajectory examples with almost no human labeling effort.We demonstrate our approach in a non-prehensile tool-based manipulation task, specifically picking up shoes with a hook. In hardware experiments, PtPNet generates motion plans (open-loop trajectories) that reliably (89% success over 189 trials) pick up four very different shoes from a range of positions and orientations, and reliably picks up a shoe it has never seen before. Compared with a traditional sense-plan-act paradigm, our system has the advantages of operating on sparse information (single RGB-D frame), producing high-quality trajectories much faster than the expert planner (300ms versus several seconds), and generalizing effectively to previously unseen shoes. Video available at https://youtu.be/voIkyiBtwn4.

We present a system enabling a modular robot to autonomously build structures in order to accomplish high-level tasks. Building structures allows the robot to surmount large obstacles, expanding the set of tasks it can perform. This addresses a common weakness of modular robot systems, which often struggle to traverse large obstacles. This paper presents the hardware, perception, and planning tools that comprise our system. An environment characterization algorithm identifies features in the environment that can be augmented to create a path between two disconnected regions of the environment. Specially-designed building blocks enable the robot to create structures that can augment the environment to make obstacles traversable. A high-level planner reasons about the task, robot locomotion capabilities, and environment to decide if and where to augment the environment in order to perform the desired task. We validate our system in hardware experiments.

The PaintPot manufacturing process is a new way to create low-cost, low-profile, highly customizable potentiometers for position sensing in robotic applications. It uses widely accessible materials, requires no special expertise, and creates custom potentiometers in a variety of shapes and sizes, including curved surfaces. PaintPots offer accuracy and precision performance comparable with commercial (non-customizable) options through a calibration process that trades small computation for cost. This paper includes detailed PaintPot manufacturing and calibration processes, and experiments that validate the accuracy, precision, and lifetime performance of PaintPots, comparable to commercial sensors. We also provide a case-study application in the SMORES-EP modular robot, and show how the PaintPot process can be used to create resistive surfaces capable of sensing position in 2D on planes and spheres.

The advantage of modular robot systems lies in their flexibility, but this advantage can only be realized if there exists some reliable, effective way of generating configurations (shapes) and behaviors (controlling programs) appropriate for a given task. In this paper, we present an end-to-end system for addressing tasks with modular robots, and demonstrate that it is capable of accomplishing challenging multi-part tasks in hardware experiments. The system consists of four tightly integrated components: (1) A high-level mission planner, (2) A design library spanning a wide set of functionality, (3) A design and simulation tool for populating the library with new configurations and behaviors, and (4) Modular robot hardware. This paper condenses the material originally presented in Jing et al. 2016 into a shorter format suitable for a broad audience.

We present the EP-Face connector, a novel connector for hybrid chain-lattice type modular robots that is highstrength (88.4N), compact, fast, power efficient, and robust to position errors. The connector consists of an array of electro-permanent magnets (EP magnets) embedded in a planar face. EP magnets are solid-state magnets that can be turned on and off and require power only when changing state. In this paper, we present the design of the connector, manufacturing process, detailed experimental characterization of the connector strength under different loading conditions, and compare its performance to existing magnetic and mechanical connectors. We also illustrate the functional benefits of the EPFace by demonstrating reconfiguration with the SMORES-EP robot.

We address the problem of detecting embeddability of modular robots: namely, to decide automatically whether a given modular robot design can simulate the functionality of a seemingly different design. To that end, we introduce a novel graph representation for modular robots and formalize the notion of embedding through topological and kinematic conditions. Based on that, we develop an algorithm that decides embeddability when the two involved designs have tree topologies. Our algorithm performs two passes and involves dynamic programming and maximum cardinality matching. We demonstrate our approach on real modular robots and show that we can detect embeddability of complex designs efficiently.

We present the methodology, algorithms, system design, and experiments addressing the self-assembly of large teams of autonomous robotic boats into floating platforms. Identical self-propelled robotic boats autonomously dock together and form connected structures with controllable variable stiffness. These structures can self-reconfigure into arbitrary shapes limited only by the number of rectangular elements assembled in brick-like patterns. An O(m2) complexity algorithm automatically generates assembly plans which maximize opportunities for parallelism while constructing operator-specified target configurations with m components. The system further features an O(n3) complexity algorithm for the concurrent assignment and planning of trajectories from n free robots to the growing structure. Such peer-to-peer assembly among modular robots compares favorably to a single active element assembling passive components in terms of both construction rate and potential robustness through redundancy. We describe hardware and software techniques to facilitate reliable docking of elements in the presence of estimation and actuation errors, and we consider how these local variable stiffness connections may be used to control the structural properties of the larger assembly. Assembly experiments validate these ideas in a fleet of 0.5 m long modular robotic boats with onboard thrusters, active connectors, and embedded computers.

This paper addresses the self-assembly of a large team of autonomous boats into floating platforms. We describe the design of individual boats, the systems concept, the algorithms, the software architecture and experimental results with prototypes that are 1:12 scale realizations of modified ISO shipping containers, with the goal of demonstrating self-assembly into large maritime structures such as air strips, bridges, harbors or sea bases. Each container is a robotic module capable of holonomic motion that can dock in a brick pattern to form arbitrary shapes. Over 60 modules were built of varying capability. The docking mechanism is designed to be robust to large disturbances that can be expected in the high seas. The docking mechanism also incorporates adjustable stiffness so that the conglomerate can comply to waves representative of sea state three, and have the ability to dynamically stiffen as required. The component modules for autonomous assembly, docking and simultaneous collision-free planning as well as the software architecture are presented along with the description of experimental verification.