Tangram assembly, the art of human intelligence and manipulation dexterity, is a new challenge for robotics and reveals the limitations of state-of-the-arts. Here, we describe our initial exploration and highlight key problems in reasoning, planning, and manipulation for robotic tangram assembly. We present MRChaos (Master Rules from Chaos), a robust and general solution for learning assembly policies that can generalize to novel objects. In contrast to conventional methods based on prior geometric and kinematic models, MRChaos learns to assemble randomly generated objects through self-exploration in simulation without prior experience in assembling target objects. The reward signal is obtained from the visual observation change without manually designed models or annotations. MRChaos retains its robustness in assembling various novel tangram objects that have never been encountered during training, with only silhouette prompts. We show the potential of MRChaos in wider applications such as cutlery combinations. The presented work indicates that radical generalization in robotic assembly can be achieved by learning in much simpler domains. The code will be available https://robotll.github.io/MasterRulesFromChaos/.

This paper tackles the task of singulating and grasping paper-like deformable objects. We refer to such tasks as paper-flipping. In contrast to manipulating deformable objects that lack compression strength (such as shirts and ropes), minor variations in the physical properties of the paper-like deformable objects significantly impact the results, making manipulation highly challenging. Here, we present Flipbot, a novel solution for flipping paper-like deformable objects. Flipbot allows the robot to capture object physical properties by integrating exteroceptive and proprioceptive perceptions that are indispensable for manipulating deformable objects. Furthermore, by incorporating a proposed coarse-to-fine exploration process, the system is capable of learning the optimal control parameters for effective paper-flipping through proprioceptive and exteroceptive inputs. We deploy our method on a real-world robot with a soft gripper and learn in a self-supervised manner. The resulting policy demonstrates the effectiveness of Flipbot on paper-flipping tasks with various settings beyond the reach of prior studies, including but not limited to flipping pages throughout a book and emptying paper sheets in a box. The code is available here: https://robotll.github.io/Flipbot/.

We present a data-driven approach for effective bin picking from clutter. Recent bin picking solutions usually lead to a direct pinch grasp on a target object without addressing any other potential contact interaction in clutter. However, appropriate physical interaction can be essential to successful singulation and subsequent secure picking, the goal of bin picking. In this work, we contribute a framework that learns physically interactive actions for object picking end-to-end from a visual input in a self-supervised manner. The learned actions enable the robot to purposefully interact with a target object by performing a digging operation through the clutter. By leveraging a fully convolutional network (FCN), we predict picking success probabilities for a set of interactive action primitives that will in turn specify an optimal action to perform. The FCN is trained in a simulated environment through trial and error. Moreover, new datasets are collected using the latest network through iterative self-supervision. Extensive real-world bin picking experiments show the effectiveness and generalizability of the approach.

Bin picking is a challenging problem in robotics due to high dimensional action space, partially visible objects, and contact-rich environments. State-of-the-art methods for bin picking are often simplified as planar manipulation, or learn policy based on human demonstration and motion primitives. The designs have escalated in complexity while still failing to reach the generality and robustness of human picking ability. Here, we present an end-to-end reinforcement learning (RL) framework to produce an adaptable and robust policy for picking objects in diverse real-world environments, including but not limited to tilted bins and corner objects. We present a novel solution to incorporate object interaction in policy learning. The object interaction is represented by the poses of objects. The policy learning is based on two neural networks with asymmetric state inputs. One acts on the object interaction information, while the other acts on the depth observation and proprioceptive signals of robots. The results of experiment shows remarkable zero-shot generalization from simulation to the real world and extensive real-world experiments show the effectiveness of the approach.