Assembling parts into an object is a combinatorial problem that arises in a variety of contexts in the real world and involves numerous applications in science and engineering. Previous related work tackles limited cases with identical unit parts or jigsaw-style parts of textured shapes, which greatly mitigate combinatorial challenges of the problem. In this work, we introduce the more challenging problem of shape assembly, which involves textureless fragments of arbitrary shapes with indistinctive junctions, and then propose a learning-based approach to solving it. We demonstrate the effectiveness on shape assembly tasks with various scenarios, including the ones with abnormal fragments (e.g., missing and distorted), the different number of fragments, and different rotation discretization.

We propose a tree search-based planning algorithm for a robot manipulator to rearrange objects and grasp a target in a dense space. We consider environments where tasks cannot be completed with prehensile planning only. As assuming that a manipulator is only allowed to grasp from the top, we aim to minimize the number of rearrangement actions and the total execution time, which affects the efficiency of manipulation. The proposed search algorithm determines the optimal sequence of object rearrangement with prehensile and non-prehensile grasping until grasping a target. For non-prehensile grasping, a heuristic function is employed to model frictions and contacts between objects and a table. Experimental results in a realistic simulated environment show that the proposed algorithm can reduce the number of rearranged obstacles up to 27% and the total execution time up to 15% with 14 objects compared to the previous work.

Discovering a solution in a combinatorial space is prevalent in many real-world problems but it is also challenging due to diverse complex constraints and the vast number of possible combinations. To address such a problem, we introduce a novel formulation, combinatorial construction, which requires a building agent to assemble unit primitives (i.e., LEGO bricks) sequentially -- every connection between two bricks must follow a fixed rule, while no bricks mutually overlap. To construct a target object, we provide incomplete knowledge about the desired target (i.e., 2D images) instead of exact and explicit volumetric information to the agent. This problem requires a comprehensive understanding of partial information and long-term planning to append a brick sequentially, which leads us to employ reinforcement learning. The approach has to consider a variable-sized action space where a large number of invalid actions, which would cause overlap between bricks, exist. To resolve these issues, our model, dubbed Brick-by-Brick, adopts an action validity prediction network that efficiently filters invalid actions for an actor-critic network. We demonstrate that the proposed method successfully learns to construct an unseen object conditioned on a single image or multiple views of a target object.

In this article, we present a task and motion planning method for retrieving a target object from clutter using a robotic manipulator. We consider dense and cluttered environments where some objects must be removed in order to retrieve the target without collisions. To ensure a successful execution, the interplay between task planning (what to remove in what order) and motion planning (how to remove) is crucial. Thus, the task and motion planning approach combining a symbolic task planner and a geometric motion planner becomes one of the major paradigms in manipulation planning. However, motion planning in dense clutter often leads to frequent failures, so repetitive task replanning is inevitable. Although symbolic task planners are general and domain-independent, they do not scale; so we need an efficient task planner specialized for dense clutter for fast completion of tasks. We propose a polynomial-time task planner for object manipulation in clutter that can be combined with any motion planner. We aim to optimize the number of pick-and-place actions which often determines the efficiency of object manipulation tasks. We consider common situations that could occur in clutter: 1) all object locations are known, 2) some hidden objects are revealed while relocating some front objects, and 3) the target is hidden until some objects are removed. Our method is shown to reduce the number of pick-and-place actions compared to baseline methods (e.g., at least 28.0% of reduction in a known static environment with 20 objects). We also deploy the proposed method to two physical robots with vision systems to show that our method can solve real-world problems.

We present an algorithm determining where to relocate objects inside a cluttered and confined space while rearranging objects to retrieve a target object. Although methods that decide what to remove have been proposed, planning for the placement of removed objects inside a workspace has not received much attention. Rather, removed objects are often placed outside the workspace, which incurs additional laborious work (e.g., motion planning and execution of the manipulator and the mobile base, perception of other areas). Some other methods manipulate objects only inside the workspace but without a principle so the rearrangement becomes inefficient.In this work, we consider both monotone (each object is moved only once) and non-monotone arrangement problems which have shown to be $\mathcal{N}\mathcal{P}$-hard. Once the sequence of objects to be relocated is given by any existing algorithm, our method aims to minimize the number of pick-and-place actions to place the objects until the target becomes accessible. From extensive experiments, we show that our method reduces the number of pick-and-place actions and the total execution time (the reduction is up to 23.1% and 28.1% respectively) compared to baseline methods while achieving higher success rates.

This paper presents a task and motion planning (TAMP) framework for a robotic manipulator in order to retrieve a target object from clutter. We consider a configuration of objects in a confined space with a high density so no collision-free path to the target exists. The robot must relocate some objects to retrieve the target without collisions. For fast completion of object rearrangement, the robot aims to optimize the number of pick-and-place actions which often determines the efficiency of a TAMP framework.We propose a task planner incorporating motion planning to generate executable plans which aims to minimize the number of pick-and-place actions. In addition to fully known and static environments, our method can deal with uncertain and dynamic situations incurred by occluded views. Our method is shown to reduce the number of pick-and-place actions compared to baseline methods (e.g., at least 28.0% of reduction in a known static environment with 20 objects).

We present an algorithm that produces a plan for relocating obstacles in order to grasp a target in clutter by a robotic manipulator without collisions. We consider configurations where objects are densely populated in a constrained and confined space. Thus, there exists no collision-free path for the manipulator without relocating obstacles. Since the problem of planning for object rearrangement has shown to be NP-hard, it is difficult to perform manipulation tasks efficiently which could frequently happen in service domains (e.g., taking out a target from a shelf or a fridge). Our proposed planner employs a collision avoidance scheme which has been widely used in mobile robot navigation. The planner determines an obstacle to be removed quickly in real time. It also can deal with dynamic changes in the configuration (e.g., changes in object poses). Our method is shown to be complete and runs in polynomial time. Experimental results in a realistic simulated environment show that our method improves up to 31% of the execution time compared to other competitors.