It is important to enable a robot to manipulate a target object that has no 3-D model information and is situated in an environment with other unknown objects nearby. This poses an open problem of how to combine perception and manipulation to enable the robot to build an appearance-based model of the target object on the spot to facilitate further manipulation of the object while avoiding the other unknown obstacles in the way. In this paper, we introduce an approach to enable a continuum manipulator, which is apt to maneuver through a crowded environment, to gradually build a 3-D surface model of the target object by moving an RGB-D sensor around the object while also detecting and avoiding surrounding unknown obstacles. Our approach interleaves perception and manipulation such that perception guides the manipulator movement, which in turn allows more perception of the target object for object model building and further manipulator motion. Our approach is characterized by a progressive strategy to register RGB-D images of the target object to build and extend a partial model of the object and the corresponding motion planning strategy for the continuum robot to carry out model building and avoid obstacles at the same time. To demonstrate the effectiveness of our approach, experiments on progressive model building of real objects from real RGB-D images are conducted, where a simulated continuum robot plans and executes its motion to carry the RGB-D camera around a target object for taking those images in an augmented reality setting.

In this paper, we propose a novel approach for both 3-D object detection and six-degree-of-freedom (6-DoF) pose estimation based on smooth 3-D surface segments and their visual signatures in cluttered scenes with objects partially occluded. It is an appearance-based approach that does not require precise geometric models of the objects. We introduce a robust and flexible strategy to build object models based on segmented smooth surfaces of RGB-D images. We next introduce a strategy to detect objects and estimate their 6-DoF poses from a single image by taking full advantage of the surface-based object models. Our approach can reconstruct objects in cluttered scenes from a single view, where different objects or multiple instances of the same object can be stacked together or physically attached and occlude one another. It detects objects and estimates their 6-DoF poses even if the objects have transparent regions. Experimental results and comparison with related work demonstrate the effectiveness of our approach.

It is important to understand which view is better recognizing and reconstructing a scene for many robotic applications, especially in a cluttered environment, where objects interact and may occlude one another in all views. In this paper, we introduce a novel, learning-based approach to evaluate scene recognizability from a view based on the quality and quantity of recognized objects, the recognition uncertainty, and the background recognizability, rather than the visibility. Our study shows that increasing visibility does not guarantee better recognizability of objects. The introduced view evaluator can better characterize which view is more useful for the purpose of autonomous object recognition and scene reconstruction. The approach is validated through experiments, and the effects of many factors to scene recognizability are discussed based on the experimental results.

3D object detection and pose estimation often requires a 3D object model, and even so, it is a difficult problem if the object is heavily occluded in a cluttered scene. In this paper, we introduce a novel approach for recognizing and localizing 3D objects based on their appearances through segmentation of 3D surfaces. The approach can identify multiple occluded objects in a scene, which may include different instances of the same object, and estimate the pose of each entire object even if the object can only be seen partially due to occlusion.

A continuum manipulator, such as a multi-section trunk/tentacle robot, is promising for deft manipulation of a wide range of objects of different shapes and sizes. Given an object, a continuum manipulator tries to grasp it by wrapping tightly around it. Autonomous grasping requires realtime determination of whether an object can be grasped after it is identified, and if so, the feasible whole-arm wrapping around configurations of the robot to grasp it, which we call grasping configurations, as well as the path leading to a grasping configuration. In this paper, we describe the process for autonomous grasping from object detection to executing the grasping motion and achieving force-closure grasps, with a focus on a general analysis of all possible types of planar grasping configurations of a three-section continuum manipulator. We further provide conditions for existence of solutions and describe how to find a valid grasping configuration and the associated path automatically if one exists. Experimental results with the OctArm manipulator validate our approach, and shows that the entire process to determine an autonomous grasping operation, which includes automatic detection of the target object and determination of a grasping configuration and a path to the grasping configuration that avoids obstacles, can take just a small fraction of a second. Once a grasping configuration is reached, the manipulator can lift the object stably, i.e., a force-closure grasp can be achieved.