We address the problem of clearing a pile of unknown objects using an autonomous interactive perception approach. Our robot hypothesizes the boundaries of objects in a pile of unknown objects (object segmentation) and verifies its hypotheses (object detection) using deliberate interactions. To guarantee the safety of the robot and the environment, we use compliant motion primitives for poking and grasping. Every verified segmentation hypothesis can be used to parameterize a compliant controller for manipulation or grasping. The robot alternates between poking actions to verify its segmentation and grasping actions to remove objects from the pile. We demonstrate our method with a robotic manipulator. We evaluate our approach with real-world experiments of clearing cluttered scenes composed of unknown objects.

We present an interactive perceptual skill for segmenting, tracking, and modeling the kinematic structure of 3D articulated objects. This skill is a prerequisite for general manipulation in unstructured environments. Robot-environment interactions are used to move an unknown object, creating a perceptual signal that reveals the kinematic properties of the object. The resulting perceptual information can then inform and facilitate further manipulation. The algorithm is computationally efficient, handles partial occlusions, and depends on little object motion; it only requires sufficient texture for visual feature tracking. We conducted experiments with everyday objects on a robotic manipulation platform equipped with an RGB-D sensor. The results demonstrate the robustness of the proposed method to lighting conditions, object appearance, size, structure, and configuration.

We incorporate a randomized kinodynamic path planning approach with image-based control of a robotic arm equipped with an in-hand camera. The proposed approach yields continuously differentiable camera trajectories by taking camera dynamics into account, while accounting for a critical set of image and physical constraints at the planning stage. The proposed planner explores the camera state space for permissible trajectories by iteratively extending a search tree in this space and simultaneously tracking these trajectories in the robot configuration space. The planned camera trajectories are projected into the image space to obtain desired feature trajectories which are then tracked using an image-based visual servoing scheme. We validate the effectiveness of the proposed framework in incorporating the aforementioned constraints through a number of visual servoing experiments on a six-degree-of-freedom robotic arm. We also provide empirical results that demonstrate its performance in the presence of uncertainties, and accordingly suggest additional planning strategies to increase robustness with respect to possible deviations from planned trajectories.

We describe the software components of a robotics system designed to autonomously grasp objects and perform dexterous manipulation tasks with only high-level supervision. The system is centered on the tight integration of several core functionalities, including perception, planning and control, with the logical structuring of tasks driven by a Behavior Tree architecture. The advantage of the implementation is to reduce the execution time while integrating advanced algorithms for autonomous manipulation. We describe our approach to 3-D perception, real-time planning, force compliant motions, and audio processing. Performance results for object grasping and complex manipulation tasks of in-house tests and of an independent evaluation team are presented.

We address the problem of incorporating path planning with image-based control of a wheeled mobile manipulator (WMM) performing visually-guided tasks in complex environments. The WMM consists of a wheeled (non-holonomic) mobile platform and an on-board robotic arm equipped with a camera mounted at its end-effector. The visually-guided task is to move the WMM from an initial to a desired location while respecting image and physical constraints. We propose a kinodynamic planning approach that explores the camera state space for permissible trajectories by iteratively extending a search tree in this space and simultaneously tracking these trajectories in the WMM configuration space. We utilize weighted pseudo-inverse Jacobian solutions combined with a null space optimization technique to effectively coordinate the motion of the mobile platform and the arm. We also present the preliminary results obtained by executing the planned trajectories on a real WMM system via a decoupled control scheme where the on-board arm is servo controlled along the planned feature trajectories while the mobile platform is simultaneously controlled along its trajectory using a state feedback tracking method.

In this paper we incorporate a randomized kinodynamic path planning approach with image-based control for a robotic arm equipped with an in-hand camera in a servoing task. The proposed approach yields C2-smooth camera trajectories by taking camera dynamics into account while accounting for a critical set of image and physical constraints. The proposed planner explores the camera state space (i.e., a space of camera poses and velocities) for permissible trajectories by iteratively extending a search tree in this space and simultaneously tracking these trajectories in the robot configuration space (i.e., joint space). The planned camera trajectories are then projected into the image space to obtain desired feature trajectories. In the execution stage an image-based visual servoing scheme is then adopted to track the feature trajectories. The effectiveness of the proposed approach has been experimentally demonstrated on a robotic arm with an in-hand camera executing servoing tasks in complex environments.

We incorporate sampling-based global path planning with Visual Servoing (VS) for a robotic arm equipped with an in-hand camera. The path planning accounts for a number of constraints: 1) maintaining continuous visibility of the target within the camera's field of view, 2) avoiding visual occlusion of target features caused by the workspace obstacles, robot's body, or the target itself, 3) avoiding collision with physical obstacles or self collision, and 4) joint limits. Incorporating these constraints enhances the applicability of VS to significantly more complex environments/tasks, thereby making the resulting VS much more robust. The proposed planner explores the camera space, i.e. 3D Cartesian space, for permissible camera paths satisfying the aforementioned constraints by iteratively extending a search tree in camera space and simultaneously tracking these paths in the robot's joint space using a local planner. The planned camera path is then projected into the image space and tracked using an image-based visual servoing scheme. The validity and effectiveness of the proposed approach in accomplishing VS tasks in complex environments are demonstrated through a number of simulations on a 6-dof robot arm moving among obstacles.

The concept of C-space entropy for sensor-based exploration and view planning for general robot-sensor systems has been introduced in [?], [?], [?], [?]. The robot plans the next sensing action (also called the next best view) to maximize the expected C-space entropy reduction, (known as Maximal expected Entropy Reduction, or MER). It gives priority to those areas that increase the maneuverable space around the robot, taking into account its physical size and shape, thereby facilitating reachability for further views. However, previous work had assumed a Poisson point process model for obstacle distribution in the physical space, a simplifying assumption. In this paper we derive an expression for MER criterion assuming an occupancy grid map, a commonly used representation for workspace representation in much of the mobile robot community. This model is easily obtained from typical range sensors such as laser range finders, stereo vision, etc., and furthermore, we can incorporate occlusion constraints and their effect in the MER formulation, making it more realistic. Simulations show that even for holonomic mobile robots with relatively simple geometric shapes (such as a rectangle), the MER criterion yields improvement in exploration efficiency (number of views needed to explore the C-space) over physical space based criteria.

A new hybrid motion planning technique based on Harmonic Functions (HF) and Probabilistic Roadmaps (PRM) is presented. The proposed approach consists of incrementally building a Probabilistic Roadmap using information obtained about the workspace topology through the Fluid Dynamic (FD) paradigm based on HFs. The crux of our approach is to identify narrow passages using FD paradigm and pass the information obtained over to a PRM method to build a roadmap to capture the connectivity of free configuration space (C-space) especially in narrow regions. As an extension to our recent works on using Harmonic Function-based Probabilistic Roadmaps (HFPRM) for robotic navigation [1], we propose an Incremental HFPRM (IHFPRM) technique which is more general and can be applied to virtually any type of robot. Simulation results presented in this paper show that the combination of the HF and the PRM works better than each individual in terms of finding a collision free path in environments where narrow passages exist. This technique can be extended to the sensor-based motion planning of robots (mobile and/or articulated) which is the long-term objective in carrying out this research.

This paper presents a new hybrid motion planning technique based on harmonic functions (HF) and probabilistic roadmaps (PRM). The proposed harmonic function based probabilistic roadmap (HFPRM) method comprises three phases: in phase one, the Laplace's equation, pertinent to potential flow, in an environment cluttered with obstacles is solved. In phase two, a probabilistic roadmap with a novel sampling scheme is constructed based on information obtained about the environment topology through the HF technique developed in phase one. The roadmap is then searched for the shortest path in phase three. Simulation results presented in this paper show that the combination of the HF and the PRM works better than each individual in terms of finding a collision free path in environments where narrow passages exist. The proposed HFPRM method can be extended to sensor-based motion planning problem in environments not known a priori.