An extension based on attention and anticipation of a robot vision pipeline for object recognition in RGBD images from low-cost sensors like MS Kinect or ASUS Xtion is presented. This work originated in research on an industrial application scenario, namely shipping-container unloading, but it is applicable to advanced manipulation tasks in unstructured environments in general where the perception must be very robust while being as fast as possible. For these scenarios, we build on our previous work that proved to be competitive in cluttered scenes in table-top scenarios and which forms the backbone of our RGBD object recognition. It is further enhanced by two main contributions. First, a simple but very effective form of anticipation as top-down expectations of the evolution of the scene due to the actions of the robot is used to speed up the processing. Second, attention is used as a mechanism for further speed-up by focusing processing only on certain regions of interest of the scene based also on an anticipation mechanism. The method is analyzed in experiments using real-world data from an industrial demonstration set-up.

Given that 3D scan matching is such a central part of the perception pipeline for robots, thorough and large-scale investigations of scan matching performance are still surprisingly few. A crucial part of the scientific method is to perform experiments that can be replicated by other researchers in order to compare different results. In light of this fact, this paper presents a thorough comparison of 3D scan registration algorithms using a recently published benchmark protocol which makes use of a publicly available challenging data set that covers a wide range of environments. In particular, we evaluate two types of recent 3D registration algorithms - one local and one global. Both approaches take local surface structure into account, rather than matching individual points. After well over 100 000 individual tests, we conclude that algorithms using the normal distributions transform (NDT) provides accurate results compared to a modern implementation of the iterative closest point (ICP) method, when faced with scan data that has little overlap and weak geometric structure. We also demonstrate that the minimally uncertain maximum consensus (MUMC) algorithm provides accurate results in structured environments without needing an initial guess, and that it provides useful measures to detect whether it has succeeded or not. We also propose two amendments to the experimental protocol, in order to provide more valuable results in future implementations.

We present an approach for object class learning using a part-based shape categorization in RGB-augmented 3D point clouds captured from cluttered indoor scenes with a Kinect-like sensor. We propose an unsupervised hierarchical learning procedure which allows to symbolically classify shape parts by different specificity levels of detailedness of their surface-structural appearance. Further, a hierarchical graphical model is learned that reflects the constellation of classified parts from the set of specificity levels learned in the previous step. Finally an energy minimization inference procedure is applied on the hierarchical graphical model to obtain the corresponding shape category of an object instance consisting of a set of shape parts. Experiments show, due to the proposed classification by learning the hierarchy of shape parts combined with generating the hierarchical graphical constellation model of shape parts, the additional evidence on different levels of shape detailedness is a major factor that leads to a more robust and accurate categorization compared to a flat approach. The experiments are conducted for shape categories (sack, barrel and parcel) which typically appear in visuoperceptual-challenging scenarios of logistic processes.

An autonomous aerial or underwater robot can create photo-maps using a downward looking camera to not only compute its odometry visually but also to provide a useful and intuitively understandable representation of the environment explored by it. The improved Fourier Mellin Invariant (iFMI) registration is a spectral registration method, which has specific benefits, especially high robustness in featureless scenarios, but it only allows registrations of 2D translations, rotation, and scale. The method is extended here to incorporate tilt using parallax information. To this end, we extend the well-known four-point algorithm for planar homography. We show that using the decomposition of the planar homography to compute the tilt is very noise-prone, and propose a way of increasing this accuracy based on a parallax to noise metric. Although our general approach can be used with local scale invariant image features, we implement the tilt-correction based on an extension of our frequency-based approach to determine the image motion-field. Two experiments are presented to show the efficacy and applicability of our approach: An analysis of a simulated data set with ground truth is used to quantify the robustness of our novel method relative to the same four-point method using only SIFT (Scale Invariant Feature Transform) features. A second data set is used to present similar results with real-world data.

This paper presents an approach to acquire textured 3D models of objects without the need for sophisticated hardware infrastructures. The approach is inexpensive, using a low-cost Microsoft Kinect RGB-D sensor and Augmented Reality (AR) markers printed on paper sheets. The AR-markers can be freely placed in the scene, allowing the modeling of objects of various sizes, and the sensor can be moved by the hand of an untrained person. To generate usable models with this very inexpensive and simple setup, the sequence of RGB-D scans is embedded in a graph-based optimizer for automatic post-refinement. The main novelty of this contribution is the development of an uncertainty model for an AR-marker. The AR-marker uncertainty models are used as constraints in an optimization problem to better estimate the object pose. The models are in the end further fine-tuned by a standard point-based registration algorithm. The results section presents realistic models of various objects generated using this system, e.g., parcels, sport balls, human dolls etc. Additionally, a quantitative analysis is presented using objects of known dimensions.

We present an approach for object class learning using a part-based shape categorization in RGB-augmented 3D point clouds captured from cluttered indoor scenes with a Kinect-like sensor. A graph representation is used to detect and categorize object instances based on part-constellations found in scenes. No assumptions like objects being placed on planar surfaces or constraints on their poses are required. Our approach consists of the following steps: 1) a Mean-Shift-based over-segmentation of a point cloud into atomic patches; 2) use of topological and geometric features to merge surface-homogeneous atomic patches into super patches; 3) an unsupervised classification of these parts that allows to symbolically label distinctively unknown object parts by their surface-structural appearance; and finally, 4) a graph generation procedure that reflects the constellation of the detected parts from object instances of certain shape categories. Furthermore, an inference procedure is presented that processes extracted part constellations of a scene to detect and categorize object instances. Experiments with challenging, cluttered scenes show that the segmentation procedure provides salient parts of objects which lead to a good categorization performance using the graph-based constellation model concept.

In this paper, we give an overview of the Jacobs Robotics entry to the ICRA'11 Solutions in Perception Challenge. We present our multi-pronged strategy for object recognition and localization based on the integrated geometric and visual information available from the Kinect sensor. Firstly, the range image is over-segmented using an edge-detection algorithm and regions of interest are extracted based on a simple shape-analysis per segment. Then, these selected regions of the scene are matched with known objects using visual features and their distribution in 3D space. Finally, generated hypotheses about the positions of the objects are tested by back-projecting learned 3D models to the scene using estimated transformations and sensor model. Our method won the second place among eight competing algorithms, only marginally losing to the winner.

A core challenge in probabilistic mapping is to extract meaningful uncertainty information from data registration methods. While this has been investigated in ICP-based scan matching methods, other registration methods have not been analyzed. In this paper, an uncertainty analysis of a Fourier Mellin based image registration algorithm is introduced, which to our knowledge is the first of its kind involving spectral registration. A covariance matrix is extracted from the result of a Phase-Only Matched Filter, which is interpreted as a probability mass function. The method is embedded in a pose graph implementation for Simultaneous Localization and Mapping (SLAM) and validated with experiments in the underwater domain.

Fitting optimum quadrics on segmented noisy range-image is a challenging task. The algebraic least-squares fit usually considered in the literature is biased, although it is fast to compute. We extend our previous work in planar patch extraction using a detailed sensor noise model to the extraction of quadrics. First, the fast least-squares method is modified to detect degeneracy automatically and it is used to aid segmentation. The final segmented quadric patch is then refined using a numerical maximum likelihood approach which employs the sensor range error model. Experimental results for artificial data and two different 3D sensors are provided to show the feasibility of our approach.

The recently introduced Minimum Uncertainty Maximum Consensus (MUMC) algorithm for 3D scene registration using planar-patches is tested in a large outdoor urban setting without any prior motion estimate whatsoever. With the aid of a new overlap metric based on unmatched patches, the algorithm is shown to work successfully in most cases. The absolute accuracy of its computed result is corroborated for the first time by ground-truth obtained using reflective markers. There were a couple of unsuccessful scan-pairs. These are analyzed for the reason of failure by formulating two kinds of overlap metrics: one based on the actual overlapping surface-area and another based on the extent of agreement of range-image pixels. We conclude that neither metric in isolation is able to predict all failures, but that both taken together are able to predict the difficulty level of a scan-pair vis-à-vis registration by MUMC.

Surface-patches based 3D mapping in a real world underwater scenario is presented. It is based on a 6 degrees of freedom registration of sonar data. Planar surfaces are fitted into the sonar data and the subsequent registration method maximizes the overall geometric consistency within a search-space to determine correspondences between the planes. This approach has previously only been used on high quality range data from sensors on land robots like laser range finders. It is shown here that the algorithm is also applicable to very noisy, coarse sonar data. The 3D map presented is of a large underwater structure, namely the Lesumer Sperrwerk, a flood gate north of the city of Bremen, Germany. It is generated from 18 scans collected using a Tritech Eclipse sonar.

We present a robot-pose-registration algorithm, which is entirely based on large planar-surface patches extracted from point clouds sampled from a three-dimensional (3-D) sensor. This approach offers an alternative to the traditional point-to-point iterative-closest-point (ICP) algorithm, its point-to-plane variant, as well as newer grid-based algorithms, such as the 3-D normal distribution transform (NDT). The simpler case of known plane correspondences is tackled first by deriving expressions for least-squares pose estimation considering plane-parameter uncertainty computed during plane extraction. Closed-form expressions for covariances are also derived. To round-off the solution, we present a new algorithm, which is called minimally uncertain maximal consensus (MUMC), to determine the unknown plane correspondences by maximizing geometric consistency by minimizing the uncertainty volume in configuration space. Experimental results from three 3-D sensors, viz., Swiss-Ranger, University of South Florida Odetics Laser Detection and Ranging, and an actuated SICK S300, are given. The first two have low fields of view (FOV) and moderate ranges, while the third has a much bigger FOV and range. Experimental results show that this approach is not only more robust than point- or grid-based approaches in plane-rich environments, but it is also faster, requires significantly less memory, and offers a less-cluttered planar-patches-based visualization.

In this work, we utilize a recently studied more accurate range noise model for 3D sensors to derive from scratch the expressions for the optimum plane which best fits a point-cloud and for the combined covariance matrix of the plane's parameters. The parameters in question are the plane's normal and its distance to the origin. The range standard-deviation model used by us is a quadratic function of the true range and is a function of the incidence angle as well. We show that for this model, the maximum-likelihood plane is biased, whereas the least-squares plane is not. The plane-parameters' covariance matrix for the least-squares plane is shown to possess a number of desirable properties, e.g., the optimal solution forms its null-space and its components are functions of easily understood terms like the planar-patch's center and scatter. We verify our covariance expression with that obtained by the eigenvector perturbation method. We further compare our method to that of renormalization with respect to the theoretically best covariance matrix in simulation. The application of our approach to real-time range-image registration and plane fusion is shown by an example using a commercially available 3D range sensor. Results show that our method has good accuracy, is fast to compute, and is easy to interpret intuitively.

This article addresses fast 3D mapping by a mobile robot in a predominantly planar environment. It is based on a novel pose registration algorithm based entirely on matching features composed of plane-segments extracted from point-clouds sampled from a 3D sensor. The approach has advantages in terms of robustness, speed and storage as compared to the voxel based approaches. Unlike previous approaches, the uncertainty in plane parameters is utilized to compute the uncertainty in the pose computed by scan-registration. The algorithm is illustrated by creating a full 3D model of a multi-level robot testing arena.

Pixel array based 3D range sensors like time of flight sensors (e.g. Swiss-ranger) are commonly used for spatial mapping. Analogous to a laser range finder beam, each pixel measures distances within its conical field of view. However, unlike laser range finders, these sensors use incoherent visible or near visible light for range measurement. This, combined with their relatively long maximum ranges, means that the convenient thin beam assumption can no longer be used to update an occupancy grid map using a probabilistic sensor model. We present an analysis of a scenario where a pixel beam is intersected by more than one object and where a unimodal probabilistic distribution assumption causes spurious objects to appear in the detected scene. This adversely affects path-planning based on such maps because perfectly good escape routes are blocked off. A two step multimodal Gaussian mixture model based procedure is presented which is able to detect multiple obstacles per pixel and hence ameliorate the problem. The results of experiments done using a time of flight based sensor are presented.

A fast but nevertheless accurate approach for surface extraction from noisy 3D point clouds is presented. It consists of two parts, namely a plane fitting and a polygonalization step. Both exploit the sequential nature of 3D data acquisition on mobile robots in form of range images. For the plane fitting, this is used to revise the standard mathematical formulation to an incremental version, which allows a linear computation. For the polygonalization, the neighborhood relation in range images is exploited. Experiments are presented using a time-of-flight range camera in form of a Swissranger SR-3000. Results include lab scenes as well as data from two runs of the rescue robot league at the RoboCup German Open 2007 with 1,414, respectively 2,343 sensor snapshots. The 36ldr106, respectively 59ldr106points from the two point clouds are reduced to about 14ldr103, respectively 23ldr103planes with only about 0.2 sec of total computation time per snapshot while the robot moves along. Uncertainty analysis of the computed plane parameters is presented as well.

This paper presents a new technique for the update of a probabilistic spatial occupancy grid map using a forward sensor model. Unlike currently popular inverse sensor models, forward sensor models can be found experimentally and can represent sensor characteristics better. The formulation is applicable to both 2D and 3D range sensors and does not have some of the theoretical and practical problems associated with the current approaches which use forward models. As an illustration of this procedure, a new prototype 3D forward sensor model is derived using a beam represented as a spherical sector. Furthermore, this model is used for fusion of point-clouds obtained from different 3D sensors, in particular, time-of-flight sensors (Swiss-ranger, laser range finders), and stereo vision cameras. Several techniques are described for an efficient data-structure representation and implementation. The range beams from different sensors are fused in a common local Cartesian occupancy map. Experimental results of this fusion are presented and evaluated using Hough-transform performed on the grid.

The motion of a free-floating space robot is characterized by the principle of conservation of angular momentum. It is well known that these angular momentum equations are nonholonomic, i.e., are nonintegrable rate equations. If the base of the free-floating robot is partially actuated, it is difficult to attain trajectories of the joints that result in point-to-point motion of the entire robot system in its configuration space. However, if the drift-less system associated with the angular momentum conservation equations is shown to be differentially flat, point-to-point maneuvers of the free-floating robot in its configuration space can be constructed. However, an open research problem in the current literature is to show the property of differential flatness for a general space robot. The primary contributions of this paper are as follows: (i) study systematically the structure of the nonholonomic rate constraint equations of a free-floating open-chain space robot with arbitrarily oriented joints and two momentum wheels; (ii) establish the design conditions under which the system exhibits differential flatness; (iii) exploit these design conditions for point-to-point trajectory planning and control of the space robot

In this paper, we address the design of an autopilot for autonomous landing of a helicopter on a rocking ship, due to rough sea. A tether is used for landing and securing a helicopter to the deck of the ship in rough weather. A detailed nonlinear dynamic model for the helicopter is used. This model is underactuated, where the rotational motion couples into the translation. This property is used to design controllers which separate the time scales of rotation and translation. It is shown that the tether tension can be used to couple the translation of the helicopter to the rotation. Two controllers are proposed in this paper. In the first, the rotation time scale is chosen much shorter than the translation, and the rotation reference signals are created to achieve a desired controlled behavior of the translation. In the second, due to coupling of the translation of the helicopter to the rotation through the tether, the translation reference rates are created to achieve a desired controlled behavior of the attitude and altitude. Controller A is proposed for use when the helicopter is far away from the goal, while Controller B is for the case when the helicopter is close to the ship. The proposed control schemes are proved to be robust to the tracking error of its internal loop and results in local exponential stability. The performance of the control system is demonstrated by computer simulations. Currently, work is in progress to implement the algorithm using an instrumented model of a helicopter with a tether.

In this paper, we address the design of an autopilot for autonomous landing of a helicopter on a rocking ship, due to rough sea. The deck is modeled to have a sinusoidal motion. The goal of the helicopter is to land on it during motion. In this work, we use a tether to help in target tracking. Based on the measurement of the angle between the cable and the helicopter/ship, a novel hierarchical two time-scale controller has been proposed to ensure landing of the helicopter on the ship. The system is demonstrated by computer simulation. Currently, work is under progress to implement the algorithm using an instrumented model of a helicopter using a tether.

The objective of this paper is to design a controller based on a nonlinear hover-mode model for a helicopter, which can be seamlessly integrated in an existing spatial path-planning approach. Unlike traditional approaches, this novel two-level hierarchical controller takes ‘ small-body forces’ explicitly into consideration. A higher level controller (HLC), with slower time-scale, controls the translational motion. This controller chooses the main rotor thrust, and a reference signal for the fuselage rotation matrix. This reference rotation matrix is then tracked by a lower level controller (LLC) at a faster time-scale, to select the two flapping angles of the tip path plane (TPP) and the tail rotor thrust. The reference rotation matrix is chosen by the HLC so as to meet the objective of local translation, specified by a local potential function. The local potential field is represented as an obstacle-free spherical region, a ‘ bubble’, in which the helicopter is constrained to fly. A string of such bubbles is generated by an existing spatial path-planning algorithm. The helicopter switches its controlling bubbles in a sequence to produce gross motion. The system is demonstrated by computer simulation, using a scaled model.

In this paper, navigation and control of autonomous mobile unicycle robots in a complex and partially known obstacle-ridden environment is considered. The unicycle dynamic model used has two differentially-driven wheels, with the two wheel motor torques as the system input. Two novel controllers are derived which stabilize the robot within a surrounding disk-shaped area (henceforth called a bubble) of arbitrary size for any initial velocities. The first controller takes the unicycle to the center of its bubble while the second corrects its orientation. The torque-levels can be controlled by adjusting gains. The control laws are independent of inertial parameters. An existing global planner is used by each robot to create a string of bubbles connecting its start point to its goal point, with each bubble’s size indicative of the radial obstacle-clearance available from its center. Each robot then uses its local bubble-controllers to follow its global planned path. This method is then extended to an experimental setup of multiple robots, each equipped with proximity sensors. The same algorithm is run on each robot in a distributed manner with no information sharing. Results are presented to illustrate the robustness of the system.

In this paper, navigation and control of an autonomous mobile unicycle robot in an obstacle-ridden environment is considered. The unicycle dynamic model used has two differentially driven wheels, with the motor torques as the system input. Two novel potential-field-based controllers are derived, which stabilize the robot within a surrounding circular area (henceforth called a bubble) of arbitrary size. The first controller takes the unicycle to the center of its bubble, while the second corrects its orientation. The designed potentials also work with a kinematic model. Explicit bounds for permissible initial speeds are derived, such that maximum torque limits and/or maximum speed limits are not violated once the controller is activated. These controllers are then embedded in a navigation framework. An existing global planner is used to first create a string of variable-sized bubbles which connect the start point to the goal point, with each bubble's size indicative of the radial obstacle clearance available from its center. The robot then keeps itself within a fixed-sized bubble, which it then moves in discrete steps, according to the direction provided by the global plan, while repulsively avoiding unexpected obstacles. Hence, the gross movement is created by switching local potential-field-based controllers. This scheme is first verified in computer simulation of a single robot moving in a maze. It is then implemented on an experimental setup of robots equipped with proximity sensors. Results are presented to illustrate the effectiveness of the system.

In this paper, the dynamic model of a wheeled inverted pendulum (e.g., Segway, Quasimoro, and Joe) is analyzed from a controllability and feedback linearizability point of view. First, a dynamic model of this underactuated system is derived with respect to the wheel motor torques as inputs while taking the nonholonomic no-slip constraints into considerations. This model is compared with the previous models derived for similar systems. The strong accessibility condition is checked and the maximum relative degree of the system is found. Based on this result, a partial feedback linearization of the system is obtained and the internal dynamics equations are isolated. The resulting equations are then used to design two novel controllers. The first one is a two-level velocity controller for tracking vehicle orientation and heading speed set-points, while controlling the vehicle pitch (pendulum angle from the vertical) within a specified range. The second controller is also a two-level controller which stabilizes the vehicle's position to the desired point, while again keeping the pitch bounded between specified limits. Simulation results are provided to show the efficacy of the controllers using realistic data.