Semantic segmentation of images is an important issue for intelligent vehicles and mobile robotics because it offers basic information which can be used for complex reasoning and safe navigation. Different solutions have been proposed for this problem along the last two decades, where recent deep neural networks approaches have shown very promising results in the context of urban navigation. One of the main problems when comparing different semantic segmentation solutions is how to select an appropriate metric to evaluate their accuracy. On the one hand, classic metrics do not measure properly the accuracy on the object contours, which is important in urban driving to differentiate road from sidewalk for instance. On the other hand, contour-based metrics [1] disregard the information far from class contours. This paper explores the problem multi-modal image segmentation, and presents a new metric to leverage global and contour accuracy in a simple formulation. This metric is validated with the evaluation of several semantic segmentation solutions that exploit RGB-D images to rank these solutions taking into account the quality of the segmented contours. We also present a comparative analysis of several commonly used metrics together with a statistical analysis of their correlation.

This paper deals with the problem of active sensing control for nonlinear differentially flat systems. The objective is to improve the estimation accuracy of an observer by determining the inputs of the system that maximise the amount of information gathered by the outputs over a time horizon. In particular, we use the Observability Gramian (OG) to quantify the richness of the acquired information. First, we define a trajectory for the flat outputs of the system by using B-Spline curves. Then, we exploit an online gradient descent strategy to move the control points of the B-Spline in order to actively maximise the smallest eigenvalue of the OG over the whole planning horizon. While the system travels along its planned (optimized) trajectory, an Extended Kalman Filter (EKF) is used to estimate the system state. In order to keep memory of the past acquired sensory data for online re-planning, the OG is also computed on the past estimated state trajectories. This is then used for an online replanning of the optimal trajectory during the robot motion which is continuously refined by exploiting the state estimation obtained by the EKF. In order to show the effectiveness of our method we consider a simple but significant case of a planar robot with a single range measurement. The simulation results show that, along the optimal path, the EKF converges faster and provides a more accurate estimate than along other possible (non-optimal) paths.

Image and point cloud registration methods compute the relative pose between two images. Commonly used registration algorithms are iterative and rely on the assumption that the motion between the images is small. In this work, we propose a fast pose estimation technique to compute a rough estimate of large motions between depth images, which can be used as initialization to dense registration methods. The main idea is to explore the properties given by planar surfaces with co-visibility and their normals from two distinct viewpoints. We present, in two decoupled stages, the rotation and then the translation estimation, both based on the normal vectors orientation and on the depth. These two stages are efficiently computed by using low resolution depth images and without any feature extraction/matching. We also analyze the limitations and observabilty of this approach, and its relationship to ICP point-to-plane. Notably, if the rotation is observable, at least five degrees of freedom can be estimated in the worst case. To demonstrate the effectiveness of the method, we evaluate the initialization technique in a set of challenging scenarios, comprising simulated spherical images from the Sponza Atrium model benchmark and real spherical indoor sequences.

Mimicking human navigation is a challenging goal for autonomous robots. This requires to explicitly take into account not only geometric representation but also high-level interpretation of the environment. In this paper, we demonstrate the capability to infer a route in a global map by using semantics. Our approach relies on an object-based representation of the world automatically built by robots from spherical images. In addition, we propose a new approach to specify paths in terms of high-level robot actions. This path description provides robots with the ability to interact with humans in an intuitive way. We perform experiments on simulated and real-world data, demonstrating the ability of our approach to deal with complex large-scale outdoor environments whilst dealing with labelling errors.

This paper proposes an environmental representation approach based on hybrid metric and topological maps as a key component for mobile robot navigation. Focus is made on an ego-centric pose graph structure by the use of Keyframes to capture the local properties of the scene. With the aim of reducing data redundancy, suppress sensor noise whilst maintaining a dense compact representation of the environment, neighbouring augmented spheres are fused in a single representation. To this end, an uncertainty error model propagation is formulated for outlier rejection and data fusion, enhanced with the notion of landmark stability over time. Finally, our algorithm is tested thoroughly on a newly developed wide angle 360° field of view (FOV) spherical sensor where improvements such as trajectory drift, compactness and reduced tracking error are demonstrated.

Mapping evolving environments requires an update mechanism to efficiently deal with dynamic objects. In this context, we propose a new approach to update maps pertaining to large-scale dynamic environments with semantics. While previous works mainly rely on large amount of observations, the proposed framework is able to build a stable representation with only two observations of the environment. To do this, scene understanding is used to detect dynamic objects and to recover the labels of the occluded parts of the scene through an inference process which takes into account both spatial context and a class occlusion model. Our method was evaluated on a database acquired at two different times with an interval of three years in a large dynamic outdoor environment. The results point out the ability to retrieve the hidden classes with a precision score of 0.98. The performances in term of localisation are also improved.

This paper presents a methodology to combine information from a sequence of RGB-D spherical views acquired by a home-made multi-stereo device in order to improve the computed depth images both in terms of accuracy and completeness. This methodology is embedded in a larger visual mapping framework aiming to produce accurate and dense topometric urban maps. Our method is based on two main filtering stages. Firstly, we perform a segmentation process considering both geometric and photometric image constraints, followed by a regularization step (spatial-integration). We then proceed to a fusion stage where the geometric information is further refined by considering the depth images of nearby frames (temporal integration). This methodology can be applied to other projective models, such as perspective stereo images. Our approach is evaluated within the frameworks of image registration, localization and mapping, demonstrating higher accuracy and larger convergence domains over different datasets.

Navigation in large scale environments is challenging because it requires accurate local map and global relocation ability. We present a new hybrid metric-topological-semantic map structure, called MTS-map, that allows a fine metric-based navigation and fast coarse query-based localisation. It consists of local sub-maps connected through two topological layers at metric and semantic levels. Semantic information is used to build concise local graph-based descriptions of sub-maps. We propose a robust and efficient algorithm that relies on MTS-map structure and semantic description of sub-maps to relocate very fast. We combine the discriminative power of semantics with the robustness of an interpretation tree to compare the graphs very fast and outperform state-of-the-art-techniques. The proposed approach is tested on a challenging dataset composed of more than 13000 real world images where we demonstrate the ability to relocate within 0.12ms.

The integration of several range cameras in a mobile platform is useful for applications in mobile robotics and autonomous vehicles that require a large field of view. This situation is increasingly interesting with the advent of low cost range cameras like those developed by Primesense. Calibrating such combination of sensors for any geometric configuration is a problem that has been recently solved through visual odometry (VO) and SLAM. However, this kind of solution is laborious to apply, requiring robust SLAM or VO in controlled environments. In this paper we propose a new uncomplicated technique for extrinsic calibration of range cameras that relies on finding and matching planes. The method that we present serves to calibrate two or more range cameras in an arbitrary configuration, requiring only to observe one plane from different viewpoints. The conditions to solve the problem are studied, and several practical examples are presented covering different geometric configurations, including an omnidirectional RGB-D sensor composed of 8 range cameras. The quality of this calibration is evaluated with several experiments that demonstrate an improvement of accuracy over design parameters, while providing a versatile solution that is extremely fast and easy to apply.

Social spacing in human-robot interactions is among the main useful features when integrating human social intelligence into robot perception and action skills. One of the main challenges, is to capture the transitions incurred by the human and further take into account robot constraints. Towards this goal, we introduce a novel methodology that can instantiate diverse social spacing models depending on the context and further as a function of uncertainty and robot perception capacity. Our method is based on the use of non-stationary, skew-normal probability density functions for the space of individuals and on treating multi-person space interactions through social mapping. The utility of our approach is shown on an indoor robot operating in the presence of humans, allowing it to exhibit socially intelligent responses.

We present a generative approach to perform robot mapping that is based on an intelligent integration of static and dynamic entity classes within an environment, in order to extrapolate map information at various resolutions. Our framework differentiates from the conventional standpoint where different mapping levels are overlaid on one another, by fusing information from different mapping levels that allows us to infer new information within partially mapped environments. Towards this goal, we develop a class-dependent map extrapolation function that captures the discriminative relation between an environment entity and the mapping procedure. We illustrate the advantages in using heterogeneous contextual information when mapping an environment using a prototype implementation of our approach on an indoor robot platform, giving very promising results.

The visual 3D reconstruction of transparent objects in unstructured scenes is challenging due to the complex image formation principles underlying their visual appearance. Most state-of-the-art reconstruction methods ignore this problem and assume Lambertian reflection. Yet, transparent objects are relevant scene information for applications in intelligent robotics (such as grasping) or virtual reality. In this work, we present an approach to detect non-planar transparent objects, like bottles or glasses, by specifically searching for geometry inconsistencies caused by refraction or reflection. Depth information is acquired using a Kinect sensor, which is moved within the scene in order to acquire multiple views. The individual measurements are combined into a 3D volume, yielding the objects' location and a rough shape estimate. Results are presented using various household objects made of glass or plastic.

Navigating in large scale, complex and dynamic environments requires reliable representations able to capture metric, topological and semantic aspects of the scene for supporting path planing and real time motion control. In a previous work [11], we addressed metric and topological representations thanks to a multi-cameras system which allows building of dense visual maps of large scale 3D environments. The map is a set of locally accurate spherical panoramas related by 6d of poses graph. The work presented here is a further step toward a semantic representation. We aim at detecting the changes in the structural properties of the scene during navigation. Structural properties are estimated online using a global descriptor relying on spherical harmonics which are particularly well-fitted to capture properties in spherical views. A change-point detection algorithm based on a statistical Neyman-Pearson test allows us to find optimal transitions between topological places. Results are presented and discussed both for indoors and outdoors experiments.

Topological navigation consists for a robot in navigating in a topological graph which nodes are topological places. Either for indoor or outdoor environments, segmentation into topological places is a challenging issue. In this paper, we propose a common approach for indoor and outdoor environment segmentation without elaborating a complete topological navigation system. The approach is novel in that environment sensing is performed using spherical images. Environment structure estimation is performed by a global structure descriptor specially adapted to the spherical representation. This descriptor is processed by a custom designed algorithm which detects change-points defining the segmentation between topological places.

International audience

This paper presents a novel approach to self calibrate the extrinsic parameters of a camera mounted on a mobile robot in the context of fusion with the odometry sensor. Calibrating precisely such a system can be difficult if the camera is mounted on a vehicle where the frame is difficult to localize precisely (like on a car for example). However, the knowledge of the camera pose in the robot frame is essential in order to make a consistent fusion of the sensor measurements. Our approach is based on a Simultaneous Localization and Mapping (SLAM) framework: the estimation of the parameters is made when the robot moves in an unknown environment which is only viewed by the camera. First, a study of the observability properties of the system is made in order to characterize conditions that its inputs have to satisfy to make the calibration process possible. Then, we show on a real experiment with an omnidirectional camera the validity of the conditions and the quality of the estimation of the 3D pose of the camera with respect to the odometry frame.

Consumer depth cameras, such as the Microsoft Kinect, are capable of providing frames of dense depth values at real time. One fundamental question in utilizing depth cameras is how to best extract features from depth frames. Motivated by local descriptors on images, in particular kernel descriptors, we develop a set of kernel features on depth images that model size, 3D shape, and depth edges in a single framework. Through extensive experiments on object recognition, we show that (1) our local features capture different aspects of cues from a depth frame/view that complement one another; (2) our kernel features significantly outperform traditional 3D features (e.g. Spin images); and (3) we significantly improve the capabilities of depth and RGB-D (color+depth) recognition, achieving 10–15% improvement in accuracy over the state of the art.

Vision sensors give mobile robots a relatively cheap means of obtaining rich 3D information of their environment, but lack the depth information that a laser range finder can provide. This paper describes a novel composite sensor approach that combines the information given by an omnidirectional camera and a laser range finder to efficiently solve the indoor Simultaneous Localization and Mapping problem and reconstruct a 3D representation of the environment. We report the results of validating our methodology using a mobile robot equipped with a 2D laser range finder and an omnidirectional camera.

This paper proposes the Euclidean homography matrix as visual feature in an image-based visual servoing scheme in order to control an aircraft along the approach and landing phase. With a trajectory defined in the image space by a sequence of equidistant key images along the glidepath, an interpolation in the homography space is also proposed in order to reduce the database size and ensure the required smoothness of the control task. In addition, a pan-tilt control was taken into account to respect the dynamics of the aircraft during manoeuvres and in the presence of wind perturbations. An optimal control design based on the linearized model of the aircraft dynamics is then consider to cancel the visual error function. To demonstrate the proposed concept, simulation results under realistic atmospheric disturbances are presented.

This paper describes an efficient hybrid laser/vision appearance-based approach to provide a mobile robot with rich 3D information about its environment. By combining the information from an omnidirectional camera and a laser range finder, reliable 3D positioning and an accurate 3D representation of the environment is obtained subject to illumination changes even in the presence of occluding and moving objects. A scan matching technique is used to initialize the tracking algorithm in order to ensure rapid convergence and reduce computational cost. The proposed method is validated in an indoor environment using data taken from a mobile robot equipped with a 2D laser range finder and an omnidirectional camera.

This paper describes a generic method for vision-based navigation in real urban environments. The proposed approach relies on a representation of the scene based on spherical images augmented with depth information and a spherical saliency map, both constructed in a learning phase. Saliency maps are built by analyzing useful information of points which best condition spherical projections constraints in the image. During navigation, an image-based registration technique combined with robust outlier rejection is used to precisely locate the vehicle. The main objective of this work is to improve computational time by better representing and selecting information from the reference sphere and current image without degrading matching. It will be shown that by using this pre-learned global spherical memory no error is accumulated along the trajectory and the vehicle can be precisely located without drift.

This paper concerns the stability analysis of image-based visual servoing control laws with respect to uncertainties on the 3-D parameters needed to compute the interaction matrix for any calibrated central catadioptric camera. In the recent past, research on image-based visual servoing has been concentrated on potential problems of stability and on robustness with respect to camera-calibration errors. Only little attention, if any, has been devoted to the robustness of image-based visual servoing to estimation errors on the 3-D structure. It is generally believed that a rough approximation of the 3-D structure is sufficient to ensure the stability of the control law. In this paper, we prove that this is not always true and that an extreme care must be taken when approximating the depth distribution to ensure stability of the image-based control law. The theoretical results are obtained not only for conventional pinhole cameras but for the entire class of central catadioptric systems as well.

This paper addresses the problem of motion estimation and 3-D reconstruction through visual tracking with a single-viewpoint sensor and, in particular, how to generalize tracking to calibrated omnidirectional cameras. We analyze different minimization approaches for the intensity-based cost function (sum of squared differences). In particular, we propose novel variants of the efficient second-order minimization (ESM) with better computational complexities and compare these algorithms with the inverse composition (IC) and the hyperplane approximation (HA). Issues regarding the use of the IC and HA for 3-D tracking are discussed. We show that even though an iteration of ESM is computationally more expensive than an iteration of IC, the faster convergence rate makes it globally faster. The tracking algorithm was validated by using an omnidirectional sensor mounted on a mobile robot.

The majority of visual simultaneous localization and mapping (SLAM) approaches consider feature correspondences as an input to the joint process of estimating the camera pose and the scene structure. In this paper, we propose a new approach for simultaneously obtaining the correspondences, the camera pose, the scene structure, and the illumination changes, all directly using image intensities as observations. Exploitation of all possible image information leads to more accurate estimates and avoids the inherent difficulties of reliably associating features. We also show here that, in this case, structural constraints can be enforced within the procedure as well (instead ofa posteriori), namely the cheirality, the rigidity, and those related to the lighting variations. We formulate the visual SLAM problem as a nonlinear image alignment task. The proposed parameters to perform this task are optimally computed by an efficient second-order approximation method for fast processing and avoidance of irrelevant minima. Furthermore, a new solution to the visual SLAM initialization problem is described whereby no assumptions are made about either the scene or the camera motion. Experimental results are provided for a variety of scenes, including urban and outdoor ones, under general camera motion and different types of perturbations.

This paper describes a new image-based approach to tracking the 6DOF trajectory of a stereo camera pair using a corresponding reference image pairs instead of explicit 3D feature reconstruction of the scene. A dense minimisation approach is employed which directly uses all grey-scale information available within the stereo pair (or stereo region) leading to very robust and precise results. Metric 3D structure constraints are imposed by consistently warping corresponding stereo images to generate novel viewpoints at each stereo acquisition. An iterative non-linear trajectory estimation approach is formulated based on a quadrifocal relationship between the image intensities within adjacent views of the stereo pair. A robust M-estimation technique is used to reject outliers corresponding to moving objects within the scene or other outliers such as occlusions and illumination changes. The technique is applied to recovering the trajectory of a moving vehicle in long and difficult sequences of images.

This paper presents a flexible approach for calibrating omnidirectional single viewpoint sensors from planar grids. These sensors are increasingly used in robotics where accurate calibration is often a prerequisite. Current approaches in the field are either based on theoretical properties and do not take into account important factors such as misalignment or camera-lens distortion or over-parametrised which leads to minimisation problems that are difficult to solve. Recent techniques based on polynomial approximations lead to impractical calibration methods. Our model is based on an exact theoretical projection function to which we add well identified parameters to model real-world errors. This leads to a full methodology from the initialisation of the intrinsic parameters to the general calibration. We also discuss the validity of the approach for fish-eye and spherical models. An implementation of the method is available as OpenSource software on the author's Web page. We validate the approach with the calibration of parabolic, hyperbolic, folded mirror, wide-angle and spherical sensors.

Traditionally in monocular SLAM, interest features are extracted and matched in successive images. Outliers are rejected a posteriori during a pose estimation process, and then the structure of the scene is reconstructed. In this paper, we propose a new approach for computing robustly and simultaneously the 3D camera displacement, the scene structure and the illumination changes directly from image intensity discrepancies. In this way, instead of depending on particular features, all possible image information is exploited. That problem is solved by using an efficient second-order optimization procedure and thus, high convergence rates and large domains of convergence are obtained. Furthermore, a new solution to the visual SLAM initialization problem is given whereby no assumptions are made either about the scene or the camera motion. The proposed approach is validated on experimental and simulated data. Comparisons with existing methods show significant performance improvements.

International audience

This paper presents several methods for estimating the relative position of a central catadioptric camera (including perspective cameras) and a laser range finder in order to obtain depth information in the panoramic image. The problem is analysed from a robotic perspective and according to the available information (visible or invisible laser beam, partial calibration, drift of laser data,...). The feasibility of the calibration is also discussed. The feature extraction process and real data are presented

This work proposes a new vision-based framework to control a robot within model-free large-scale scenes, where the desired pose has never been attained beforehand. Thus, the desired image is not available. It is important to remark that existing visual servoing techniques cannot be applied in this context. The rigid, unknown scene (i.e. the metric model is also not available) is represented as a collection of planar regions, which may leave the field-of-view continuously as the robot moves toward its distant goal. Hence, a novel approach to detect new planes that enter the field-of-view, which is robust to large camera calibration errors, is then deployed here. In fact, it is well-known that representing the scene as composed by planes, the estimation processes are improved in terms of accuracy, stability, and rate of convergence. This extended 3D vision-based control technique is also based on an efficient second-order method for plane-based tracking and pose reconstruction. The framework is validated by using simulated data with artificially created scenes as well as with real images, and accurate navigation tasks are shown

The objective of this paper is to propose an innovative visual servoing method in order to improve the 3D reconstruction of objects for quantitative measurements. The method uses a Stereovision system that allows us to obtain several shots of an object, at regular intervals according to a predefined trajectory. In our case, the stereo rig is equipped with two different cameras, consequently the intrinsic parameters are not the same. The first one is fixed while the other one is mounted on a pan and tilt. Simulation and preliminary experiments in laboratory conditions shows the validity of our approach

This paper presents a parametric approach for tracking piecewise planar scenes with central catadioptric cameras (including perspective cameras). We extend the standard notion of homography to this wider range of devices through the unified projection model on the sphere. We avoid unwarping the image to a perspective view and take into account the non-uniform pixel resolution specific to non-perspective central catadioptric sensors. The homography is parametrised by the Lie algebra of the special linear group SL(3) to ensure that only eight free parameters are estimated. With this model, we use an efficient second-order minimisation technique leading to a fast tracking algorithm with a complexity similar to a first-order approach. The developed algorithm was tested on the estimation of the displacement of a mobile robot in a real application and proved to be very precise

This work presents a method for segmenting image patches which correspond to planar regions in the scene. The method consists of an efficient and robust solution for detecting multiple planar regions in a global optimal sense. Moreover, in contrast with existing techniques which also work on intensity images, neither assumptions about the scene are made nor heuristic hypotheses are formulated. More specifically, the proposed method is based on a systematic, progressive voting procedure from the solution of a linear system, which exploits the two-view geometry. Hence, besides avoiding intermediary depth maps, the progressive mechanism together with such a convergence mapping drastically reduce the computational and storage complexities of the approach. Results from both synthetic and real-world scenes in different scenarios and under various kinds of strong noise confirm its effectiveness and robustness against large camera calibration errors and to the presence of outliers

This paper presents a methodology for improving consistency of the simultaneous localization and mapping (SLAM) in large scale cyclic environments. The SLAM problem is embedded in a reactive sensor-based navigation approach and exploits data provided by a rotative laser range finder. The model of the unknown indoor environment is structured as an hybrid representation, both topological and metric, which is incrementally built during the exploration task. A global likelihood function is modeled from the geometric elastic relationships between different places of the environment, constrained by a rigid metric model inside each place. The inconsistencies in the final resulted map are minimized by deforming the hybrid model with the optimization of the global likelihood of the system by applying a relaxation methodology. Results are presented which shows the minimization of inconsistencies related to an autonomous constructed hybrid model by the application of the proposed methodology

In this paper, we present a complete system for car platooning using visual tracking. The visual tracking is achieved by directly estimating the projective transformation (in our case a homography) between a selected reference template attached to the leading vehicle and the corresponding area in the current image. The relative position and orientation of the servoed car with regard to the leading one is computed by decomposing the homography. The control objective is stated in terms of path following task in order to cope with the non-holonomic constraints of the vehicles.

This paper presents a methodology of error correction applied to autonomously mapping large scale cyclic environment by mobile robots based on laser scan measurements. The model of the unknown indoor environment is structured as a hybrid representation, both topological and geometrical, which is incrementally built during the exploration task. The correction methodology is based on the minimization of geometric elastic relations between different places of the environment, constrained by rigid geometrical model inside each place. Results are presented which shows the minimization of inconsistencies related to an autonomous constructed hybrid model by the application of the proposed methodology.

In the present paper, an image-based visual servoing scheme is presented for a road following task with an autonomous airship. A new set of visual signals is introduced, namely with the vanishing point coordinates and the vanishing line parameters, which have the advantage of decoupling the rotation DOF and respecting the natural characteristics of the vehicle dynamics. An optimal control design is used for a first implementation of the approach with the simulation platform of the AURORA airship. Simulation results are presented and discussed demonstrating a fair performance even in realistic wind conditions.

This paper presents a new methodology for large scale environment modelling by mobile robots. The model of the indoor environment is structured as an hybrid representation, both topological and geometrical, which is incrementally built during the exploration task. The topological aspect of the model captures the connectivity and accessibility of the different places in the environment, and the geometrical model supports a precise robot localization method and a map building of the free space. Experiments are shown which confirm the advantages of merging the topology and metrics of the environment in an hybrid model.

This paper describes an original method to compute the relative motion of an uncalibrated stereo rig in urban environments from features lying on the road. The extraction of significant reliable features on the road remains the critical step of this method. We nevertheless detect them according to the stereo constraints and an a priori knowledge on the scene. The motion between two frames of the stereo rig is considered as rigid: the homography computation is enforced by the redundancy of the feature locations in multiple views. The method has been tested on video sequences recorded from a test vehicle that was driven in an urban environment. Promising results from these experiments are presented.

This paper presents a robust probabilistic approach based on the Bayesian estimation theory to extract and tracking line segment parameters from successive laser range scans, acquired during the evolution of a mobile robot in an indoor environment. In this methodology, a likelihood function is modelled and associated to the existence of structured objects around the robot, an uncertainty model associated to the telemetric data is derived and used to update the likelihood function. In this way, the distances to the near objects around the robot are estimated and used in the feedback loop of a sensor-based control navigation strategy. Experiments are performed using a mobile robot equiped with a 2D laser scanner device, validating the application of the Bayesian segmentation methodology in the laser-based robot navigation.

This paper concerns the stability analysis of image-based visual servoing with respect to uncertainties on the depths of the observed object. In the recent past, research on image-based visual servoing has been concentrated on potential problems of stability and on robustness with respect to camera calibration errors. Only little attention, if any, has been devoted to the robustness of image-based visual servoing to depth estimation errors. It is generally believed that a rough approximation of the depth distribution is sufficient to ensure the stability of the control law. In this paper, we prove that the robustness domain is not so wide and that an extern care must be taken when approximating the depth distribution.

This paper presents a new approach to active affine reconstruction without an exact knowledge of the robot's kinematic model nor camera intrinsic parameters. Affine reconstruction from perspective image pairs is easy if the motion of the camera between the two images is a pure translation. If the robot is not well calibrated, a pure translation achieved with an open loop control would lead to a bias in the reconstruction. The problem can be solved by using a 2 1/2 D visual servoing technique in order to close the loop and control the camera trajectory. The affine reconstruction is equivalent to the estimation of the depths of the 3D points of the scene. Thus, affine reconstruction is very useful to implement several visual servoing approaches which need an estimation of the depths.

In this paper, we address the problem of computing the egomotion of a vehicle in an urban environment using dynamic vision. We assume a planar piecewise world where the planes are mainly distributed along three principal directions corresponding to the axes of a reference frame linked to the ground plane with a vertical z-axis. We aim to estimate both the motion of the car and the principal planes in the scene corresponding to the road and the frontages of the building from a sequence of images provided by an on-board uncalibrated camera. In this paper, we present preliminary results concerning the robust segmentation of the road using projective properties of the scene. We develop a two-stage algorithm in order to increase robustness. The first stage detects the borders of the road using a contour-based approach and primarily allows us to estimate the dominant vanishing point (DVP). The DVP and the borders of the road are then used to constrain the region where the points of interest, corresponding to the road lane markers, can be extracted. The second stage uses a robust technique based on projective invariant to match the lines and points between two consecutive images in the sequence. Finally, we compute the homography relating the points and lines lying on the road into the two images.

Addresses the issue of automatic hovering of an outdoor autonomous airship using image-based visual servoing. The hovering controller is designed using a full dynamic model of the airship, in a PD error feedback scheme, taking the visual signals as output and extracted from an on-board camera. The behavior and stability of the airship motion during the task execution and subjected to the wind disturbance is studied. The approach is finally validated in simulation using an accurate airship model.

This paper deals with the robust tracking of features extracted from a sequence of images taken with an uncalibrated camera mounted on a mobile robot. Unlike most vision systems, the 3D structure of the observed objects is completely unknown. Thus, position-based visual servoing cannot be used. Similarly, one must be careful when using image-based visual servoing since the depths of the features are unknown. On the other hand, 2 1/2 D visual servoing can easily deal with unknown environments since it is only based on projective reconstruction. In order to obtain a good projective reconstruction for a safe vision-based control, we propose a multi-scale real-time approach to extract robust features. Experiments show that our algorithm can be used in robotics applications when images are noisy and uncontrolled perturbations can break the continuity of the robot motion.

A pose estimation method applied to an indoor mobile robot equipped with a laser range-finder that periodically delivers a range scan of the environment is presented. The method is based on the fusion between the range readings delivered by the laser and the feedback control input computed to constrain the robot to move in its free workspace. The relative motion of the robot between two successive sampling times of the control loop is estimated:(1) computing a prediction by comparing the local maps built at these instants, (2) integrating the feedback control input computed from the laser readings to produce the desired motion and (3) performing a fusion between these two predictions. These computations are performed at sampling rate during the robot's evolution and are embedded in a probabilistic approach taking into account the uncertainties associated to the laser readings. Experimental results are presented validating the proposed method.

In this paper we propose a new approach for visual-servoing with respect to a set of 3D straight lines. The main difference with respect to previous approaches is that the new scheme can be used with a zooming camera or even if the reference image has been learned with a different camera. The zoom is particularly useful in order to keep the visual features in the camera field of view and/or to bound their size in the image reducing the influence of noise on features extraction. Experiments with a zooming camera have validated the vision-based control law.

A bounded-error methodology applied to estimate relative displacements of an indoor mobile robot equipped with a laser range-finder that periodically delivers a range scan of the environment is presented. The method is based on the fusion between the range readings delivered by the laser and the feedback control input used to constraint the robot to move in its free workspace. A weighted least square estimation method, previously published (Victorino, Rives, and Borrelly, 2002), is recalled and a comparison between the proposed bounded-error methodology and the least square method is presented. Experimental results validate the application of the bounded-error estimation method.

This work presents a new methodology for navigating mobile robots in unknown environments based on the robot's perception. The interaction between the robot and the environment is modelled, and control laws are designed so that the robot is constrained to move on the Voronoi diagram of the environment. The proposed method enables the robot to explore an unknown scene without any reference trajectory or any prior knowledge about the environment and avoiding the obstacles. We apply this approach on an indoor mobile robot using a 2D laser range finder. We show that the displacement errors remain bounded when the movements of the robot are constrained by sensor based control laws, that results in a precise self-localization and reliable map building. Experimental results shown in this article validate this methodology.

In this paper, we propose a new image-based approach using epipolar geometry. The problem which is addressed can be stated as follows: starting from a Cartesian situation, we want to drive the robot to a desired one using only image data provided during the robot motion. With regard to classical image-based visual servoing, we assume no prior knowledge about the 3D structure or about a desired image to reach. Simulation and experimental results are shown.

Addresses the problem of simultaneous localization and map building for a mobile robot moving in an unknown environment. We show that the robustness of the techniques currently used can be notably enhanced when they are embedded in a sensor-based control scheme. We are developing such an approach applied to a navigation task of an indoor mobile robot using a 2-D laser range finder. The control objectives related to the sensor based control loop are defined thanks to the properties of the generalized Voronoi diagram. A roadmap of the environment is incrementally built from the laser range information. In this way the robot can explore an unknown scene without any reference trajectory or any prior knowledge about the environment. The localization and mapping process is performed in a probabilistic sense by fusing laser and odometry data thanks to an extended Kalman filter.

We deal with the control of an autonomous underwater vehicle by means of visual servoing techniques. Both theoretical and implementation issues are addressed. After having briefly presented the visual servoing framework used, we focus on its application to a pipe following task. The last part of the paper presents results obtained both in simulation and on the Vortex underwater vehicle which is an experimental ROV built by the Ifremer company.

We present a visual servoing approach applied to the control of an autonomous underwater vehicle executing a pipe inspection task. After having briefly recalled the theoretical framework used, we design a closed loop control scheme using visual data as feedback. This scheme is analyzed using the facilities of our homemade Simparc simulation package which can handle both control and sensory aspects. The last part of the paper presents results obtained on, the Vortex underwater vehicle which is an experimental ROV built by the Ifremer company.

In a visual servoing approach, the control objective is stated in terms of regulating an output function directly expressed in the image space. This approach is particularly usefull when the robotic task can be naturally specified as a relative positioning of a sensor frame handled by the robot with regard to a peculiar part of the environment. In our previous works (Espiau et al. (1992), Rives et al. (1996)), we developed a framework which allows us to handle the different steps of a visual servoing application from the specification level to the synthesis of a robust controller. In this paper, we focus on the evaluation of such a framework applied to a specific pipe inspection task using the Ifremer's ROV Vortex. We show simulation results which validate the approach in presence of modeling errors and measurement noises. We also present experimental issues both in terms of implementation and performances.

Discusses both theoretical and implementation issues of a vision based control approach applied to a mobile robot. After a brief presentation of the visual servoing framework used, the authors point out some key problems associated to its application in the case of nonholonomic mobile robots. To overcome these problems, the authors propose to use a hand-eye system mounted on the mobile robot in order to introduce redundancy with respect to the task. The second part of the paper presents the authors' first results obtained on their experimental testbed.

For underwater vehicles to be self-sufficient in an a priori unknown environment, feedback from the environment through altitude sensors is essential. This paper presents how this feedback can be efficiently achieved for AUVs in which different simultaneous tasks have to concur to the achievement of a mission. The approach is based on the task-function approach, with a continuous switching between the involved tasks. The presented application concerns the use of acoustic range/altitude sensors in the control of a fully-actuated AUV.

Vision-based control in robotics based on considering a vision system as a specific sensor dedicated to a task and included in a control servo loop is described. Once the necessary modeling stage is performed, the framework becomes one of automatic control, and stability and robustness questions arise. State-of-the-art visual servoing is reviewed, and the basic concepts for modeling the concerned interactions are given. The interaction screw is thus defined in a general way, and the application to images follows. Starting from the concept of task function, the general framework of the control is described, and stability results are recalled. The concept of the hybrid task is presented and then applied to visual sensors. Simulation and experimental results are presented, and guidelines for future work are drawn in the conclusion.>

An application is described of the so-called visual servoing approach to robot positioning with respect to an object and to target tracking. After briefly discussing how the task function approach can be applied to tasks which include the use of visual features, the authors give a simplified control expression which explicitly takes into account the case of moving objects. Estimating the target velocity while performing the tracking control leads to a kind of adaptive control scheme. The authors then consider the specific case of a square target and derive all the components of the control scheme. They present some experimental results obtained at the video rate in an experimental system composed of a camera mounted on the end effector of a six-DOF robot.>

This paper presents a scheme allowing to estimate parameters which describe geometrical structures in a 3D scene by only using informations issued from a sequence of images provided by a mobile vision sensor with known motion. We first recall the basic used models : points and lines, and then relate their perspective projection in the image plane to the camera motion. Some techniques of recursive filtering are used into the sequence of images to incrementaly build the 3D scene all along the displacement of the camera. Some experimental results in the field of robotics are given.