Vehicle-to-everything (V2X) collaborative perception has emerged as a promising solution to address the limitations of single-vehicle perception systems. However, existing V2X datasets are limited in scope, diversity, and quality. To address these gaps, we present Mixed Signals, a comprehensive V2X dataset featuring 45.1k point clouds and 240.6k bounding boxes collected from three connected autonomous vehicles (CAVs) equipped with two different configurations of LiDAR sensors, plus a roadside unit with dual LiDARs. Our dataset provides point clouds and bounding box annotations across 10 classes, ensuring reliable data for perception training. We provide detailed statistical analysis on the quality of our dataset and extensively benchmark existing V2X methods on it. The Mixed Signals dataset is ready-to-use, with precise alignment and consistent annotations across time and viewpoints. Dataset website is available at https://mixedsignalsdataset.cs.cornell.edu/.

Ensuring connectivity and coordination in heterogeneous multi-robot systems (MRS) navigating complex environments is a critical challenge, especially when communication constraints and obstacles cause robots to become lost or disconnected. This paper presents a novel approach integrating Model Predictive Control (MPC) with Generalized Connectivity Maintenance (GCM) to enable real-time path adaptation while preserving connectivity. We introduce a decentralized decision-making framework that enables robots to recover lost members dynamically. When reconnection is infeasible, the system adapts the mission to continue while accounting for disconnected robots. Our method is evaluated through extensive simulations, showing its scalability and effectiveness in maintaining connectivity and ensuring mission success. Additionally, we propose a new evaluation metric that comprehensively assesses system performance, considering connectivity, coordination, and mission success in challenging environments.

In this paper, we propose a theoretical framework for designing a multi-robot formation equipped with Ultra-wideband (UWB) sensors to localize a target robot. In the presence of noisy range measurements, the accuracy of the target robot’s pose estimation is highly dependent on the chosen formation geometry. Different from existing works, we account for the heterogeneous standard deviations of range measurements across different UWB transmitter-receiver pairs. We establish new optimality conditions for formation geometries and conduct a sensitivity analysis of optimal formations under robot positioning errors. In a 2D setting, we derive necessary and sufficient conditions for both optimality and robustness to robot positioning uncertainty. Experimental results confirm the heterogeneous standard deviations of UWB range measurements and validate the target robot’s confidence ellipse model. An experimental comparison of formation geometries, optimized with and without considering heterogeneous noise, emphasizes the importance of accounting for the heterogeneous standard deviations of range measurements. In addition, we experimentally demonstrate that robust formation geometries improve the target robot’s confidence ellipse in the presence of positioning errors.

Automated vehicles perform well in simple environments with clear rules, but urban traffic presents significant challenges due to the unpredictable behavior of road users, sometimes beyond traffic rules. Achieving full autonomy in such settings requires a systematic approach to modeling the possible actions of all agents. This paper presents a multi-rules reachability analysis framework that integrates graph-based maps with real-time perception data to dynamically characterize the surrounding space. By leveraging the semantic richness and modularity of Lanelet2 maps, our method provides a structured representation that enhances situational awareness. This allows for the extraction of navigation-relevant information, with the goal of supporting safer and more efficient decision-making in complex urban environments.

Stereo-matching is one of the most important low-level visual perception tasks. Currently, two-stage 2D-3D networks are the main solutions. These methods involve creating a cost volume using low-resolution stereo feature maps, which separate the network into a feature net and a matching net. However, two-stage methods may accumulate errors, and the use of a low-resolution cost volume may result in the loss of some of the matching information. To overcome these problems, we propose the first one-stage deep stereo network, named StereoOne. It has an efficient module that builds a cost volume at image resolution in real-time. The feature extraction and matching are learned in a single 3D network. Based on the experiments, the new network outperforms 2D-3D network baselines and achieves competitive performance with the state-of-the-art.

This paper presents a novel approach for Visual Servoing (VS) using a multispectral camera, where the number of data are more than three times that of a standard color camera. To meet real-time feasibility, the multispectral data captured by the camera are processed using dimensionality reduction techniques. Instead of relying on traditional approaches that select a subset of bands, the proposed method unlocks the full potential of a multispectral camera by pinpointing individual pixels that hold the richest information across all bands. While sacrificing spectral resolution for enhanced spatial resolution - crucial for precise robotic control in forested environments - this fusion process offers a powerful tool for robust and real-time VS in natural settings. Validated through simulations and real-world experiments, the proposed approach demonstrates its efficacy by leveraging the full spectral information of the camera while preserving spatial details.

This work focuses on localizing a single target robot with multi-robot formations in 2D space. The cooperative robots employ inter-robot range measurements to assess the target position. In the presence of noisy measurements, the choice of formation geometries significantly impacts the accuracy of the target robot’s pose estimation. While an infinite number of geometries exists to optimize localization accuracy, the current practice is to choose the final formation geometry based on convenience criteria such as simplicity or proximity to the initial position of the robots. The former leads to the selection of regular polygon-shaped formations, while the latter results in behaviour-based formations. Different from existing works, we conduct a complete robustness study of formation geometries in the presence of deviations from the desired formation and range measurement errors. In 2D scenarios, we establish necessary and sufficient conditions for formation geometries to be robust against robot positioning errors. This result substantiates the extensive use of regular polygon formations. However, our analysis reveals the lack of robustness of the commonly used square formation geometry, which stands as an exception. Simulation results illustrate the advantages of these robust geometries in enhancing target localization accuracy.

In the field of lidar odometry for autonomous navigation, the Iterative Closest Point (ICP) algorithm is a prevalent choice for estimating robot motion by comparing point clouds. However, ICP accuracy is strictly dependent on the nature of the features involved, but also on the directional choice of the extraction and matching, either from the current to the reference point cloud or vice-versa. Point-to-line or point-to-plane correspondences have been proven to provide the more accurate odometry results. The matching is generally done in a mono-directional framework: extract the features (lines or planes) in the current point cloud and match them to points in the reference point cloud. This paper introduces a novel formulation, named Balanced ICP, that performs feature extraction (lines or planes) in both point clouds and consequent matching in both the directions. Therefore, the cost function is designed to perform a simultaneous optimization of all available data balancing the noise and extraction errors. The experiments, conducted both on simulated and real data from the KITTI dataset, reveal that our method outperform the classical mono-directional formulations, in terms of robustness, accuracy and stability.

Occlusion presents a significant challenge for safety-critical applications such as autonomous driving. Collaborative perception has recently attracted a large research interest thanks to the ability to enhance the perception of autonomous vehicles via deep information fusion with intelligent roadside units (RSU), thus minimizing the impact of occlusion. While significant advancement has been made, the data-hungry nature of these methods creates a major hurdle for their realworld deployment, particularly due to the need for annotated RSU data. Manually annotating the vast amount of RSU data required for training is prohibitively expensive, given the sheer number of intersections and the effort involved in annotating point clouds. We address this challenge by devising a label-efficient object detection method for RSU based on unsupervised object discovery. Our paper introduces two new modules: one for object discovery based on a spatial temporal aggregation of point clouds, and another for refinement. Furthermore, we demonstrate that fine-tuning on a small portion of annotated data allows our object discovery models to narrow the performance gap with, or even surpass, fully supervised models. Extensive experiments are carried out in simulated and real-world datasets to evaluate our method†.

Autonomous mobile robot navigation in a human populated and encumbered environment is recognized as a hard problem to be solved in real-time. Most of the time, robots face the so-called ‘Freezing Robot Problem’, that occurs when the robot stops because no feasible and safe motion can be found. In order to provide to the robot the capability of proactive navigation, in this work we generalize the classical Social Force Model into a Universal Social Force Model (USFM) that attributes to any object surrounding the robot (humans, robots, obstacles) a social behavior. Nonlinear Model Predictive Control (MPC) can be used to solve the autonomous navigation problem since it can take into account all the possible constraints coming from the interaction model between the robot and the different surrounding objects. However, to be effective, MPC requires a sufficiently large prediction horizon, which generally implies a high computational cost. In order to considerably reduce the computational cost, we propose a new control parametrisation based on Thin Plate Spline Radial Basis Functions that allow us to have a large prediction horizon with fewer parameters. The global control framework is validated in simulation with virtual pedestrians, and in real world environments.

Visual odometry is an important part of the perception module of autonomous robots. Recent advances in deep learning approaches have given rise to hybrid visual odometry approaches that combine both deep networks and traditional pose estimation methods. One limitation of deep learning approaches is the availability of ground truth data needed to train the neural networks. For example, it is extremely difficult, if not impossible, to obtain a ground truth dense depth map of the environment to be used for stereo visual odometry. Even if unsupervised training of networks has been investigated, supervised training remains more reliable and robust. In this paper, we propose a new hybrid dense stereo visual odometry approach in which a dense depth map is obtained with a network that is supervised using ground truth poses that can be more easily obtained than ground truth depths maps. The depth map obtained from the neural network is used to warp the current image into the reference frame and the optimal pose is obtained by minimizing a cost function that encodes the similarity between the warped image and the reference image. The experimental results show that the proposed approach, not only improves state-of-the-art depth maps estimation networks on some of the standard benchmark datasets, but also outperforms the state-of-the-art visual odometry methods.

This paper addresses the problem of stabilizing a robot at a pose specified via a reference image. Specifically, this paper focuses on six degrees-of-freedom visual servoing techniques that require neither metric information of the observed object nor precise camera and/or robot calibration parameters. Not requiring them improves the flexibility and robustness of servoing tasks. However, existing techniques within the focused class need prior knowledge of the object shape and/or of the camera motion. We present a new visual servoing technique that requires none of the aforementioned information. The proposed technique directly exploits 1) the projective parameters that relate the current image with the reference one and 2) the pixel intensities to obtain these parameters. The level of versatility and accuracy of servoing tasks are, thus, further improved. We also show that the proposed nonmetric scheme allows for path planning. In this way, the domain of convergence is greatly enlarged as well. Theoretical proofs and experimental results demonstrate that visual servoing can, indeed, be highly accurate and robust, despite unknown objects and imaging conditions. This naturally encompasses the cases of color images and illumination changes.

This paper concerns the stability analysis of image-based visual servoing control laws with respect to uncertainties on the 3-D parameters needed to compute the interaction matrix for any calibrated central catadioptric camera. In the recent past, research on image-based visual servoing has been concentrated on potential problems of stability and on robustness with respect to camera-calibration errors. Only little attention, if any, has been devoted to the robustness of image-based visual servoing to estimation errors on the 3-D structure. It is generally believed that a rough approximation of the 3-D structure is sufficient to ensure the stability of the control law. In this paper, we prove that this is not always true and that an extreme care must be taken when approximating the depth distribution to ensure stability of the image-based control law. The theoretical results are obtained not only for conventional pinhole cameras but for the entire class of central catadioptric systems as well.

In the last decade, many vision-based robot controllers have been designed using Cartesian information encoded in the homography transformation that links two images of a planar object. For any approach, the performance of the closed-loop system depends on the quality of the homography estimates obtained. In this paper, we exploit the special linear Lie-group structure of the set of all homographies to develop a dynamic observer to estimate homographies online. The resulting estimates are effective and can be used to improve closed-loop response of several visual servoing algorithms.

This paper addresses the problem of tracking a planar region of the scene using an uncalibrated omnidirectional camera. Omnidirectional cameras are a popular choice of visual sensors in robotics because the large field of view is well adapted to motion estimation and obstacle avoidance. The novelty of this work resides in simplifying the calibration phase by providing a direct approach to tracking without any prior knowledge of the camera, lens or mirror parameters. We deal with a nonlinear optimization problem that can be solved for small displacements between two images like those acquired at video rate by a camera mounted on a robot. In order to assess the performance of the proposed method, we perform experiments with synthetic and real data.

To date, there exist only few works on the use of color images for visual servoing. Perhaps, this is due to the difficulties usually found to cope with illumination changes in these images. This paper presents new parametric models and optimization methods for robustly and directly registering color images. Direct methods refer to those that exploit the pixel intensities, without resorting to image features. We then show how a robust and generic visual servoing scheme can be constructed using the obtained optimal parameters. The proposed models ensure robustness to arbitrary illumination changes in color images, do not require prior knowledge (including the spectral ones) of the object, illuminants or camera, and naturally encompass gray-level images. Furthermore, the exploitation of all information within the images, even from areas where no features exist, allow the algorithm to achieve high levels of accuracy. Various results are reported to show that visual servoing can indeed be highly accurate and robust despite unknown objects and unknown imaging conditions.

This paper addresses the problem of motion estimation and 3-D reconstruction through visual tracking with a single-viewpoint sensor and, in particular, how to generalize tracking to calibrated omnidirectional cameras. We analyze different minimization approaches for the intensity-based cost function (sum of squared differences). In particular, we propose novel variants of the efficient second-order minimization (ESM) with better computational complexities and compare these algorithms with the inverse composition (IC) and the hyperplane approximation (HA). Issues regarding the use of the IC and HA for 3-D tracking are discussed. We show that even though an iteration of ESM is computationally more expensive than an iteration of IC, the faster convergence rate makes it globally faster. The tracking algorithm was validated by using an omnidirectional sensor mounted on a mobile robot.

The majority of visual simultaneous localization and mapping (SLAM) approaches consider feature correspondences as an input to the joint process of estimating the camera pose and the scene structure. In this paper, we propose a new approach for simultaneously obtaining the correspondences, the camera pose, the scene structure, and the illumination changes, all directly using image intensities as observations. Exploitation of all possible image information leads to more accurate estimates and avoids the inherent difficulties of reliably associating features. We also show here that, in this case, structural constraints can be enforced within the procedure as well (instead ofa posteriori), namely the cheirality, the rigidity, and those related to the lighting variations. We formulate the visual SLAM problem as a nonlinear image alignment task. The proposed parameters to perform this task are optimally computed by an efficient second-order approximation method for fast processing and avoidance of irrelevant minima. Furthermore, a new solution to the visual SLAM initialization problem is described whereby no assumptions are made about either the scene or the camera motion. Experimental results are provided for a variety of scenes, including urban and outdoor ones, under general camera motion and different types of perturbations.

In this paper, we investigate how to improve the robustness of visual tracking methods with respect to generic lighting changes. We propose a new approach to the direct image alignment of either Lambertian or non-Lambertian objects under shadows, inter-reflections, glints as well as ambient, diffuse and specular reflections which may vary in power, type, number and space. The method is based on a proposed model of illumination changes together with an appropriate geometric model of image motion. The parameters related to these models are obtained through an efficient second-order optimization technique which minimizes directly the intensity discrepancies. Comparison results with existing direct methods show significant improvements in the tracking performance. Extensive experiments confirm the robustness and reliability of our method.

This paper describes a new image-based approach to tracking the 6DOF trajectory of a stereo camera pair using a corresponding reference image pairs instead of explicit 3D feature reconstruction of the scene. A dense minimisation approach is employed which directly uses all grey-scale information available within the stereo pair (or stereo region) leading to very robust and precise results. Metric 3D structure constraints are imposed by consistently warping corresponding stereo images to generate novel viewpoints at each stereo acquisition. An iterative non-linear trajectory estimation approach is formulated based on a quadrifocal relationship between the image intensities within adjacent views of the stereo pair. A robust M-estimation technique is used to reject outliers corresponding to moving objects within the scene or other outliers such as occlusions and illumination changes. The technique is applied to recovering the trajectory of a moving vehicle in long and difficult sequences of images.

Traditionally in monocular SLAM, interest features are extracted and matched in successive images. Outliers are rejected a posteriori during a pose estimation process, and then the structure of the scene is reconstructed. In this paper, we propose a new approach for computing robustly and simultaneously the 3D camera displacement, the scene structure and the illumination changes directly from image intensity discrepancies. In this way, instead of depending on particular features, all possible image information is exploited. That problem is solved by using an efficient second-order optimization procedure and thus, high convergence rates and large domains of convergence are obtained. Furthermore, a new solution to the visual SLAM initialization problem is given whereby no assumptions are made either about the scene or the camera motion. The proposed approach is validated on experimental and simulated data. Comparisons with existing methods show significant performance improvements.

Image-based deformations are generally used for visual tracking of deformable objects moving in the 3D space. For the visual tracking of deformable objects, this assumption has shown to give good results. However it is not satisfying for the visual tracking of 3D rigid objects as the underlying structure cannot be directly estimated. The general belief is that obtaining the 3D structure directly is difficult. In this article, we propose a parameterization that is well adapted either to track deformable objects or to recover the structure of 3D objects. Furthermore, the formulation leads to an efficient implementation that can considerably reduce the computational load and it is therefore more adapted to real-time robotic applications. Experiments with simulated and real data validate the approach for deformable object visual tracking and 3D structure estimation. The computational efficiency is also compared to standard methods.

Existing visual servoing techniques which do not need metric information require, on the other hand, prior knowledge about the object's shape and/or the camera's motion. In this paper, we propose a new visual servoing technique which does not require any of them. The method is direct in the sense that: the intensity value of all pixels is used (i.e. we avoid the feature extraction step which introduces errors); and that the proposed control error as well as the control law are fully based on image data (i.e. metric measures are neither required nor estimated). Besides not relying on prior information, the scheme is robust to large errors in the camera's internal parameters. We provide the theoretical proofs that the proposed task function is locally isomorphic to the camera pose, that the approach is motion- and shape-independent, and also that the derived control law ensures local asymptotic stability. Furthermore, the proposed control error allows for simple, smooth, physically valid, singularity-free path planning, which leads to a large domain of convergence for the servoing. The approach is validated through various results using objects of different shapes, large initial displacements as well as large errors in the camera's internal parameters.

International audience

In the last decade, research on vision-based robot control has been concentrated on two main issues: the narrow field of view of the conventional camera and the model dependency of the standard visual servoing approaches. In this paper, we propose a simple and elegant solution to these issues. To enlarge the field of view of the cameras, we use omnidirectional cameras. And to overcome the model dependency problem, we propose a new visual servoing method for omnidirectional cameras that does not need any measure of the 3D structure of the observed target with respect to which the visual servoing is performed. Only visual information measured from the reference and the current images are needed in order to compute a task function isomorphic to the camera pose and to compute the control law to be applied to the robot. We provide the theoretical proof of the existence of the isomorphism and the theoretical proof of the local stability of the control law

The objective of this paper is to propose a new homography-based approach to image-based visual servoing. The visual servoing method does not need any measure of the 3D structure of the observed target. Only visual information measured from the reference and the current image are needed to compute the task function (isomorphic to the camera pose) and the control law to be applied to the robot. The control law is designed in order to make the task function converge to zero. We provide the theoretical proof of the existence of the isomorphism between the task function and the camera pose and the theoretical proof of the local stability of the control law. The experimental results, obtained with a 6 d.o.f. robot, show the advantages of the proposed method with respect to the existing approaches

This work proposes a new vision-based framework to control a robot within model-free large-scale scenes, where the desired pose has never been attained beforehand. Thus, the desired image is not available. It is important to remark that existing visual servoing techniques cannot be applied in this context. The rigid, unknown scene (i.e. the metric model is also not available) is represented as a collection of planar regions, which may leave the field-of-view continuously as the robot moves toward its distant goal. Hence, a novel approach to detect new planes that enter the field-of-view, which is robust to large camera calibration errors, is then deployed here. In fact, it is well-known that representing the scene as composed by planes, the estimation processes are improved in terms of accuracy, stability, and rate of convergence. This extended 3D vision-based control technique is also based on an efficient second-order method for plane-based tracking and pose reconstruction. The framework is validated by using simulated data with artificially created scenes as well as with real images, and accurate navigation tasks are shown

This papers deals with the problem of tracking a piecewise-planar scene in a video sequence and, at the same time, with the problem of estimating accurately the 3D displacement of the camera for robotic applications. A new approach to the problem is proposed and two noticeable contributions are given. Firstly, the explicit dependency between the 2D image transformation parameters (a homography for each plane) and the 3D camera displacement parameters is computed. Secondly, a second-order optimization algorithm is proposed. The second-order optimization considerably increases the convergence domain and the convergence rate of standard first-order optimization algorithms while having an almost equivalent computational complexity

The objective of this paper is to propose an innovative visual servoing method in order to improve the 3D reconstruction of objects for quantitative measurements. The method uses a Stereovision system that allows us to obtain several shots of an object, at regular intervals according to a predefined trajectory. In our case, the stereo rig is equipped with two different cameras, consequently the intrinsic parameters are not the same. The first one is fixed while the other one is mounted on a pan and tilt. Simulation and preliminary experiments in laboratory conditions shows the validity of our approach

The goal of this paper is to present an overview of robust estimation techniques with a special focus on robotic vision applications. In this particular context, constraints due computation time have to be considered in the choice of the estimation algorithm. Among the numerous techniques proposed in the literature to obtained robust estimation we have, not being exhaustive, Hough transform, RANSAC (Random Sample Consensus), the LMedS (Least Median of Squares), the M-estimators, etc. In this overview, we describe these various approaches in the light of a simple example. Finally, we illustrate the use of robust estimation techniques by various examples in real-time robot vision

This paper presents a parametric approach for tracking piecewise planar scenes with central catadioptric cameras (including perspective cameras). We extend the standard notion of homography to this wider range of devices through the unified projection model on the sphere. We avoid unwarping the image to a perspective view and take into account the non-uniform pixel resolution specific to non-perspective central catadioptric sensors. The homography is parametrised by the Lie algebra of the special linear group SL(3) to ensure that only eight free parameters are estimated. With this model, we use an efficient second-order minimisation technique leading to a fast tracking algorithm with a complexity similar to a first-order approach. The developed algorithm was tested on the estimation of the displacement of a mobile robot in a real application and proved to be very precise

In this paper we present an analysis of 3D line projections for central catadioptric cameras from a projective perspective. Most algorithms consider the projection of lines as general conics in the image plane with five degrees of freedom. However, in the calibrated case, only two parameters are needed to represent lines. We describe methods to obtain fast extraction and estimation algorithms. We then explain how classical edge-tracking algorithms can be adapted to these sensors. To this avail, we introduce two parametric equations for lines in central catadioptric images. We then propose a minimal representation for the euclidean transformation in the structure from motion problem and introduce possible metrics between a point and a central catadioptric line. These metrics are evaluated on simulated data. The structure from motion algorithm, from the line extraction process to the tracking and the reconstruction is tested on a real sequence

This work presents a method for segmenting image patches which correspond to planar regions in the scene. The method consists of an efficient and robust solution for detecting multiple planar regions in a global optimal sense. Moreover, in contrast with existing techniques which also work on intensity images, neither assumptions about the scene are made nor heuristic hypotheses are formulated. More specifically, the proposed method is based on a systematic, progressive voting procedure from the solution of a linear system, which exploits the two-view geometry. Hence, besides avoiding intermediary depth maps, the progressive mechanism together with such a convergence mapping drastically reduce the computational and storage complexities of the approach. Results from both synthetic and real-world scenes in different scenarios and under various kinds of strong noise confirm its effectiveness and robustness against large camera calibration errors and to the presence of outliers

This paper concerns the continuity of vision based robot control law. Recently, we have proposed a way of allowing the temporary disappearance of image features during the control task based on weighted features. Futher more, we have redefined the camera invariant visual servoing approach in order to take into account the change of image features when zooming in or out during a positioning task. In this paper, we study ho to select some parameters of the weight function and we propose a local stability analysis of the invariant visual servoing approach with weighted features. Finally, experimental results demonstrate the improvements that can be achieved in the performance of vision-based control task.

In this paper, we present a complete system for car platooning using visual tracking. The visual tracking is achieved by directly estimating the projective transformation (in our case a homography) between a selected reference template attached to the leading vehicle and the corresponding area in the current image. The relative position and orientation of the servoed car with regard to the leading one is computed by decomposing the homography. The control objective is stated in terms of path following task in order to cope with the non-holonomic constraints of the vehicles.

In the recent past, the visibility problem in vision-based control has been widely investigated. The proposed solutions generally have a common goal: to always keep the object in the camera's field of view during the visual servoing. Contrary to this solution, we propose a new approach based on the concept of allowing the changes of visibility in image features during the control task. To this aim, the camera invariant visual-servoing approach has been redefined in order to take into account the changes of visibility in image features. A new smooth task function using weighted features is presented, and a continuous control law is obtained starting from it by imposing its exponential decrease to zero. Furthermore, the local stability analysis of the invariant visual-servoing approach with weighted features is presented. Finally, this promising way of dealing with the visibility issue has been successfully tested with an eye-in-hand robotic system.

In this paper, several vision-based robot control methods are classified following an analogy with well known minimization methods. Comparing the rate of convergence between minimization algorithms helps us to understand the difference of performance of the control schemes. In particular, it is shown that standard vision-based control methods have in general low rates of convergence. Thus, the performance of vision-based control could be improved using schemes which perform like the Newton minimization algorithm that has a high convergence rate. Unfortunately, the Newton minimization method needs the computation of second derivatives that can be ill-conditioned causing convergence problems. In order to solve these problems, this paper proposes two new control schemes based on efficient second-order minimization techniques.

The tracking algorithm presented in this paper is based on minimizing the sum-of-squared-difference between a given template and the current image. Theoretically, amongst all standard minimization algorithms, the Newton method has the highest local convergence rate since it is based on a second-order Taylor series of the sum-of-squared-differences. However, the Newton method is time consuming since it needs the computation of the Hessian. In addition, if the Hessian is not positive definite, convergence problems can occur. That is why several methods use an approximation of the Hessian. The price to pay is the loss of the high convergence rate. The aim of this paper is to propose a tracking algorithm based on a second-order minimization method which does not need to compute the Hessian.

This paper presents a new method for the self-calibration of the lens distortion of a zooming camera, which appears for short focal lengths. The proposed technique does not need any special calibration pattern nor any prior knowledge about the environment. The key idea is to match points between a distorted image and an undistorted image taken at different resolutions. A new method for automatically matching points in the two images is proposed. The scale factor between the images is not needed for the matching algorithm. Matched points are used to compute invariants to the pinhole camera parameters. Then, lens distortion parameters are estimated in order to obtain the same invariants in both images. This approach is well suited to autonomous robotic vision applications. In fact, the self-calibration of the camera is done before moving the robot. Experiment with ground truth and tests on real images provide good results.

This paper deals with the problem occurring when features go in or out of the image during the visual servoing task. The appearance/disappearance of image features during the control task produces discontinuities in the control law that affect the performance of the system. In this paper, we propose a solution in order to avoid these discontinuities by the use of weighted image features. In particular, we redefine the camera invariant visual servoing approach In order to take into account the change of image features when zooming in or out during a positioning task. Simulations and experimental results demonstrate the improvements that can be obtained in the performance of the vision-based control task.

Standard bundle adjustment techniques for Euclidean reconstruction consider camera intrinsic parameters as unknowns in the optimization process. Obviously, the speed of an optimization process is directly related to the number of unknowns and to the form of the cost function. The scheme proposed in this paper differs from previous standard techniques since unknown camera intrinsic parameters are not considered in the optimization process. Considering fewer unknowns in the optimization process produces a faster algorithm, which is more adapted to time-dependent applications such as robotics. Computationally expensive metric reconstruction, using for example several zooming cameras, considerably benefits from an intrinsics-free bundle adjustment.

This paper concerns the stability analysis of image-based visual servoing methods with respect to uncertainties on the 3D parameters introduced in the central catadioptric interaction matrix. Motivated by the growing interest for omnidirectional sensors on robotic applications and particularly on vision-based control, we extend recent results obtained for conventional cameras to the entire class of central catadioptric systems. In this paper, it is shown that with such sensors extreme care must be taken when approximating 3D parameters to ensure stability of the image based control law.

This paper presents a new visual servoing scheme which is invariant to changes in camera-intrinsic parameters. Current visual servoing techniques are based on the learning of a reference image with the same camera used during the servoing. With the new method, it is possible to position a camera (with eventually varying intrinsic parameters), with respect to a nonplanar object, given a "reference image" taken with a completely different camera. The necessary and sufficient conditions for the local asymptotic stability show that the control law is robust in the presence of large calibration errors. Local stability implies that the system can accurately track a path in the invariant space. The path can be chosen such that the camera follows a straight line in the Cartesian space. Simple sufficient conditions are given in order to keep the tracking error bounded. This promising approach has been successfully tested with an eye-in-hand robotic system.

This paper concerns the stability analysis of image-based visual servoing with respect to uncertainties on the depths of the observed object. In the recent past, research on image-based visual servoing has been concentrated on potential problems of stability and on robustness with respect to camera calibration errors. Only little attention, if any, has been devoted to the robustness of image-based visual servoing to depth estimation errors. It is generally believed that a rough approximation of the depth distribution is sufficient to ensure the stability of the control law. In this paper, we prove that the robustness domain is not so wide and that an extern care must be taken when approximating the depth distribution.

This paper presents a new approach to active affine reconstruction without an exact knowledge of the robot's kinematic model nor camera intrinsic parameters. Affine reconstruction from perspective image pairs is easy if the motion of the camera between the two images is a pure translation. If the robot is not well calibrated, a pure translation achieved with an open loop control would lead to a bias in the reconstruction. The problem can be solved by using a 2 1/2 D visual servoing technique in order to close the loop and control the camera trajectory. The affine reconstruction is equivalent to the estimation of the depths of the 3D points of the scene. Thus, affine reconstruction is very useful to implement several visual servoing approaches which need an estimation of the depths.

This paper deals with the robust tracking of features extracted from a sequence of images taken with an uncalibrated camera mounted on a mobile robot. Unlike most vision systems, the 3D structure of the observed objects is completely unknown. Thus, position-based visual servoing cannot be used. Similarly, one must be careful when using image-based visual servoing since the depths of the features are unknown. On the other hand, 2 1/2 D visual servoing can easily deal with unknown environments since it is only based on projective reconstruction. In order to obtain a good projective reconstruction for a safe vision-based control, we propose a multi-scale real-time approach to extract robust features. Experiments show that our algorithm can be used in robotics applications when images are noisy and uncontrolled perturbations can break the continuity of the robot motion.

This paper concerns the stability analysis of a new vision-based control which is invariant to camera intrinsic parameters. The necessary and sufficient conditions for the local asymptotic stability show that the control law is robust in the presence of large calibration errors. Local stability implies that the system can accurately track a path in the invariant space. Even if the camera is uncalibrated, the path can be chosen such that the camera follows a straight line in the Cartesian space. Simple sufficient conditions are given in order to keep the tracking error bounded.

In this paper we propose a new approach for visual-servoing with respect to a set of 3D straight lines. The main difference with respect to previous approaches is that the new scheme can be used with a zooming camera or even if the reference image has been learned with a different camera. The zoom is particularly useful in order to keep the visual features in the camera field of view and/or to bound their size in the image reducing the influence of noise on features extraction. Experiments with a zooming camera have validated the vision-based control law.

In this paper, we describe an efficient method to impose the constraints existing between the collineations between images which can be computed from a sequence of views of a planar structure. These constraints are usually not taken into account by multiview techniques in order not to increase the computational complexity of the algorithms. However, imposing the constraints is very useful since it allows a reduction of geometric errors in the reprojected features and provides a consistent set of collineations which can be used for several applications such as mosaicing, reconstruction, and self-calibration. In order to show the validity of our approach, this paper focus on self-calibration from unknown planar structures proposing a method exploiting the consistent set of collineations. Our method can deal with an arbitrary number of views and an arbitrary number of planes and varying camera internal parameters. However, for simplicity, this papers will only discuss the case with one plane in several views. The results obtained with synthetic and real data are very accurate and stable even when using only few images.

This paper concerns the stability analysis of a new class of model-free visual servoing methods. These methods are "model-free" since they are based on the estimation of the relative camera orientation between two views of an object without knowing its 3-D model. The visual servoing is decoupled by controlling the rotation of the camera separately from the rest of the system. The way the remaining degrees of freedom are controlled differentiates the methods within the class. For all the methods of the class, the robustness with respect to both camera and hand-eye calibration errors can be analytically studied. In some cases, necessary and sufficient conditions can be found not only for the local asymptotic stability but also for the global asymptotic stability. In the other cases, simple conditions on the calibration errors are sufficient to ensure the global asymptotic stability of the control law. In addition to the theoretical proof of the stability, the experimental results prove the validity of the control strategy proposed in the paper.

Most visual servoing schemes are based on the learning of a reference image with the same camera used for servoing. The scheme proposed in this paper differs from previous ones since it is independent on the camera used for learning. With the new scheme it is possible to position a camera (with eventually varying intrinsic parameters) with respect to a non-planar object given a reference image taken with a completely different camera. This promising approach has been successfully tested with an eye-in-hand robotic system.

In this paper, we propose a new self-calibration technique for cameras with changing zoom observing only a planar structure. The method does not need any metric or topologic knowledge about the structure since it is based on the estimation of the collineations existing between several views of a plane (thus only image correspondences are needed). The constraints existing between all the collineations are imposed using a very simple and efficient technique which does not need the solution of a complex optimisation problem. Finally, even if the structure of the plane is unknown it must be the same for all the images and this provides some constraints which allow the recovering of the varying focal length.

We describe in this paper potential problems that may appear in image-based visual servoing when the initial camera position is far away from its desired position. We show by concrete examples that local minima or a singularity of the image Jacobian can be reached during the servoing. We then recall recent results obtained to avoid these drawbacks. It consists in combining visual features obtained directly from the image, and position-based features. This approach, called 2 1/2 D visual servoing, also provides supplementary advantages with regard to the mean the features are combined with.

We present a complete system for segmenting, matching and tracking planar contours for use in visual servoing. Our system can be used with arbitrary contours of any shape and without any prior knowledge of their models. The system is first shown the target view. A selected contour is automatically extracted and its image shape is stored. The robot and object are then moved and the system automatically identifies the target. The matching step is done together with the estimation of the homography matrix between the two views of the contour. Then, a 2 1/2 D visual servoing technique is used to reposition the end-effector of a robot at the target position relative to the planar contour. The system has been successfully tested on several contours with very complex shapes such as leaves, keys and the coastal outlines of islands.

In this paper, the classical visual servoing techniques have been extended to the use of several cameras observing different parts of an object. The multi-camera visual servoing has been designed as a part of the task function approach. The particular choice of the task function allows one to simplify the design of the control law and the stability analysis. A positioning task on a cumbersome object has been realized using 2D and 2 1/2 D visual servoings with two cameras, mounted on a manipulator robot, and observing two different parts of the object.

In this paper we describe a method to compute the collineation matrix between two unmatched images of an unknown planar contour described using a B-spline snake. The two images of the contour are matched and the collineation matrix is used to servo a camera mounted on the robot end-effector using a 2 1/2 D visual servoing technique. The experimental results, obtained using common planar objects, show that our method give very good results and allow the robot end-effector to be positioned with a great precision. 1 Introduction The visual servoing scheme of robot manipulators can be divided in three steps. In the first off-line learning step, the reference image of the object corresponding to a desired position of the robot is acquired and some image features are extracted. In general, objects are represented by free-form curves, i.e., arbitrary space curves of the type found in practice. A curve is usually described as a set of chained points. The reference image can be obtaine...

We propose an approach to vision-based robot control, called 2 1/2 D visual servoing, which avoids the respective drawbacks of classical position-based and image-based visual servoing. Contrary to the position-based visual servoing, our scheme does not need any geometric three-dimensional model of the object. Furthermore and contrary to image-based visual servoing, our approach ensures the convergence of the control law in the whole task space. 2 1/2 D visual servoing is based on the estimation of the partial camera displacement from the current to the desired camera poses at each iteration of the control law. Visual features and data extracted from the partial displacement allow us to design a decoupled control law controlling the six camera DOFs. The robustness of our visual servoing scheme with respect to camera calibration errors is also analyzed: the necessary and sufficient conditions for local asymptotic stability are easily obtained. Then, due to the simple structure of the system, sufficient conditions for global asymptotic stability are established. Finally, experimental results with an eye-in-hand robotic system confirm the improvement in the stability and convergence domain of the 2 1/2 D visual servoing with respect to classical position-based and image-based visual servoing.

In this paper we propose a new vision-based robot control approach halfway between the classical position-based and image-based visual servoings. It allows to avoid their respective disadvantages. The homography between some planar feature points extracted from two images (corresponding to the current and desired camera poses) is computed at each iteration. Then, an approximate partial-pose, where the translational term is known only up to a scale factor, is deduced, from which can be designed a closed-loop control law controlling the six camera DOF. Contrarily to the position-based visual servoing, our scheme does not need any geometric 3D model of the object. Furthermore and contrarily to the image-based visual servoing, our approach ensures the convergence of the control law in all the task space.

We propose a simple and efficient control algorithm that combines visual servo control and force feedback within the impedance control approach. The control scheme involves, at the low level, a position based impedance controller with an external force sensor feedback loop. The reference trajectory fed to this impedance controller is generated online by a vision based control loop. In spite of its simplicity, this approach provides satisfactory experimental behavior. Peg in hole insertion experiments involving large initial errors, are performed using a 7 axis robot manipulator without any computation of the peg trajectory.

In this paper, the robustness of a new visual servoing scheme with respect to camera calibration errors is analyzed. This scheme, called 2D 1/2 visual servoing, is based on the estimation of the partial camera displacement from the current to the desired camera poses at each iteration of the control law. Visual features and data extracted from the partial displacement allow as to design a decoupled control law controlling the six camera DOF. The necessary and sufficient conditions for local asymptotic stability in presence of camera calibration errors are easily obtained. Then, thanks to the simple structure of the system, sufficient conditions for global asymptotic stability are proposed. Finally, experimental results show the validity of our approach and its robustness not only with respect camera calibration errors but also to robot calibration errors.