The increasing complexity of urban environments has underscored the potential of effective collective perception systems. To address these challenges, we present the CoopScenes dataset, a large-scale, multi-scene dataset that provides synchronized sensor data from both the ego-vehicle and the supporting infrastructure. The dataset provides 104 minutes of spatially and temporally synchronized data at 10Hz, resulting in 62,000 frames. It achieves competitive synchronization with a mean deviation of only 2.3 ms. Additionally the dataset includes a novel procedure for precise registration of point cloud data from the ego-vehicle and infrastructure sensors, automated annotation pipelines, and an open-source anonymization pipeline for faces and license plates. Covering nine diverse scenes with 100 maneuvers, the dataset features scenarios such as public transport hubs, city construction sites, and high-speed rural roads across three cities in the Stuttgart region, Germany. The full dataset amounts to 527 GB of data and is provided in the .4mse format, making it easily accessible through our comprehensive development kit. By providing precise, large-scale data, CoopScenes facilitates research in collective perception, real-time sensor registration, and cooperative intelligent systems for urban mobility, including machine learning-based approaches.

In mini-invasive surgery, the laparoscopic ultra-sound probe is visible in the laparoscopic image. We address the problem of estimating the probe pose with respect to the laparoscope without using markers and additional sensors. We propose the first method using a single standard laparoscopic monocular RGB image. It is robust, initialization-free and runs at 10 fps, thus forming a promising tool to improve robotic and augmented reality-based surgery.

Visuotactile sensors have gained attention during the last years in robotics because they are able to reconstruct with high precision the 3D contact shape (or mesh) between the robotic fingers and the object. A new slip detection and avoidance algorithm is proposed based on the dynamic variation of the height of the contact mesh. Firstly, the contact mesh is reconstructed in real time by applying a neural network that estimates normal vectors from color variations along all the pixels of the images recorded by the camera inside the tactile sensor. The contact mesh corresponding to this height map is used for detecting slip with higher success rates in comparison with previous approaches based on machine learning methods directly applied to contact images or the analysis of markers integrated into the sensor’s surface. The proposed algorithm is validated experimentally in multiple directions not only for different types of objects (volumetric/planar/linear, deformable/rigid) but also with different resolutions of the contact mesh.

In this work, we present a novel approach for general object segmentation from a monocular image, eliminating the need for manually labeled training data and enabling rapid, straightforward training and adaptation with minimal data. Our model initially learns from LiDAR during the training process, which is subsequently removed from the system, allowing it to function solely on monocular imagery. This study leverages the concept of the Stixel-World to recognize a medium level representation of its surroundings. Our network directly predicts a 2D multi-layer Stixel-World and is capable of recognizing and locating multiple, superimposed objects within an image. Due to the scarcity of comparable works, we have divided the capabilities into modules and present a free space detection in our experiments section. Furthermore, we introduce an improved method for generating Stixels from LiDAR data, which we use as ground truth for our network.

In this article, we propose a general unified tracking-servoing approach for controlling the shape of elastic deformable objects using robotic arms. Our approach works by forming a lattice around the object, binding the object to the lattice, and tracking and servoing the lattice instead of the object. This makes our approach have full control over the deformation of elastic deformable objects of any general form (linear, thin-shell, and volumetric) in 3-D space. Furthermore, it decouples the runtime complexity of the approach from the objects' geometric complexity. Our approach is based on the as-rigid-as-possible deformation model. It requires no mechanical parameter of the object to be known and can drive the object toward desired shapes through large deformations. The inputs to our approach are the point cloud of the object's surface in its rest shape and the point cloud captured by a 3-D camera in each frame. Overall, our approach is more broadly applicable than existing approaches. We validate the efficiency of our approach through numerous experiments with elastic deformable objects of various shapes and materials (paper, rubber, plastic, and foam).

Dynamic movement primitives (DMPs) provide an effective method of learning manipulation skills from human demonstration. DMPs can be especially useful for imitating industrial manipulation tasks which are performed by humans and are difficult to model, for instance, deformable object manipulation. In this work the effectiveness of a conventional Cartesian space DMP is enhanced using a compact and efficient representation of dual quaternions (DQ). We demonstrate that our DQ based DMP learning approach that utilizes the geometrical meaning of screw-based kinematics, outperforms traditional decoupled task-space DMPs in terms of accuracy during learning in certain situations. Our DMP formulation affords two additional applications: (1) Filter the noisy and irregular sensing of human demonstration; (2) Limit the robotic manipulator's task-space velocity during teleoperation, thus improving the safety of the robot and the environment. The learning and filtering strategies are validated on a bimanual robotic system and a motion capture system. We demonstrate the effectiveness of DMP based manipulation of deformable object by learning a bimanual deformation trajectory and then using it to perform the same task in new scenarios.

In this paper, we propose a formation control system for deforming and transporting simultaneously a de-formable object with a team of robots, modeled with double-integrator dynamics. The goal is to reach a target configuration, defined as a combination of shape, scale, orientation and position of the formation. We augment this controller with a set of control barrier functions (CBFs). The CBFs allow us to satisfy fundamental constraints for the success of the task: avoidance of agent-to-agent, agent-to-obstacle and object-to-obstacle collisions, and of excessive stretching. We test the performance of our proposal in different simulation scenarios.

We propose a new approach to control the shape of deformable objects with robots. Specifically, we consider a fixed-length elastic linear object lying on a 2D workspace. Our main idea is to encode the object's deformation behavior in an offline constant Jacobian matrix. To derive this Jacobian, we use geometric deformation modeling and combine recent work from the fields of deformable object control and multirobot systems. Based on this Jacobian, we then propose a robotic control law that is capable of driving a set of shape features on the object toward prescribed values. Our contribution relative to existing approaches is that at run-time we do not need to measure the full shape of the object or to estimate/simulate a deformation model. This simplification is achieved thanks to having abstracted the deformation behavior as an offline model. We illustrate the proposed approach in simulation and in experiments with real deformable linear objects.

Most deformable object manipulation tasks still rely on skillful human operators. To automate such tasks, a robotic system should not only be able to deform an object to a desired shape but also servo its deformation along a specific path towards the desired shape. We propose a shape servoing control scheme to automate such tasks. Our scheme controls the deformation trajectory towards the desired shape by imposing task-focused convergence constraints. The constraints impose how fast the different regions of the object converge to the desired shape. Integrating such a behavior in shape servoing forms our main contribution. Experiments, carried out on rubber layer assembly tasks, show that our control scheme outperforms a state-of-the-art shape servoing scheme.

The planning of robotic manipulation and grasping tasks depends on the reconstruction of the 3D object’s shape. Most of the existing 3D object reconstruction methods are based on visual sensing that are limited due to the lack of the object’s occluded side information. The goal of this paper is to overcome these limitations and improve the 3D objects’ reconstruction by adding the tactile sensing to the visual data. In this paper, a novel multi-modal (visual and tactile) semi-supervised generative model is presented to reconstruct the complete 3D object’s shape using a single arbitrary depth-view and a single dexterous-hand’s touch. The presented approach takes the strength of the autoencoder and generative networks to provide an end-to-end trainable model with high generalization ability. The 3D voxel grids of the depth and tactile data are the only requirements of the proposed model to predict a high resolution voxel grids of 643for the incomplete shape. This research generates its tactile dataset based on the kinematic model of the shadow dexterous hand. The developed dataset has aligned depth, tactile and ground truth voxel grids of different resolutions (403, 643and 1283) from different camera views. Experimental results show that the proposed multi-modal model outperforms other state-of-the-art methods.

We address the monocular visual shape servoing problem. This pushes the challenging visual servoing problem one step further from rigid object manipulation towards deformable object manipulation. Explicitly, it implies deforming the object towards a desired shape in 3D space by robots using monocular 2D vision. We specifically concentrate on a scheme capable of controlling large isometric deformations. Two important open subproblems arise for implementing such a scheme. (P1) Since it is concerned with large deformations, perception requires tracking the deformable object's 3D shape from monocular 2D images which is a severely underconstrained problem. (P2) Since rigid robots have fewer degrees of freedom than a deformable object, the shape control becomes underactuated. We propose a template-based shape servoing scheme in which we solve these two problems. The template allows us to both infer the object's shape using an improved Shape-from-Template algorithm and steer the object's deformation by means of the robots' movements. We validate the scheme via simulations and real experiments.

Addressing the problem of enclosing and tracking a target requires multiple agents with adequate motion strategies. We consider a team of unicycle robots with a standard camera on board. The robots must maintain the desired enclosing formation while dealing with their nonholonomic motion constraints. The reference formation trajectories must also guarantee permanent visibility of the target by overcoming the limited field of view of the cameras. In this article, we present a novel approach to characterize the conditions on the robots' trajectories taking into account the motion and visual constraints. We also propose online and offline motion planning strategies to address the constraints involved in the task of enclosing and tracking the target. These strategies are based on maintaining the formation shape with variable size or, alternatively, on maintaining the size of the formation with flexible shape.

We present a novel method to control the relative positions of the members of a robotic team. The application scenario we consider is the cooperative manipulation of a deformable object in 2D space. A typical goal in this kind of scenario is to minimize the deformation of the object with respect to a desired state. Our contribution, then, is to use a global measure of deformation directly in the feedback loop. In particular, the robot motions are based on the descent along the gradient of a metric that expresses the difference between the team's current configuration and its desired shape. Crucially, the resulting multirobot controller has a simple expression and is inexpensive to compute, and the approach lends itself to analysis of both the transient and asymptotic dynamics of the system. This analysis reveals a number of properties that are interesting for a manipulation task: fundamental geometric parameters of the team (size, orientation, centroid, and distances between robots) can be suitably steered or bounded. We describe different policies within the proposed deformation-based control framework that produce useful team behaviors. We illustrate the methodology with computer simulations.

This paper proposes a flexible multi-layer and multi-controller architecture for a dynamic navigation in the formation of a group of autonomous vehicles in constrained environments. The main objectives of this architecture are to ensure reliable navigation in the formation of the vehicles and to guarantee the stable and smooth reconfiguration of the fleet shape. A precise review and analysis of the main used leader-follower modeling for the control of a fleet of autonomous vehicles is conducted. After highlighting their advantages and drawbacks, an appropriate leader-follower approach based on deformable shape is proposed. At each sample time, the leader's state (pose and velocity), defined as the main dynamic target, is taken as a reference to guide the overall fleet dynamic. In addition, an analytic formulation of the maximum linear and angular velocities of the leader is proposed in order to guarantee the asymptotic stability of the navigation in formation as well as the fleet reconfiguration phases (between different formation shapes). An important focus of this paper corresponds to the proposition of a reliable strategy for the fleet reconfiguration, according to the environmental context (when, for instance, obstacles are detected). The safety of the fleet is formally demonstrated using an appropriate reconfiguration matrix, which takes into account the vehicles' set-points inter-distances to avoid any inter-vehicles collisions. In addition, an estimation of the formation parameters, according to an authorized minimum distance between the vehicles, is given. Simulations and experiments in different scenarios are performed to demonstrate the flexibility, reliability, and efficiency of the proposed dynamic navigation of a fleet of vehicles in formation.

Precise robot manipulation of deformable objects requires an accurate and fast estimation of their shape as they deform. So far, visual sensing has been mostly used to solve this issue, but vision sensors are sensitive to occlusions, which might be inevitable when manipulating an object with robot. To address this issue, we present a modular pipeline to track the shape of a soft object in an online manner by coupling tactile sensing with a deformation model. Using a model of a tactile sensor, we compute the magnitude and location of a contact force and apply it as an external force to the deformation model. The deformation model then updates the nodal positions of a mesh that describes the shape of the deformable object. The proposed sensor model and pipeline, are evaluated using a Shadow Dexterous Hand equipped with BioTac sensors on its fingertips and an RGB-D sensor.

An emerging application of multirobot systems is the monitoring of a dynamic event. Here, the goal is to enclose and track a moving target by attaining a desired geometric formation around it. By considering a circular pattern configuration for the target enclosing, the multirobot system is able to perform full perception of the target along its motion. In the proposed system, the robots rely only on their onboard vision sensor without external input to complete the task. The key problem resides in overcoming the motion and visual constraints of the agents. In particular, differential-drive robots with limited sensing, that must maintain visibility of the moving target as it navigates in the environment, are considered. A novel approach to characterize the motion of the robots in the formation that allows to enclose and track the target while overcoming their limited field of view (FOV) is presented. The proposed approach is illustrated through simulations.

Conventional active incremental structure from motion (ISfM) schemes require a precise knowledge of the linear and angular velocities of the vision system to compute the three-dimensional structure of the observed scene. Furthermore, they are generally coupled and nonlinear, which makes the reconstruction inaccurate. In this paper, we present a novel active ISfM scheme to overcome these difficulties. It relies on two appropriate transformations: the first one allows efficient decoupling of the nonlinear model, whereas the second one transforms the decoupled model into the Brunovsky's linear form. We then show that a Luenberger's observer can be designed to estimate the unavailable states, making our scheme globally asymptotically convergent and robust to noise on measurements. We present simulated and experimental results to illustrate the performance of the described methodology.

This paper shows how to push an unknown object in the plane from an initial pose to a target pose with two cooperating mobile robots. On the object motion, we deliberately impose non-holonomic velocity constraint with pushing mobile robots. This yields smooth and efficient trajectories. Pushing manipulation is performed, for the first time, with a new uncalibrated image-based control scheme. This is achieved by transforming the image information to a scaled Euclidean space without using any metric information or calibration. Stability of the control law is also demonstrated.

Incremental Structure from Motion (SfM) algorithms require, in general, precise knowledge of the camera linear and angular velocities in the camera frame for estimating the 3D structure of the scene. Since an accurate measurement of the camera own motion may be a non-trivial task in several robotics applications (for instance when the camera is onboard a UAV), we propose in this paper an active SfM scheme fully independent from the camera angular velocity. This is achieved by considering, as visual features, some rotational invariants obtained from the projection of the perceived 3D points onto a virtual unitary sphere (unified camera model). This feature set is then exploited for designing a rotation-free active SfM algorithm able to optimize online the direction of the camera linear velocity for improving the convergence of the structure estimation task. As case study, we apply our framework to the depth estimation of a set of 3D points and discuss several simulations and experimental results for illustrating the approach.

This paper describes a new vision-based control method to drive a set of robots moving on the ground plane to a desired formation. As the main contribution, we propose to use multiple camera-equipped unmanned aerial vehicles (UAVs) as control units. Each camera views, and is used to control, a subset of the ground team. Thus, the method is partially distributed, combining the simplicity of centralized schemes with the scalability and robustness of distributed strategies. Relying on a homography computed for each UAV-mounted camera, our approach is purely image-based and has low computational cost. In the control strategy we propose, if a robot is seen by multiple cameras, it computes its motion by combining the commands it receives. Then, if the intersections between the sets of robots viewed by the different cameras satisfy certain conditions, we formally guarantee the stabilization of the formation, considering unicycle robots. We also propose a distributed algorithm to control the camera motions that preserves these required overlaps, using communications. The effectiveness of the presented control scheme is illustrated via simulations and experiments with real robots.

Over the past decade, image moments have been exploited in several visual servoing schemes for their ability to represent object regions, objects defined by contours or a set of discrete points. Moments have also been useful to achieve control decoupling properties and to choose a minimal number of features to control the whole degrees of freedom (DOFs) of a camera. However, the choice of moment-based features to control the rotational motions around the x-axis and y-axis simultaneously with the translational motions along the same axis remains a key issue. In this paper, we introduce new visual features computed from low-order “shifted moments invariant.” Importantly, they allow us 1) to define a unique combination of visual features to control the whole six DOFs of an eye-in-hand camera independently from the object shape and (2) to significantly enlarge the convergence domain of the closed-loop system.

Loop-closure detection, which is the ability to recognize a previously visited place, is of primary importance for robotic localization and navigation problems. We here introduce SAIL-MAP, a method for loop-closure detection based on vision only, applied to topological simultaneous localization and mapping (SLAM). Our method allows the matching of camera images using a novel saliency-based feature detector and descriptor. These features have been designed to benefit from the robustness to viewpoint change and image perturbations of bio-inspired saliency algorithms. Additionally, the same algorithm is used for the detector and descriptor. The results obtained on different large-scale data sets demonstrate the efficiency of the proposed solution for localization problems.

This paper proposes a complete procedure for the structural synthesis of dexterous hands. This procedure fuses the theories already developed for the structural synthesis of dexterous hands and parallel robots. Unlike others, this procedure allows one to synthesize any kind of dexterous hand with the desired structural design parameters: mobility, connectivity, overconstraint, and redundancy. Two examples of dexterous hands, which are synthesized to have 3 dof planar motion and 6 dof spatial motion, are also given.

This paper deals with pose estimation using an iterative scheme. We show that using adequate visual information, pose estimation can be performed iteratively with only three independent unknowns, which are the translation parameters. Specifically, an invariant to rotational motion is used to estimate the camera position. In addition, an adequate transformation is applied to the proposed invariant to decrease the nonlinearities between the variations in image space and 3-D space. Once the camera position is estimated, we show that the rotation can be estimated efficiently using two different direct methods. The proposed approach is compared against two other methods from the literature. The results show that using our method, pose tracking in image sequences and the convergence rate for randomly generated poses are improved.

We study the feature-based map merging problem in robot networks. Each robot observes the environment and builds a local map. Simultaneously, robots communicate and compute the global map of the environment; this communication is range-limited. We propose a dynamic strategy based on consensus algorithms that is fully distributed and does not rely on any particular communication topology. Robots reach consensus on the latest global map, using the increments between their previous and current local maps. Under mild connectivity conditions, our merging algorithm asymptotically converges to the global map. We give proofs of unbiasedness of this global map, at each step and robot. Our approach has been validated using real RGB-D images.

In image-based visual servoing, the choice of visual features has a strong influence on the performance of the control system. In the last decade, image-moments have been exploited in several visual servoing schemes for their ability to represent object region, object defined by contours or a set of discrete points. Despite the many recent advances, the choice of moment-based features to control the most critical Degrees Of Freedom (DOF), i.e those used to control the rotational motions around the x-axis and y-axis of the camera remains a key issue. In this paper, a new feature formula to control these critical DOF that does not depend on the object shape is proposed. The new features are computed from shifted moments and selected in order to provide nice invariant properties. Additionally, low order shifted moments can be exploited to minimize the impact of measurement noise on the control performances.

A topological mapping framework designed for omnidirectional images is presented. Omnidirectional images acquired by the robot are organized as places which are represented as nodes in the topological graph/map. Places are regions in the environment over which the global scene appearance of all acquired images is consistent. A hierarchical loop closure algorithm is proposed which quickly sifts through the places to retrieve the most similar places and another level of thorough similarity analysis is performed over the images belonging to the retrieved places. An Image similarity metric based on spatial shift of local image features across omnidirectional/panoramic image pairs is proposed. Newly proposed VLAD (Vector of Locally Aggregated Descriptors) descriptors have been used for loop closure at place and image levels. Accuracy and efficiency of our system are corroborated with experimental results on three publicly available datasets. It is shown that our approach achieves good loop closure recall rates even without using epi-polar geometry verification common among many other approaches.

This paper deals with pose estimation using an iterative scheme. We show that using adequate visual information, pose estimation can be performed in a decoupling the estimation of translation and rotation. More precisely, we show that pose estimation can be achieved iteratively as a function of only three independent unknowns, which are the translation parameters. An invariant to rotational motion is used to estimate the camera position. Once the camera position is estimated, we show that the rotation can be estimated efficiently using a direct method. The proposed approach is compared against two classical methods from the literature. The results show that using our method, pose tracking in image sequences and the convergence rate for randomly generated poses are improved.

This paper deals with reactive and flexible humanlike autonomous vehicle navigation. A human driver reactively guides his vehicle, performing a smooth trajectory within the roads limits until reaching the defined goal. To obtain a similar behavior with an unmanned ground vehicle (UGV), this paper proposes a flexible control law to drive a vehicle towards desired static or dynamic targets based on a novel definition of control variables and Lyapunov stability analysis. Moreover, a target assignment strategy, combined with an appropriate sigmoid function, that allow to perform smooth, flexible and safe vehicle navigation through successive waypoints is presented. The stability of the proposed control strategy is proved according to Lyapunov synthesis. Simulations and experiments are performed in different cases to demonstrate the reliability and efficiency of the control strategy.

This paper deals with the design of a generic visual tracking algorithm suitable for a large class of camera (single viewpoint sensors). It is based on the estimation of the relationship between observations and motion on the sphere. This is efficiently achieved using a kernel-based regression function on a generic linearly-weighted sum of non-linear basis functions. We also present two set of experiments. The first one shows the efficiency of our algorithm through the tracking in video sequences acquired with three types of cameras (conventional, dioptric-fisheye and catadioptric). The real-time performances will be shown by tracking one or several planes. The second set of experiments presents an application of our tracking algorithm to visual servoing with a fisheye camera.

Most of the existing appearance based topological mapping algorithms produce dense topological maps in which each image stands as a node in the topological graph. Sparser maps can be built by representing groups of visually similar images as nodes of a topological graph. In this paper, we present a sparse topological mapping framework which uses Image Sequence Partitioning (ISP) techniques to group visually similar images as topological graph nodes. We present four different ISP techniques and evaluate their performance. In order to take advantage of the afore mentioned maps, we make use of Hierarchical Inverted Files (HIF) which enable efficient hierarchical loop closure. Outdoor experimental results demonstrating the sparsity, efficiency and accuracy achieved by the combination of ISP and HIF in performing loop closure are presented.

A central clue for implementation of visual memory based navigation strategies relies on efficient point matching between the current image and the key images of the memory. However, the visual memory may become out of date after some times because the appearance of real-world environments keeps changing. It is thus necessary to remove obsolete information and to add new data to the visual memory over time. In this paper, we propose a method based on short-term and long term memory concepts to update the visual memory of mobile robots during navigation. The results of our experiments show that using this method improves the robustness of the localization and path-following steps.

This paper addresses the problem of visual control of a set of mobile robots. In our framework, the perception system consists of an uncalibrated flying camera performing an unknown general motion. The robots are assumed to undergo planar motion considering nonholonomic constraints. The goal of the control task is to drive the multirobot system to a desired rendezvous configuration relying solely on visual information given by the flying camera. The desired multirobot configuration is defined with an image of the set of robots in that configuration without any additional information. We propose a homography-based framework relying on the homography induced by the multirobot system that gives a desired homography to be used to define the reference target, and a new image-based control law that drives the robots to the desired configuration by imposing a rigidity constraint. This paper extends our previous work, and the main contributions are that the motion constraints on the flying camera are removed, the control law is improved by reducing the number of required steps, the stability of the new control law is proved, and real experiments are provided to validate the proposal.

This paper addresses the problem of visual control of a set of mobile robots. In our framework, the perception system consists of a calibrated flying camera looking downward to the mobile robots. The robots are assumed to undergo planar motion considering nonholonomic constraints. The goal of the task is to control the multi-robot system to a desired configuration relying solely on visual information given by the flying camera. The desired multi-robot configuration is defined with an image of the set of robots in that configuration. Then, any arbitrary configuration can be easily defined by this image without any additional information. As contribution, a new image-based control scheme is presented relying on the homography induced by the multi-robot system to lead the robots to the desired configuration. The stability of the control law is analyzed and simulations are provided to illustrate the proposal.

Vision has often been considered as not suitable for dynamic control of robots. The experimental results presented in this paper show that it is possible to perform better with a vision based dynamic control than with a model-based control. These results were obtained using a Cartesian computed torque control fed back, without any joint sensing, by a novel Cartesian pose and velocity estimator. The latter is designed as a virtual visual servoing scheme based on sequential acquisition of sub-images and a constant acceleration motion assumption.

In this paper, we propose a visual path following control scheme for wheeled mobile robots based on the epipolar geometry. The control law only requires the position of the epipole computed between the current and target views along the sequence of a visual memory. The proposed approach has two main advantages: explicit pose parameters decomposition is not required and the rotational velocity is smooth or eventually piece-wise constant avoiding discontinuities that generally appear when the target image changes. The translational velocity is adapted as required for the path and the approach is independent of this velocity. Furthermore, our approach is valid for all cameras obeying the unified model, including conventional, central catadioptric and some fisheye cameras. Simulations as well as real-world experiments with a robot illustrate the validity of our approach.

This paper concerns the stability analysis of image-based visual servoing control laws with respect to uncertainties on the 3-D parameters needed to compute the interaction matrix for any calibrated central catadioptric camera. In the recent past, research on image-based visual servoing has been concentrated on potential problems of stability and on robustness with respect to camera-calibration errors. Only little attention, if any, has been devoted to the robustness of image-based visual servoing to estimation errors on the 3-D structure. It is generally believed that a rough approximation of the 3-D structure is sufficient to ensure the stability of the control law. In this paper, we prove that this is not always true and that an extreme care must be taken when approximating the depth distribution to ensure stability of the image-based control law. The theoretical results are obtained not only for conventional pinhole cameras but for the entire class of central catadioptric systems as well.

This paper proposes a generic decoupled image-based control scheme for cameras obeying the unified projection model. The scheme is based on the spherical projection model. Invariants to rotational motion are computed from this projection and used to control the translational degrees of freedom (DOFs). Importantly, we form invariants that decrease the sensitivity of the interaction matrix to object-depth variation. Finally, the proposed results are validated with experiments using a classical perspective camera as well as a fisheye camera mounted on a 6-DOF robotic platform.

This paper extends the recent work proposed in [21]. In this work, it has been noted that three visual features (to control three degrees of freedom) obtained from the spherical projection of 3D spheres allows nice decoupling properties and global stability. However, even if such an approach is theoretically attractive, it is limited by a major practical issue since spherical objects have to be observed while only three degrees of freedom can be controlled. In this paper, we show that similar properties can be obtained by observing a set of points. The basic idea is to build a virtual 3D sphere from two 3D points and to analyse its related spherical projection. Furthermore, to control the six degrees of freedom a 2D 1/2 control scheme is proposed which allows us to fully decouple rotational motions from translational motions.

In this paper a generic decoupled imaged-based control scheme for calibrated cameras obeying the unified projection model is proposed. The proposed decoupled scheme is based on the surface of object projections onto the unit sphere. Such features are invariant to rotational motions. This allows the control of translational motion independently from the rotational motion. Finally, the proposed results are validated with experiments using a classical perspective camera as well as a fisheye camera mounted on a 6 dofs robot platform.

This paper presents a vision-based navigation strategy for a vertical take-off and landing (VTOL) unmanned aerial vehicle (UAV) using a single embedded camera observing natural landmarks. In the proposed approach, images of the environment are first sampled and stored as a set of ordered key images (visual path) and organized providing a visual memory of the environment. The robot navigation task is then defined as a concatenation of visual path subsets (called visual route) linking the current observed image and a target image belonging to the visual memory. The UAV is controlled to reach each image of the visual route using a vision-based control law adapted to its dynamic model and without explicitly planning any trajectory. This framework is largely substantiated by experiments with a X4-flyer equipped with a fisheye camera.

This paper presents a high speed visual tracking method based on non simultaneous subimages acquisition. This method is formulated as a virtual visual servoing scheme. The sequential acquisition of regions of interest has a double benefit on visual servoing. The first one is that this acquisition method allows to increase the visual control sampling frequency by reducing the data amount to acquire and to transmit by the camera. The second one is that the associated image projection model depends on the observed object pose and velocity. Thanks to this property, a new vision-based control law can be defined. The particularity of this control law is that the control output consists of the kinematic and the dynamic twists. This allows to enhance the control performance in trajectory tracking applications. The experimental results in high speed visual tracking application show the effectiveness of this approach.

In this paper, we present a complete framework for autonomous vehicle navigation using a single camera and natural landmarks. When navigating in an unknown environment for the first time, usual behavior consists of memorizing some key views along the performed path to use these references as checkpoints for future navigation missions. The navigation framework for the wheeled vehicles presented in this paper is based on this assumption. During a human-guided learning step, the vehicle performs paths that are sampled and stored as a set of ordered key images, as acquired by an embedded camera. The visual paths are topologically organized, providing a visual memory of the environment. Given an image of the visual memory as a target, the vehicle navigation mission is defined as a concatenation of visual path subsets called visual routes. When autonomously running, the control guides the vehicle along the reference visual route without explicitly planning any trajectory. The control consists of a vision-based control law that is adapted to the nonholonomic constraint. Our navigation framework has been designed for a generic class of cameras (including conventional, catadioptric, and fisheye cameras). Experiments with an urban electric vehicle navigating in an outdoor environment have been carried out with a fisheye camera along a 750-m-long trajectory. Results validate our approach.

This paper deals with the visual control of the Gough-Stewart platform using a central catadioptric camera observing the platform's legs. This allows a large field of view to be obtained and avoids the occlusion problems observed when a classical perspective camera is used. An automatic and simple method to detect the projections of the leg in the image is also proposed. The control scheme presented here is shown to encompass the classical perspective camera case, as well as catadioptric ones. Finally, experimental results comparing two kinds of visual features (leg directions and leg edges) are described.

In this paper, we present a complete framework for autonomous indoor robot navigation. We show that autonomous navigation is possible in indoor situation using a single camera and natural landmarks. When navigating in an unknown environment for the first time, a natural behavior consists on memorizing some key views along the performed path, in order to use these references as checkpoints for a future navigation mission. The navigation framework for wheeled robots presented in this paper is based on this assumption. During a human-guided learning step, the robot performs paths which are sampled and stored as a set of ordered key images, acquired by an embedded camera. The set of these obtained visual paths is topologically organized and provides a visual memory of the environment. Given an image of one of the visual paths as a target, the robot navigation mission is defined as a concatenation of visual path subsets, called visual route. When running autonomously, the control guides the robot along the reference visual route without explicitly planning any trajectory. The control consists on a vision-based control law adapted to the nonholonomic constraint. The proposed framework has been designed for a generic class of cameras (including conventional, catadioptric and fish-eye cameras). Experiments with a AT3 Pioneer robot navigating in an indoor environment have been carried on with a fisheye camera. Results validate our approach.

An efficient method for global robot localization in a memory of omnidirectional images is presented. This method is valid for indoor and outdoor environments and not restricted to mobile robots. The proposed strategy is purely vision-based and uses as reference a set of prerecorded images (visual memory). The localization consists on finding in the visual memory the image which best fits the current image. We propose a hierarchical process combining global descriptors computed onto cubic interpolation of triangular mesh and patches correlation around Harris corners. To evaluate this method, three large images data sets have been used. Results of the proposed method are compared with those obtained from state-of-the-art techniques by means of 1) accuracy, 2) amount of memorized data required per image and 3) computational cost. The proposed method shows the best compromise in term of those criteria.

This paper presents a novel method for high speed pose and velocity computation from visual sensor. The main problem in high speed vision is the bottleneck phenomenon which limits the video rate transmission. The proposed approach circles the problem out by increasing the information density instead of the data rate transmission. This strategy is based on a rotary sequential acquisition of selected regions of interest (ROI) which provides space-time data. This acquisition mode induces an image projection deformation of dynamic objects. This paper shows how to use this artifact for the simultaneous measure of both pose and velocity, at the same frequency as the ROI's acquisition one.

In this paper a new decoupled imaged-based control scheme is proposed from projection onto a unit sphere. This control scheme is based on moment invariants to 3D rotational motion. This allows the control of translational motion independently of the rotational one. First, the analytical form of the interaction matrix related to the spherical moments is derived. It is based on the projection of a set of points onto a unit sphere. From the spherical moment, six features are presented to control the full 6 degrees of freedom. Finally, the results are validated through realistic simulation results.

This paper deals with the efficient second order minimization (ESM) and the image-based visual servoing schemes. In other word, it deals with the minimization based on the pseudo-inverses of the mean of the Jacobians or on the mean of Jacobian Pseudo-inverses. Chronologically, it has been noted by Tahri and Chaumette (2003) that the (ESM) improves generally the system behavior compared to the case where only the simple Jacobian Pseudo- inverses is used. Subsequently, a mathematical explanation has been given by Malis (2004). In this paper, the proofs given by Malis are considered to deal with their validity. It will be shown that there is a limitation to the the validity of this method and some precautions should be taken, for adequate application of it. In other words, we will show that the use of ESM does not necessary ensures a better system behavior, especially in the cases where large rotational motions are considered.

In this paper, we present a method to efficiently manage visual memory for autonomous vehicle navigation in large scale environments. It relies on two crucial issues for real-time navigation: an efficient organisation of the memory and small computational cost. A software platform (SoViN) dedicated to visual memory management and navigation strategies (including vision-based memory building, localization and navigation) has been developed to fulfill these requirements. We show that using this software architecture makes possible real-time navigation in large-scale outdoor situation using a single camera and natural landmarks.

Building an accurate three dimensional map is an important task for autonomous localisation and navigation. In a sequential approach to reconstruction from video streams, we show how adding prior knowledge about camera motion improves reconstruction accuracy, obtaining a more precise trajectory estimation and preventing failures over time. We add a smoothing penalty on camera trajectory and the smoothing parameter, usually fixed by trial and error, is automatically estimated using Cross-Validation. The method is substantiated by experimental results on synthetic and real data. They show that it improves accuracy and stability in the reconstruction process, preventing several failure cases.

In this paper, we consider the problem of controlling a 6 DOF holonomic robot and a nonholonomic mobile robot from the projection of 3-D straight lines in the image plane of central catadioptric systems. A generic central catadioptric interaction matrix for the projection of 3-D straight lines is derived using an unifying imaging model valid for an entire class of cameras. This result is exploited to design an image-based control law that allows us to control the 6 DOF of a robotic arm. Then, the projected lines are exploited to control a nonholonomic robot. We show that as when considering a robotic arm, the control objectives are mainly based on catadioptric image feature and that local asymptotic convergence is guaranteed. Simulation results and real experiments with a 6 DOF eye-to-hand system and a mobile robot illustrate the control strategy.

This paper presents a hybrid decoupled vision-based control scheme valid for the entire class of central catadioptric sensors (including conventional perspective cameras). First, we consider the structure from motion problem using imaged 3D points. Geometrical relationships are exploited to enable a partial Euclidean reconstruction by decoupling the interaction between translation and rotation components of a homography matrix. The information extracted from the homography are then used to design a control law which allow us to fully decouple rotational motions from translational motions. Real time experimental results using an eye-to-hand robotic system with a paracatadioptric camera are presented and confirm the validity of our approach.

Omnidirectional cameras have a wide field of view and are thus used in many robotic vision tasks. An omnidirectional view may be acquired by a fisheye camera which provides a full image compared to catadioptric visual sensors and do not increase the size and the weakness of the imaging system with respect to perspective cameras. We prove that the unified model for catadioptric systems can model fisheye cameras with distortions directly included in its parameters. This unified projection model consists on a projection onto a virtual unitary sphere, followed by a perspective projection onto an image plane. The validity of this assumption is discussed and compared with other existing models. Calibration and partial Euclidean reconstruction results help to confirm the validity of our approach. Finally, an application to the visual servoing of a mobile robot is presented and experimented.

This work deals with the control by vision of the Gough-Stewart platform. For that, a central catadioptric camera is used to observe the platform legs. This allows to obtain a large field of view, and then avoids the occlusion problems observed when a classical perspective camera is used. The leg projections onto the catadioptric plane are used to determine their orientation in the camera frame. Finally, the computed orientations will be used in a visual servoing scheme of the platform effector.

Motivated by the growing interest for omnidirectional sensors on robotic applications and particularly on vision-based control, we present a new framework to handle in a visual servoing scheme the projection of line features into the image plane of a central catadioptric camera. As it is well known, the projection of a 3D line in the image plane of a central catadioptric camera is a conic curve. We propose to use the polar line of the image center with respect to this conic curve to define the input of the vision-based control scheme. The visual observations obtained from the polar lines lead to a minimal representation of projected lines. An efficient control scheme based only on two image features can then be designed. Simulation and experimental results confirm the validity of our approach

In this paper, we show that visual feedback reduces the complexity of parallel robot Cartesian control. Namely, 3D pose visual servoing, where the end-effector pose is indirectly measured and used for regulation, is shown to be well suited to this task since it relieves the control from the difficult forward kinematic problem. Moreover, this complexity reduction is not coming with an increase of the implementation complexity since off-the-shelf hardware and software are now available for visual servoing. It is also shown that such a control gets rid of joint sensors. All this makes 3D pose visual servoing the most straightforward Cartesian control for parallel robots. Experimental results are provided using an open source visual servoing C++ library

This paper presents a new hybrid decoupled vision-based control scheme valid for the entire class of central catadioptric sensors (including conventional perspective cameras). First, we consider the structure from motion problem using imaged 3D lines (conics). Polar lines of the principal point with respect to the conic curves are exploited to estimate a generic homography matrix from which a partial Euclidean reconstruction is obtained. The polar lines and the information extracted from the homography are then used to design a control law which allow us to fully decouple rotational motions from translational motions

When navigating in an unknown environment for the first time, a natural behavior consists in memorizing some key views along the performed path, in order to use these references as checkpoints for a future navigation mission taking a similar path. This assumption is used in this paper as the basis of a navigation framework for wheeled mobile robots in indoor environments. During a human-guided teleoperated learning step, the robot performs paths which are sampled and stored as a set of ordered key images, acquired by a standard embedded camera. The set of these obtained visual paths is topologically organized and provides a visual memory of the environment. Given an image of one of the visual paths as a target, the robot navigation mission is defined as a concatenation of visual path subsets, called visual route. When running autonomously, the robot is controlled by a visual servoing law adapted to its nonholonomic constraint. Based on the regulation of successive homographies, this control guides the robot along the reference visual route without explicitly planning any trajectory. Real experiment results illustrate the validity of the presented framework.

To close the loop between motion and vision, tracked visual features must remain in the camera field of view (visibility constraint). To overcome the visibility constraint, visual servoing methods can benefit from panoramic sensors such as catadioptric cameras (combining both mirrors and lenses). In this paper, we present a vision-based framework to control a nonholonomic mobile robot using a catadioptric imaging system. We particularly focus on a suitable catadioptric image-based control strategy of a nonholonomic robot in order to follow a 3D straight line. Such strategy can be applied to navigate in indoor or urban environment since the extraction and the tracking of straight lines are natural. First the control objectives are formulated in the catadioptric image space. The control law is then designed according to a well suited chained system for a mobile robot state vector directly expressed in the image space using a generic camera model. Simulation results illustrate the control strategy in the case of hypercatadioptric and paracatadioptric cameras.

In this paper, we present how the 2 1/2 D visual servoing scheme can be used with omnidirectional cameras. Motivated by the growing interest for omnidirectional sensors on robotic applications and particularly on vision-based control, we extend this framework to the entire class of central catadioptric systems. Indeed, conventional cameras suffer from restricted field of view. Central catadioptric systems have larger fields of view thus overcoming the visibility problem encountered when using conventional cameras. The 2 1/2 D visual servoing is based on the estimation of the partial camera displacement between two views, given by the current and desired images. Geometrical relationships are exploited to enable a partial Euclidean reconstruction by decoupling the interaction between translation and rotation components of a homography matrix. First we describe how to obtain a generic homography matrix for central catadioptric cameras from the projection model of an entire class of camera. Then the information obtained from the homography is used to develop a 2 1/2 D visual servoing scheme.

The paper presents a method for estimating the control matrix in visual servoing using approximation up to the second order of the projection function. The classical approach simply uses the first order terms (inverse of the interaction matrix). The resulting control matrix is shown to perform much better than the classical one. Peculiarly, translation are made almost completely independent of z rotations, allowing better spatial trajectories. Theoretical insight as well as comparisons in the domain of visual servoing are provided to demonstrate this assertion.

In this paper we consider the problem of controlling a robotic system using the projection of 3D lines in the image plane of central catadioptric systems. Most of the efforts in visual servoing are devoted to points, only few works have investigated the use of lines in visual servoing with traditional cameras and none has explored the case of omnidirectional cameras. First a generic central catadioptric interaction matrix for the projection of 3D straight lines is derived from the projection model of an entire class of camera. Then an image-based control law is designed and validated through simulation results.

This paper concerns the stability analysis of image-based visual servoing methods with respect to uncertainties on the 3D parameters introduced in the central catadioptric interaction matrix. Motivated by the growing interest for omnidirectional sensors on robotic applications and particularly on vision-based control, we extend recent results obtained for conventional cameras to the entire class of central catadioptric systems. In this paper, it is shown that with such sensors extreme care must be taken when approximating 3D parameters to ensure stability of the image based control law.

The principal deficiency of image-based visual servoing is that the induced (3D) trajectories are not optimal and sometimes, especially when the displacement to realize is large, these trajectories are not physically valid leading to the failure of the servoing process. Furthermore, visual control needs a matching step between the features extracted from the initial image and the desired one. This step can be problematic (or impossible) when the camera displacement between the acquisitions of the initial and desired images is large and/or for complex scenes. To resolve these deficiencies, we couple an image interpolation process between N relay images extracted from a database and an image-based trajectory tracking. The camera calibration and the model of the observed scene are not assumed to be known. The relay images are interpolated in such a way that the corresponding camera trajectory is minimal. First a closed form collineation path is obtained and then the analytical form of image features trajectories are derived and efficiently tracked using a purely image-based control. Experimental results obtained on a six DOF eye-in-hand robotic system are presented and confirm the validity of the proposed approach.

In this paper, we present a framework for cell manipulation tasks with visual servoing micromanipulation strategies. A vision based micropositioner is designed in order to address the requirement of high precision needed to perform manipulation of objects under 100 /spl mu/m in size. The system calibration (microscope-camera-micropositioner) and the model of the observed scene are not known. Experimental results for micropositioning tasks with respect to protein cells are presented and demonstrate the validity of the proposed approach.

Vision feedback control loop techniques are efficient for a large class of applications, but they come up against difficulties when the initial and desired robot positions are distant. Classical approaches are based on the regulation to zero of an error function computed from the current measurement and a constant desired one. By using such an approach, it is not obvious how to introduce any constraint in the realized trajectories or to ensure the convergence for all the initial configurations. In this paper, we propose a new approach to resolve these difficulties by coupling path planning in image space and image-based control. Constraints such that the object remains in the camera field of view or the robot avoids its joint limits can be taken into account at the task planning level. Furthermore, by using this approach, current measurements always remain close to their desired value, and a control by image-based servoing ensures robustness with respect to modeling errors. The proposed method is based on the potential field approach and is applied whether the object shape and dimensions are known or not, and when the calibration parameters of the camera are well or badly estimated. Finally, real-time experimental results using an eye-in-hand robotic system are presented and confirm the validity of our approach.

Since image-based servoing is a local control solution, it requires the definition of intermediate subgoals in the sensor space when the initial robot position is far from the desired one. This paper addresses the problem of generating and tracking realistic and optimal smooth trajectories of complex features in the image space. The model of the observed target and the internal camera parameters are assumed to be unknown. First a closed-form smooth collineation path (related to a reference plane) between given starts and end-points is obtained. This path is generated in order to correspond to an optimal camera path. The trajectories of the image features (corresponding to points belonging to or not belonging to the reference plane) are then derived and tracked using image based control.

Since image-based visual servoing is a local feedback control solution, it requires the definition of intermediate subgoals in the sensor space at the task planning level. We describe a general technique for specifying and tracking trajectories of an unknown object in the camera image space. First, physically valid C/sup 2/ image trajectories which correspond to quasi-optimal 3D camera trajectory (approaching as much as possible a straight line) are performed. Both mechanical (joint limits) and visibility constraints are taken into account at the task planning level. The good behavior of image-based control when desired and current camera positions are closed is then exploited to design an efficient control scheme. Real time experimental results using a camera mounted on the end effector of a 6 DOF robot confirm the validity of our approach.

Since image-based servoing is a local control solution, it requires the definition of intermediate subgoals in the sensor space. This paper addresses the problem of generating realistic and optimal smooth trajectories of complex features in the image space. The model of the observed target and the internal camera parameters are assumed to be unknown. First, a closed-form smooth collineation path between given starts and endpoints is obtained. This path is generated in order to correspond to an optimal camera path. The trajectories of the image features are then derived.

Vision feedback control loop techniques are efficient for a number of applications but they come up against difficulties when the initial and desired positions of the camera are distant. We propose a new approach to resolve these difficulties by planning trajectories in the image. Constraints such that the object remains in the camera field of view can be taken into account. Furthermore, using this process, current measurement always remain close to their desired value and a control by image based servoing ensures the robustness with respect to modeling errors. We apply our method when object dimension are known or not and/or when the calibration parameters of the camera are well or badly estimated. Finally, real time experimental results using a camera mounted on the end effector of a 6-DOF robot are presented.