This paper presents a visual servoing strategy for a clevis and tenon joint assembly as typically performed in aircraft manufacturing. The desired velocity in the image acquired by the cameras observing the parts is extracted from a vector field that guides the motion of the moving part so that it follows an accurate 3D straight line trajectory. The vector field is designed so that it is continuous and globally exponentially stable whatever the initial configuration. The strategy has been successfully implemented and validated on a full-scale demonstrator, with coarse camera localization, while ensuring an assembly without any collision thanks to a maximal error less than 1 mm along a 10 cm trajectory.

In this paper, we present how a proximity array attached as an end-effector can perform a positioning task with respect to a cylinder. We develop the complete modeling of the system from which a classical control law is designed. Positioning with respect to the outside of a cylinder can be used for non-contact inspection of the exterior of pipes in chemical plants or in oil/gas industries, while positioning with respect to the inside of a hollow cylinder can be used for guidance inside a pipe for non-contact inspection. We provide simulation and experimental results to validate the presented theory.

Local minima are a well-known drawback of image-based visual servoing systems. Up to now, there were no formal guarantees on their number, or even their existence, according to the considered configuration. In this work, a formal approach is presented for the exhaustive computation of all minima and unstable equilibria for a class of six well-known image-based visual servoing controllers. This approach relies on a new polynomial formulation of the equilibrium condition that avoids using the camera pose. By using modern computational algebraic geometry methods and an ad hoc symmetry breaking strategy, the formal resolution of this new equilibrium condition is rendered computationally feasible. The proposed methodology is applied to compute the equilibria of several classical visual servoing tasks, with planar and nonplanar configurations of four and five points. The effects of local minima and saddle points on the dynamics of the system are finally illustrated through intensive simulation results, as well as the effects of image noise and uncertainties on depths.

Cable-Driven Parallel Robots (CDPRs) are well-adapted to large workspaces since they replace rigid links by cables. However, they lack in positioning accuracy and new control methods are necessary to achieve profile-following tasks. This paper presents a control scheme designed for these tasks, relying on a combination of accurate boarded distance sensors and of a less accurate remote camera. The profile-following task is divided into two subtasks that are partially conflicting: maintaining a parallel orientation and a constant distance with the surface to follow, and following a trajectory between two points on the surface. The data fusion to solve the redundancy is based on the Gradient Projection Method. This control scheme is validated experimentally on a CDPR prototype and shown to provide the expected behaviour.

In this paper we propose a visual servoing approach that controls the deformation of a suspended tether cable subject to gravity from visual data provided by a RGB-D camera. The cable shape is modelled with a parabolic curve together with the orientation of the plane containing the tether. The visual features considered are the parabolic coefficients and the yaw angle of that plane. We derive the analytical expression of the interaction matrix that relates the variation of the visual features to the velocities of the cable extremities. Singularities are demonstrated to occur if and only if the cable is taut horizontally or vertically. An image processing algorithm is also developed to extract in real-time the current features fitting the parabola to the cable from the observed point cloud. Simulations and experimental results demonstrate the efficiency of our visual servoing approach to deform the tether cable toward a desired shape configuration.

In this paper, we propose a physics-based robot controller to deform a soft object toward a desired 3D shape using a limited number of handling points. For this purpose, the shape of the deformable object is represented using Fourier descriptors. We derive the analytical relation that provides the variation of the Fourier coefficients as a function of the movements of the handling points by considering a mass-spring model (MSM). A control law is then designed from this relation. Since the MSM provides an approximation of the object behavior, which in practice can lead to a drift between the object and its model, an online realignment of the model with the real object is performed by tracking its surface from data provided by a remote RGB-D camera. Simulation results validate the approach for the case where many points interact on a 2D soft object while experimental results obtained with two robotic arms demonstrate the autonomous shaping of a 3D soft object.

In this paper, we present a complete pipeline for positioning a feature point of a soft object to a desired 3D position, by acting on a different manipulation point using a robotic manipulator. For that purpose, the analytic relation between the feature point displacement and the robot motion is derived using a coarse mass-spring model (MSM), while taking into consideration the propagation delay introduced by a MSM. From this modeling step, a novel closed-loop controller is designed for performing the positioning task. To get rid of the model approximations, the object is tracked in real-time using a RGB-D sensor, thus allowing to correct on-line any drift between the object and its model. Our model-based and vision-based controller is validated in real experiments for two different soft objects and the results show promising performance in terms of accuracy, efficiency and robustness.

We present in this paper a new open-source simulator based on CoppeliaSim and ROS for the popular Franka Emika Robot (FER) fully integrated in the ViSP ecosystem, a powerful library for Visual-Servoing. The simulator features a dynamic model that has been accurately identified from a real robot, leading to more realistic simulations. The C++ Api closely follows the ViSP class of the real robot allowing to narrow the gap between simulation code and real control software deployment. Conceived as a multipurpose research simulation platform, it is well suited for visual servoing applications as well as, in general, for any pedagogical purpose in robotics. All the software, models and CoppeliaSim scenes presented in this work are publicly available under free GPL-2.0 license.

In this work we present our results on dynamic visual servoing for the case of moving targets while also exploring the possibility of using such a controller for interaction with the environment. We illustrate the derivation of a feature space impedance controller for tracking a moving object as well as an Extended Kalman Filter based on the visual servoing kinematics for increasing the rate of the visual information and estimating the target velocity for both the cases of PBVS and IBVS with image point features. Simulations are carried out to validate the estimator performance during a Peg-in-Hole insertion task with a moving part. Experiments are also conducted on a real redundant manipulator with a low-cost wrist-mounted camera. Details on several implementation issues encountered during implementation are also discussed.

In this paper, we propose a new sampling-based path planning approach, focusing on the challenges linked to autonomous exploration. Our method relies on the definition of a disk graph of free-space bubbles, from which we derive a biased sampling function that expands the graph towards known free space for maximal navigability and frontiers discovery. The proposed method demonstrates an exploratory behavior similar to Rapidly-exploring Random Trees, while retaining the connectivity and flexibility of a graph-based planner. We demonstrate the interest of our method by first comparing its path planning capabilities against state-of-the-art approaches, before discussing exploration-specific aspects, namely replanning capabilities and incremental construction of the graph. A simple frontiers-driven exploration controller derived from our planning method is also demonstrated using the Pioneer platform.

Recent Earth observation satellites are now equipped with new instrument that allows image feedback in real-time. Problematic such as ground target tracking, moving or not, can now be addressed by precisely controlling the satellite attitude. In this paper, we propose to consider this problem using a visual servoing approach. While focusing on the target, the control scheme has also to take into account the satellite motion induced by its orbit, Earth rotational velocities, potential target own motion, but also rotational velocities and accelerations constraints of the system. We show the efficiency of our system using both simulation (considering real Earth image) and experiments on a robot that replicates actual high resolution satellite constraints.

In this paper, we consider a multi-sensor arrangement of proximity sensors that forms a proximity array. A general modeling methodology is considered within the framework of Sensor-based Control. It incorporates multiple sensor signals from the proximity array by giving primary emphasis on the interaction screw. To prove its effectiveness, modeling approach is applied to the task of plane-to-plane positioning. We discuss the development of two sensor-based task functions for the specific task considered. The validity of the methodology is provided using relevant experimental results.

Cable-driven parallel robots (CDPRs) are a type of parallel robots, where cables are used instead of rigid links. This leads to many advantages, such as large workspace, low mass in motion and simple reconfiguration. The drawbacks are accuracy issues and complex cable management. Indeed, it is usual that cables become slack. That can be caused by, for example, cable mass, uncertainties in the system, and a higher number of cables than the number of degrees of freedom of the moving-platform. This reduces CDPR stiffness and degree of actuation. While visual servoing provides good accuracy and is robust to different perturbations in the system and to modeling errors, it does not deal with cable slackness. Thus, a CDPR with visual servoing can become underactuated due to cable slack. We propose in this paper to enrich visual servoing with a tension correction algorithm. Experimental results show reduction of slackness and thus avoiding slackness-related trajectory perturbations and loss of stability.

Cable-Driven Parallel Robots (CDPRs) are parallel robots with rigid links replaced by cables. As for most parallel robots the determination of the analytical solutions to the direct geometrico-static model (DGSM) is a difficult task that is often not feasible online. However, the knowledge of the moving-platform (MP) pose is necessary in order to control the CDPR, e.g. with visual servoing. When the MP pose measurement is not available, an estimation can be sufficient. This paper compares three estimation methods: (a) control-based; (b) image-based; and (c) model-based. The three methods are implemented experimentally with an open-loop velocity controller and a closed-loop visual servoing controller. Overall, very good results are shown with model-based and control-based methods for both controllers. Finally, it is shown that the visual servoing controller leads to a better accuracy of the robot than the velocity controller.

In this paper, we propose a novel approach for transferring a deep reinforcement learning (DRL) grasping agent from simulation to a real robot, without fine tuning in the real world. The approach utilises a CycleGAN to close the reality gap between the simulated and real environments, in a reverse real-to-sim manner, effectively "tricking" the agent into believing it is still in the simulator. Furthermore, a visual servoing (VS) grasping task is added to correct for inaccurate agent gripper pose estimations derived from deep learning. The proposed approach is evaluated by means of real grasping experiments, achieving a success rate of 83 % on previously seen objects, and the same success rate for previously unseen, semi-compliant objects. The robustness of the approach is demonstrated by comparing it with two baselines, DRL plus CycleGAN, and VS only. The results clearly show that our approach outperforms both baselines.

This paper introduces the use of numerical optical flow (OF) in vision-based steering techniques - that control characters locomotion trajectories by using a simulation of their visual perception. In contrast with synthetic OF that was previously used, numerical OF is sensitive to the contrast of objects, and provides, for example, uncertain results in dark areas. Thus, we here propose a locomotion control technique which is robust to such uncertainty: dark areas in the scene are processed as obstacles, that however may be traversed in case of necessity. As demonstrated in various scenarios, this tends to make character avoiding darkest areas, or traversing them more carefully, as it can be observed for real humans.

Abstract Steering and navigation are important components of character animation systems to enable them to autonomously move in their environment. In this work, we propose a synthetic vision model that uses visual features to steer agents through dynamic environments. Our agents perceive optical flow resulting from their relative motion with the objects of the environment. The optical flow is then segmented and processed to extract visual features such as the focus of expansion and time‐to‐collision. Then, we establish the relations between these visual features and the agent motion, and use them to design a set of control functions which allow characters to perform object‐dependent tasks, such as following, avoiding and reaching. Control functions are then combined to let characters perform more complex navigation tasks in dynamic environments, such as reaching a goal while avoiding multiple obstacles. Agent's motion is achieved by local minimization of these functions. We demonstrate the efficiency of our approach through a number of scenarios. Our work sets the basis for building a character animation system which imitates human sensorimotor actions. It opens new perspectives to achieve realistic simulation of human characters taking into account perceptual factors, such as the lighting conditions of the environment.

The direct use of the entire photometric image information as dense features for visual servoing brings several advantages. First, it does not require any feature detection, matching, or tracking process. Thanks to the redundancy of visual information, the precision at convergence is highly accurate. However, the corresponding highly nonlinear cost function reduces the convergence domain. In this paper, we propose visual servoing based on the analytical formulation of Gaussian mixtures to enlarge the convergence domain. Pixels are represented by two-dimensional Gaussian functions that denote a “power of attraction.” In addition to the control of the camera velocities during the servoing, we also optimize the Gaussian spreads allowing the camera to precisely converge to a desired pose even from a far initial one. Simulations show that our approach outperforms the state of the art and real experiments show the effectiveness, robustness, and accuracy of our approach.

We present a deep neural network-based method to perform high-precision, robust and real-time 6 DOF positioning tasks by visual servoing. A convolutional neural network is fine-tuned to estimate the relative pose between the current and desired images and a pose-based visual servoing control law is considered to reach the desired pose. The paper describes how to efficiently and automatically create a dataset used to train the network. We show that this enables the robust handling of various perturbations (occlusions and lighting variations). We then propose the training of a scene-agnostic network by feeding in both the desired and current images into a deep network. The method is validated on a 6 DOF robot.

We present in this paper a modular real-time model-based visual tracker. It is able to fuse different types of measurement, that is, edge points, textured points, and depth map, provided by one or multiple vision sensors. A confidence index is also proposed for determining if the outputs of the tracker are reliable or not. As expected, experimental results show that the more various measurements are combined, the more accurate and robust is the tracker. The corresponding C++ source code is available for the community in the ViSP library.

In this paper, visual servoing based on photometric moments is advocated. A direct approach is chosen by which the extraction of geometric primitives, visual tracking and image matching steps of a conventional visual servoing pipeline can be bypassed. A vital challenge in photometric methods is the change in the image resulting from the appearance and disappearance of portions of the scene from the camera field of view during the servo. To tackle this issue, a general model for the photometric moments enhanced with spatial weighting is proposed. The interaction matrix for these spatially weighted photometric moments is derived in an analytical form. The correctness of the modeling, effectiveness of the proposed strategy in handling the exogenous regions, and improved convergence domain are demonstrated with a combination of simulation and the experimental results.

This paper proposes a control framework based on auditory perception. Generally, in robot audition, the motion control of a robot from the sense of hearing relies on sound source localization. We propose in this paper an alternative approach, aural servo, which is derived from the sensor-based control framework. In this approach, robot motions are directly connected to the aural perception: The variation of low-level auditory features dictates the motions applied to the robot through a feedback loop. It has the advantage of being robust to spurious measurements and modeling approximations for a low computational cost. This paper presents the theoretical concept of the aural servo framework. Besides a theoretical analysis, the aural servo framework is validated through several experiments on different robotic platforms and under real-world conditions.

We propose in this paper a new active perception scheme based on Model Predictive Control under constraints for generating a sequence of visual servoing tasks. The proposed control scheme is used to compute the motion of a camera whose task is to successively observe a set of robots for measuring their position and improving the accuracy of their localization. This method is based on the prediction of an uncertainty model (due to actuation and measurement noise) for determining which robot has to be observed by the camera. Simulation results are presented for validating the approach.

Apart the use of catadioptric cameras, only few visual servoing works exploit the use of mirror. Such a configuration is however interesting since it allows to overpass the limited camera field of view. Based on the known projection equations involved in such a system, this paper introduces the theoretical background that allows the use of planar mirror for visual servoing in different configurations. Limitations intrinsic to such systems, such as the number of d.o.f actually controllable, is then discussed. Experiments using a mirror mounted on the end-effector of a 6 d.o.f robot validate the proposed approach.

In active vision, the camera motion is controlled in order to improve a certain visual sensing strategy. In this paper, we formulate an active vision task function to improve pose estimation. This is done by defining an optimality metric on the Fisher Information Matrix. This task is then incorporated into a weighted multi-objective optimization framework. To test this approach, we apply it on the three image point visual servoing problem which has a degenerate configuration - a singularity cylinder. The simulation results show that the singular configurations of pose estimation are avoided during visual servoing. We then discuss the potential of active vision to be integrated into more complex multi-task frameworks.

This paper presents image-based navigation from an image memory using a combination of line segments and feature points. The environment is represented by a set of key images, which are acquired during a prior mapping phase that defines the path to be followed during the navigation. The switching of key images is done exploiting the common line segments and feature points between the current acquired image and the nearby key images. Based on the key images and the current image, a control law is derived for computing the rotational velocity of a mobile robot during its visual navigation. Using our approach, real-time navigation has been performed in real indoor environment with a Pioneer 3-DX equipped with an on-board perspective camera and the humanoid robot Pepper without the need of accurate mapping and localization nor of 3D reconstruction. We also show that the combination of points and lines increases the number of features that helps in robust and successful navigation especially in those regions where few points or lines can be detected and tracked/matched.

In this paper, we address the problem of using a camera with limited field of view for controlling the motion of a quadrotor in aggressive flight regimes. We present a minimum time trajectory planning method that guarantees visibility of the image features while allowing the robot to undertake aggressive motions for which the usual near-hovering assumption is violated. We exploit differential flatness and B-Splines to parametrize the system trajectories in terms of a finite number of control points, which can then be optimized by Sequential Quadratic Programming (SQP). The control strategy is similar to a Receding Horizon Control (RHC) approach for modifying online the reference trajectory in order to account for noise, disturbances and any non-modeled effect. The algorithm is validated in a physically realistic simulation environment.

The determination of the singularity cases in visual servoing is a tricky problem, which is unsolved for most of the image-based approaches. In order to avoid singularities, redundant measurements may be used. However, they lead to the presence of local minima. Moreover, they do not always ensure that singularities can be avoided. Here, we show that a concept named the “hidden robot,” which was formerly used for understanding the singularities of a vision-based controller dedicated to parallel robots, can be used for interpreting the singularities in the visual servoing of image points. These singularity cases were already found in the case in which three points are observed, but we show that the hidden robot concept considerably simplifies the analysis by using geometric interpretations of the mapping degeneracy and tools provided by the mechanical engineering community. Moreover, to the best of our knowledge, for the first time, we provide the singularity conditions when more than three points are observed. We also discuss how these tools could be extended in order to find the singularity cases of other visual servoing techniques (e.g., when lines are observed).

This paper presents a complete framework for image-based navigation from an image memory that exploits mutual information and does not need any feature extraction, matching or any 3D information. The navigation path is represented by a set of automatically selected key images obtained during a prior learning phase. The shared information (entropy) between the current acquired image and nearby key images is exploited to switch key images during navigation. Based on the key images and the current image, the control law proposed by [1] is used to compute the rotational velocity of a mobile robot during its qualitative visual navigation. Using our approach, real-time navigation has been performed inside a corridor and inside a room with a Pioneer 3-DX equipped with an on-board perspective camera without the need of accurate mapping and localization.

This paper validates experimentally a novel approach to robot audition, sound-based control, which consists in introducing auditory features directly as inputs of a closed-loop control scheme, that is, without any explicit localization process. The applications we present rely on the implicit bearings of the sound sources computed from the time difference of arrival (TDOA) between two microphones. By linking the motion of the robot to the aural perception of the environment, this approach has the benefit of being more robust to reverberation and noise. Therefore neither complex tracking method such as Kalman filtering nor TDOA enhancement with denoising or dereverberation methods are needed to track the correct TDOA measurements. The experiments conducted on a mobile robot instrumented with a pair of microphones show the validity of our approach. In a reverberating and noisy room, this approach is able to orient the robot to a mobile sound source in real time. A positioning task with respect to two sound sources is also performed while the robot perception is disturbed by altered and spurious TDOA measurements.

This paper is a follow-up way to our previous works regarding audio-based control, that is an alternative method for auditory-based robot tasks. Conversely to classic methods oriented towards sound source localization, audio-based control is a sensor-based framework that does not localize the sound source. Instead, auditory features are used as inputs of a closed-loop control scheme. The audio-based control method presented in this paper relies on the sound signal energy measured by two microphones. By combining the interchannel level difference to the acoustic absolute energy level, the control scheme allows positioning the robot with respect to the sound source at a given distance and orientation. Moreover this method has the benefit of a low computation cost, since it only relies on the signal energy measurement. Experimental results conducted on a mobile robot validate the relevance and the robustness of this approach in dynamic and real world conditions.

Visual servoing based on photometric data is of great interest since it does not necessitate any image processing or visual tracking steps. A vital issue in such methods is the change in the image resulting from the appearance and disappearance of portions of the scene from the camera field-of-view during the visual servo. In this paper, we propose a spatial weighting scheme to counter this problem. A general model for photometric moments that allows for inclusion of spatial weights is presented. A custom weighting function is then chosen and specific properties of its derivative are exploited in order to develop the interaction matrix in analytical form. The resultant effects on the invariance properties are discussed. Experiments have been performed to validate the visual servoing with the proposed weighting scheme on a 4 dof robot arm.

One way to deal with occlusions or loss of tracking of the visual features used for visual servoing tasks is to predict the feature behavior in the image plane when the measurements are missing. Different prediction and correction methods have already been proposed in the literature. The purpose of this paper is to compare and experimentally validate some of these methods for eye-in-hand and eye-to-hand configurations. In particular, we show that a correction based both on the image and the camera/target pose provides the best results.

The selection of a suitable set of visual features for an optimal performance of closed-loop visual control or Structure from Motion (SfM) schemes is still an open problem in the visual servoing community. For instance, when considering integral region-based features such as image moments, only heuristic, partial, or local results are currently available for guiding the selection of an appropriate moment set. The goal of this paper is to propose a novel learning strategy able to automatically optimize online the shape of a given class of image moments as a function of the observed scene for improving the SfM performance in estimating the scene structure. As case study, the problem of recovering the (unknown) 3D parameters of a planar scene from measured moments and known camera motion is considered. The reported simulation results fully confirm the soundness of the approach and its superior performance over more consolidated solutions in increasing the information gain during the estimation task.

In this paper we experimentally validate and compare three different methods for estimating the 3D parameters of a planar scene from a (possibly time-varying) set of feature points acquired by a moving monocular camera. The first method, based on the classical decomposition of the homography matrix, is meant to serve as a baseline condition classically used in many previous works. The other two methods exploit an active Structure from Motion (SfM) scheme for either extracting the plane from the reconstructed 3D position of all the tracked points, or for directly estimating the plane parameters by considering a set of discrete image moments as visual input. The possible loss/gain of point features during the camera motion is considered in all three methods by, in particular, introducing a suitable weighting strategy for the image moment case. Finally, the results of an experimental validation are presented with a comparative discussion of the pros/cons of the three methods.

This paper presents a novel approach to robot audition by performing robotic tasks with auditory cues. Unlike many previous works, we propose a control scheme, that does not require any explicit sound source localization. This approach is capable of controlling all the three degrees of freedom in a plane from two microphones. Built upon the sensor-based control framework, this approach relies on implicit sound source direction obtained from the time difference of arrival (TDOA). We introduce an analytical modelling of auditory cues considering a robot equipped with a pair of microphones and multiple sound sources, from which a control scheme is designed. A stability analysis is provided as well. The results obtained in simulation show the feasibility and the suitability of this method even in reverberant area.

In this paper, we introduce the concept of tunable visual features for moments based visual servoing schemes. The main contribution of this work is the introduction of tunable shift points along with some effective methods to tune them. We propose two different metrics: the first metric ensures optimal response of the control to errors in the image space and the second metric ensures orthogonality between the interaction matrix components (vectors) related to the control of x and y rotational motions. With the proposed method, it is possible to design moment invariants-based visual features whose interaction matrix is always non-singular for any desired pose (parallel or non-parallel). Thus, this work makes a significant contribution to the difficult problem of controlling the rotational motions around the x and y axes, when all the 6dof are involved. Two case studies are presented to demonstrate the validity of the proposed ideas. Results from each case are then used to design a moment invariants-based visual feature. This visual feature is used for visual servoing with a symmetrical object using binary moments and a free-form planar target using photometric moments.

Plane detection and estimation from visual data is a classical problem in robotic vision. In this work we propose a novel active strategy in which a monocular camera tries to determine whether a set of observed point features belongs to a common plane, and, if so, what are the associated plane parameters. The active component of the strategy imposes an optimized camera motion (as a function of the observed scene) able to maximize the convergence in estimating the scene structure. Based on this strategy, two methods are then proposed to solve the plane estimation task: a classical solution exploiting the homography constraint (and, thus, almost completely based on image correspondances across distant frames), and an alternative method fully taking advantage of the scene structure estimated incrementally during the camera motion. The two methods are extensively compared in several case studies by discussing the various pros/cons.

Equipping mobile robots with an omnidirectional camera is very advantageous in numerous applications as all information about the surrounding scene is stored in a single image frame. In the given context, the present paper is concerned with detection, tracking and following of a moving object with an omnidirectional camera. The camera calibration and image formation is based on the spherical unified projection model thus yielding a representation of the omnidirectional image on the unit sphere. Detection of moving objects is performed by calculating a sparse optical flow in the image and then lifting the flow vectors on the unit sphere where they are discriminated as dynamic or static by analytically calculating the distance of the terminal vector point to a great circle arc. The flow vectors are then clustered and the center of gravity is calculated to form the sensor measurement. Furthermore, the tracking is posed as a Bayesian estimation problem on the unit sphere and the solution based on the von Mises-Fisher distribution is utilized. Visual servoing is performed for the object following task where the control law calculation is based on the projection of a point on the unit sphere. Experimental results obtained by a camera with a fish-eye lens mounted on a differential drive mobile robot are presented and discussed.

Our objective is to extend the assistance functionalities of the cable-driven parallel robot Marionet-Assist, designed principally for transfer operation, by allowing it to grasp usual objects (knives, box of medicines, phone, ···) by using visual servoing. Our crane robot has a configuration that provides three translational d.o.f., and a camera was added to its end-effector. In order to compute the translational velocity sent to the robot controller, the area and center of gravity in the image of the object to be grasped are used. Experimental results are presented. They show the robustness of our scheme with respect to modeling errors and an excellent positioning accuracy allowing grasping.

In this paper we propose a solution for coupling the execution of a visual servoing task with a recently developed active Structure from Motion strategy able to optimize online the convergence rate in estimating the (unknown) 3D structure of the scene. This is achieved by suitably modifying the robot trajectory in the null-space of the servoing task so as to render the camera motion `more informative' w.r.t. the states to be estimated. As a byproduct, the better 3D structure estimation also improves the evaluation of the servoing interaction matrix which, in turn, results in a better closed-loop convergence of the task itself. The reported experimental results support the theoretical analysis and show the benefits of the method.

Structure estimation from motion (SfM) is a classical and well-studied problem in computer and robot vision, and many solutions have been proposed to treat it as a recursive filtering/estimation task. However, the issue of actively optimizing the transient response of the SfM estimation error has not received a comparable attention. In this paper, we provide an experimental validation of a recently proposed nonlinear active SfM strategy via two concrete applications: 3D structure estimation for a spherical and a cylindrical target. The experimental results fully support the theoretical analysis and clearly show the benefits of the proposed active strategy. Indeed, by suitably acting on the camera motion and estimation gains, it is possible to assign the error transient response and make it equivalent to that of a reference linear second-order system with desired poles.

Predicting the behavior of visual features on the image plane over a future time horizon is an important possibility in many different control problems. For example when dealing with occlusions (or other constraints such as joint limits) in a classical visual servoing loop, or also in the more advanced model predictive control schemes recently proposed in the literature. Several possibilities have been proposed to perform the initial correction step for then propagating the visual features by exploiting the measurements currently available by the camera. But the predictions proposed so far are inaccurate in situations where the depths of the tracked points are not correctly estimated. We then propose in this paper a new correction strategy which tries to directly correct the relative pose between the camera and the target instead of only adjusting the error on the image plane. This correction is then analysed and compared by evaluating the corresponding improvements in the feature prediction phase.

This paper deals with pose estimation using an iterative scheme. We show that using adequate visual information, pose estimation can be performed iteratively with only three independent unknowns, which are the translation parameters. Specifically, an invariant to rotational motion is used to estimate the camera position. In addition, an adequate transformation is applied to the proposed invariant to decrease the nonlinearities between the variations in image space and 3-D space. Once the camera position is estimated, we show that the rotation can be estimated efficiently using two different direct methods. The proposed approach is compared against two other methods from the literature. The results show that using our method, pose tracking in image sequences and the convergence rate for randomly generated poses are improved.

Moving obstacle avoidance is a fundamental requirement for any robot operating in real environments, where pedestrians, bicycles, and cars are present. In this paper, we propose and validate a framework for avoiding moving obstacles during visual navigation with a wheeled mobile robot. Visual navigation consists of following a path, represented as an ordered set of key images, which have been acquired by an on-board camera in a teaching phase. While following such a path, our robot is able to avoid static and moving obstacles, which were not present during teaching, and which are sensed by an on-board lidar. The proposed approach takes explicitly into account obstacle velocities, estimated using an appropriate Kalman-based observer. The velocities are then used to predict the obstacle positions within a tentacle-based approach. Finally, our approach is validated in a series of real outdoor experiments, showing that when the obstacle velocities are considered, the robot behavior is safer, smoother, and faster than when it is not.

A framework is presented in this paper for the control of a multisensor robot under several constraints. In this approach, the features coming from several sensors are treated as a single feature vector. The core of our approach is a weighting matrix that balances the contribution of each feature, allowing the taking of constraints into account. The constraints are considered as additional features that are smoothly injected in the control law. Multisensor modeling is introduced for the design of the control law, drawing similarities with linear quadratic control. The main properties are exposed and we propose several strategies to cope with the main drawbacks. The framework is validated in a complex experiment, illustrating various aspects of the approach. The goal is the positioning of a six-DOF robot arm with 3-D visual servoing. The considered constraints are both eye-in-hand and eye-to-hand visibility, together with joint limit avoidance. The system is thus highly overdetermined, yet the task can be performed while ensuring several combinations of constraints.

In this paper, we illustrate the application of a nonlinear active structure estimation from motion (SfM) strategy to three problems, namely 3-D structure estimation for 1) a point, 2) a sphere, and 3) a cylinder. In all three cases, an appropriate parameterization reduces the problem to the estimation of a single quantity. Knowledge of this estimated quantity and of the available measurements allows for then retrieving the full 3-D structure of the observed objects. Furthermore, in the point feature case, two different parameterizations based on either a planar or a spherical projection model are critically compared. Indeed, the two models yield, somehow unexpectedly, to different convergence properties for the SfM estimation task. The reported simulative and experimental results fully support the theoretical analysis and clearly show the benefits of the proposed active estimation strategy, which is in particular able to impose a desired transient response to the estimation error equivalent to that of a reference linear second-order system with assigned poles.

In this paper, a framework for visual servo control of a humanoid robot's upper body is presented. The framework is then implemented and tested on the REEM humanoid robot. The implementation is composed of 2 controllers - a head gaze control and a hand position control. The main application is precise manipulation tasks using the hand. For this, the hand controller takes top priority. The head controller is designed to keep both the hand and object in the eye field of view. For robustness, a secondary task of joint limit avoidance is implemented using the redundancy framework and a large projection operator proposed recently. For safety, joint velocity scaling is implemented. The implementation on REEM is done using the ROS and ViSP middleware. The results presented show simulations on Gazebo and experiments on the real robot. Furthermore, results with the real robot show how visual servoing is able to overcome some deficiency in REEM's kinematic calibration.

In this paper, we propose a new type of visual features for visual servoing: photometric moments. These global features do not require any segmentation, matching or tracking steps. The analytical form of the interaction matrix is developed in closed form for these features. Results from experiments carried out with photometric moments have been presented. The results validate our modelling and the control scheme. They perform well for large camera displacements and are endowed with a large convergence domain. From the properties exhibited, photometric moments hold promise as better candidates for IBVS over currently existing geometric and pure luminance features.

Moving obstacle avoidance is a fundamental requirement for any robot operating in real environments, where pedestrians, bicycles and cars are present. In this work, we design and validate a new approach that takes explicitly into account obstacle velocities, to achieve safe visual navigation in outdoor scenarios. A wheeled vehicle, equipped with an actuated pinhole camera and with a lidar, must follow a path represented by key images, without colliding with the obstacles. To estimate the obstacle velocities, we design a Kalman-based observer. Then, we adapt the tentacles designed in [1], to take into account the predicted obstacle positions. Finally, we validate our approach in a series of simulated and real experiments, showing that when the obstacle velocities are considered, the robot behaviour is safer, smoother, and faster than when it is not.

This paper deals with pose estimation using an iterative scheme. We show that using adequate visual information, pose estimation can be performed in a decoupling the estimation of translation and rotation. More precisely, we show that pose estimation can be achieved iteratively as a function of only three independent unknowns, which are the translation parameters. An invariant to rotational motion is used to estimate the camera position. Once the camera position is estimated, we show that the rotation can be estimated efficiently using a direct method. The proposed approach is compared against two classical methods from the literature. The results show that using our method, pose tracking in image sequences and the convergence rate for randomly generated poses are improved.

In this paper, we design and validate a new tentacle-based approach, for avoiding obstacles during appearance-based navigation with a wheeled mobile robot. In the past, we have developed a framework for safe visual navigation. The robot follows a path represented as a set of key images, and during obstacle circumnavigation, the on-board camera is actuated to maintain scene visibility. In those works, the model used for obstacle avoidance was obtained using a potential vector field. Here, a more sophisticated and efficient method, that exploits the robot kinematic model, and predicts collision at look-ahead distances, is designed and integrated in that framework. Outdoor experiments comparing the two models show that the new approach presents many advantages. Higher speeds and precision can be attained, very cluttered scenarios involving large obstacles can be successfully dealt with, and the control inputs are smoother.

A challenging task of airborne operations remains the landing on the carrier deck, which limits the carrier operational efficiency during rough sea. In this paper, a method of carrier visual detection and tracking is described. With the help of the aircraft sensors, the carrier is first detected in the image using a warped patch of a reference image. This provides an initialization to a real time 3D model-based tracker estimating the camera pose during the sequence. This method is demonstrated and evaluated using a simulator with high-fidelity visualization and on real video.

A low-level sensor fusion scheme is presented for the positioning of a multi-sensor robot. This non-hierarchical framework can be used for robot arms or other velocity-controlled robots, and is part of the task function approach. A stability analysis is presented for the general case, then several control laws illustrate the versatility of the framework. This approach is applied to the multi-camera eye-in-hand/eye-to-hand configuration in visual servoing. Experimental results point out the feasibility and the effectiveness of the proposed control laws. Mono-camera and multi-camera schemes are compared, showing that the proposed sensor fusion scheme improves the behavior of a robot arm.

This paper presents an image-based visual servoing for controlling the position and orientation of a quadrotor using a fixed downward camera observing landmarks on the level ground. In the proposed method, the negative feedback to the image moments is used to control the vertical motion and rotation around the roll axis. On the other hand, the negative feedback cannot be used to control the horizontal motion due to under-actuation of a quadrotor. Thus, a novel control method is introduced to control the horizontal motion. Simulations are presented to validate the proposed method.

We present and validate a framework for visual navigation with obstacle avoidance. The approach was originally designed in [1], but major improvements and real outdoor experiments are added here. Visual navigation consists of following a path, represented as an ordered set of key images, that have been acquired in a preliminary teaching phase. While following such path, the robot is able to avoid new obstacles which were not present during teaching, and which are sensed by a range scanner. We guarantee that collision avoidance and navigation are achieved simultaneously by actuating the camera pan angle, in the presence of obstacles, to maintain scene visibility as the robot circumnavigates the obstacle. The circumnavigation verse and the collision risk are estimated using a potential vector field derived from an occupancy grid. The framework can also deal with unavoidable obstacles, which make the robot decelerate and eventually stop.

The landing on carrier is a very difficult task even for trained pilots. This paper presents a method to land automatically using aircraft sensors and three visual features inspired by visual cues used by pilots. These features whose link with the aircraft state is established, are introduced in a control scheme using a linearized aircraft model. The control law demonstrates a large convergence domain using simulated visual features and a 3D tracker applied on synthetic images.

Joint limits avoidance is a crucial issue in sensor-based control. In this paper we propose an avoidance strategy based on a low-level data fusion. The joint positions of a robot arm are considered as features that are continuously added to the control scheme when they approach the joint limits, and removed when the position is safe. We expose an optimal tuning of the avoidance scheme, ensuring the main task is disturbed as little as possible. We propose additional strategies to solve the particular cases of unsafe desired position and local minima. The control scheme is applied to the avoidance of joint limits while performing visual servoing. Both simulation and experimental results illustrate the validity of our approach.

In this paper we address the issue of hybrid 2D/3D visual servoing. Contrary to popular approaches, we consider the position-based visual servo as the core of our scheme. 2D information is only added when necessary, that is when the observed object is about to leave from the field of view, even partially. The injection of 2D information is tuned by only one parameter, that simply defines the area where a point is considered to be near to the image border. Simulations allow the comparison of the proposed scheme with both position-based and hybrid schemes, showing nice properties. Finally, experiments are performed on a real object that is tracked by a pose estimation algorithm. Results validate the approach by showing the object stays entirely in the image field of view.

In this paper, the performance of a topological–metric visual-path-following framework is investigated in different environments. The framework relies on a monocular camera as the only sensing modality. The path is represented as a series of reference images such that each neighboring pair contains a number of common landmarks. Local 3-D geometries are reconstructed between the neighboring reference images to achieve fast feature prediction. This condition allows recovery from tracking failures. During navigation, the robot is controlled using image-based visual servoing. The focus of this paper is on the results from a number of experiments that were conducted in different environments, lighting conditions, and seasons. The experiments with a robot car show that the framework is robust to moving objects and moderate illumination changes. It is also shown that the system is capable of online path learning.

This paper is concerned with the use of a spherical-projection model for visual servoing from three points. We propose a new set of six features to control a 6-degree-of-freedom (DOF) robotic system with good decoupling properties. The first part of the set consists of three invariants to camera rotations. These invariants are built using the Cartesian distances between the spherical projections of the three points. The second part of the set corresponds to the angle-axis representation of a rotation matrix measured from the image of two points. Regarding the theoretical comparison with the classical perspective coordinates of points, the new set does not present more singularities. In addition, using the new set inside its nonsingular domain, a classical control law is proven to be optimal for pure rotational motions. The theoretical results and the robustness to points range errors of the new control scheme are validated through simulations and experiments on a 6-DOF robot arm.

This paper deals with visual servoing from three points. Using the geometric properties of the spherical projection of points, a new decoupled set of six visual features is proposed. The main originality lies in the use of the distances between spherical projection of points to define three features that are invariant to camera rotations. The three other features present a linear link with respect to camera rotations. In comparison with the classical perspective coordinates of points, the new decoupled set does not present more singularities. In addition, using the new set in its non-singular domain, a classical control law is proven to be ideal for rotational motions. These theoretical results as well as the robustness to errors of the new decoupled control scheme are illustrated through simulation results.

In this paper we propose a new on-line sensor self-calibration framework. The approach is to consider the sensor/robot interaction that links the sensor signal variations to the robot velocity. By on-line calibration, we mean only the actual measurements are used to perform calibration under the condition that the interaction matrix is analytically known. This allows us to propose a very simple and versatile formulation of sensor parameter calibration. Various sensors can be considered, and calibration from different sensory data may be considered within the same process. Intrinsic and extrinsic parameters estimation are formulated as a non-linear minimization problem the Jacobian of which can be expressed analytically from the sensor model. Simulations and experiments are presented for a camera observing four points, showing good results in the case of separated intrinsic and extrinsic calibration, and illustrating the possible limitations in the case of simultaneous estimation.

In this paper, we propose a new projection operator for the redundancy framework based on a task function defined as the norm of the usual error. This projection operator allows performing secondary tasks even when the main task is full rank. To ensure the convergence of the system, a switching strategy is then defined to switch from the new projection operator to the classical one before the norm of the total error reaches zero. An adaptive gain is also defined to slow down the convergence of the main task. It allows the secondary tasks to be active for longer. The experimental results obtained show the agreement with the analytical study and demonstrate the effectiveness of the proposed projection operator with respect to the classical one.

In this paper, we propose a framework for visual navigation with simultaneous obstacle avoidance. The obstacles are modeled by using a vortex potential field, derived from an occupancy grid. Kinematic redundancy guarantees that obstacle avoidance and navigation are achieved concurrently, and the whole scheme is merely sensor-based. The problem is solved both in an obstacle-free and in a dangerous context, and the control law is smoothened in the intermediate situations. In a series of simulations, we show that with our framework, a robot can replay a taught visual path while avoiding collisions, even in the presence of visual occlusions.

In this paper, we present a new redundancy-based strategy for avoiding joint limits of a robot arm. This strategy is based on defining three functions for each joint: an activation function; an adaptive gain function; and a tuning function. These functions allow determining automatically the required sign and the suitable magnitude for the avoidance process at each joint. The problem of adding an additional task with the main task and the avoidance process is also considered and solved. As for the redundancy framework, a new large projection operator based on the norm of the usual error is used to enlarge the redundancy domain for applying our proposed avoidance strategy. The experimental results obtained on a 6 dof robot arm in eye-in hand visual servoing show that the new avoidance strategy gives smooth joint avoidance behavior without any tuning step. Using the new projection operator allows a significant improvement of the joint avoidance process, especially in the case of a full rank task function.

This paper deals with the image-based visual servoing (IBVS), subject to constraints. Robot workspace limitations, visibility constraints, and actuators limitations are addressed. These constraints are formulated into state, output, and input constraints, respectively. Based on the predictive-control strategy, the IBVS task is written into a nonlinear optimization problem in the image plane, where the constraints can be easily and explicitly taken into account. Second, the contribution of the image prediction and influence of the prediction horizon are pointed out. The image prediction is obtained due to a model. The latter can be a local model based on the interaction matrix or a nonlinear global model based on 3-D data. Its choice is discussed with respect to the constraints to be handled. Finally, simulations that were obtained with a 6-degree-of-freedom (DOF) free-flying camera highlight the potential advantages of the proposed approach with respect to the image prediction and the constraint handling.

This paper presents a visual-servoing method that is based on 2-D ultrasound (US) images. The main goal is to guide a robot actuating a 2-D US probe in order to reach a desired cross-section image of an object of interest. The method we propose allows the control of both in-plane and out-of-plane probe motions. Its feedback visual features are combinations of moments extracted from the observed image. The exact analytical form of the interaction matrix that relates the image-moments time variation to the probe velocity is developed, and six independent visual features are proposed to control the six degrees of freedom of the robot. In order to endow the system with the capability of automatically interacting with objects of unknown shape, a model-free visual servoing is developed. For that, we propose an efficient online estimation method to identify the parameters involved in the interaction matrix. Results obtained in both simulations and experiments validate the methods presented in this paper and show their robustness to different errors and perturbations, especially those inherent to the noisy US images.

This paper proposes a generic decoupled image-based control scheme for cameras obeying the unified projection model. The scheme is based on the spherical projection model. Invariants to rotational motion are computed from this projection and used to control the translational degrees of freedom (DOFs). Importantly, we form invariants that decrease the sensitivity of the interaction matrix to object-depth variation. Finally, the proposed results are validated with experiments using a classical perspective camera as well as a fisheye camera mounted on a 6-DOF robotic platform.

Image-based visual servoing has been found to give satisfactory accurate and robust results. However, singularity and local minima may appear causing stability and convergence problems. In this paper, we present new control schemes based on Halley's method as a temptative to obtain a robust system even when the desired configuration is singular. The new control scheme use the first and the second order derivatives of the error to be regulated to zero. Hessian matrices of an image point are thus determined to be used in the control schemes. Preliminary experimental results obtained on a 6 dof eye-in-hand system shows that a more accurate positioning can be obtained compared with classical methods.

This paper presents a new model-free visual servoing that is able to servo a robotized 2D ultrasound probe that interacts with a soft tissue object. It makes direct use of the B-mode ultrasound images in order to reach a desired one. This approach does not require the 3D model of the object nor its location in the 3D space. The visual features are based on image moments. The exact analytical form of the interaction matrix relating the image moments variation to the probe velocity is modelled. To perform model-free servoing, the approach combines the image points coordinates with the probe pose to estimate efficiently 3D parameters required in the control law. The approach is validated with simulation and experimental results showing its robustness to different errors and perturbations.

In this paper a generic decoupled imaged-based control scheme for calibrated cameras obeying the unified projection model is proposed. The proposed decoupled scheme is based on the surface of object projections onto the unit sphere. Such features are invariant to rotational motions. This allows the control of translational motion independently from the rotational motion. Finally, the proposed results are validated with experiments using a classical perspective camera as well as a fisheye camera mounted on a 6 dofs robot platform.

In this paper, we present a controller for visual navigation, which utilizes a time-independent varying reference in the feedback law. The navigation framework relies on a monocular camera, and the path is represented as a series of key images. The varying reference is determined using a vector field, derived from the previous and next key images. Results in a simulated environment, as well as on a real robot, show the advantages of the varying reference, with respect to a fixed one, in the image, as well as in the 3D state space.

Image-based visual servo (IBVS) is a simple, efficient and robust technique for vision-based control. Although technically a local method in practice it demonstrates almost global convergence. However IBVS performs very poorly for cases that involve large rotations about the optical axis. It is well known that re-parameterizing the problem by using polar, instead of Cartesian coordinates, of feature points overcomes this limitation. First, simulation and experimental results are presented to show the complementarity of these two parameterizations. We then describe a new hybrid visual servo strategy based on combining polar and Cartesian image Jacobians.

A catadioptric vision system combines a camera and a mirror to achieve a wide field of view imaging system. This type of vision system has many potential applications in mobile robotics. This paper is concerned with the design of a robust image-based control scheme using a catadioptric vision system mounted on a mobile robot. We exploit the fact that the decoupling property contributes to the robustness of a control method. More precisely, from the image of a point, we propose a minimal and decoupled set of features measurable on any catadioptric vision system. Using the minimal set, a classical control method is proved to be robust in the presence of point range errors. Finally, experimental results with a coarsely calibrated mobile robot validate the robustness of the new decoupled scheme.

In this paper, we investigate a range of image-based visual servo control algorithms for regulation of the position of a quadrotor aerial vehicle. The most promising control algorithms have been successfully implemented on an autonomous aerial vehicle and demonstrate excellent performance.

This paper is concerned with improvements to visual feature modeling using a spherical projection model. Three spherical targets are considered: a sphere, a sphere marked with a tangent vector to a point on its surface, and a sphere marked with two points on its surface. A new minimal and decoupled set of visual features is proposed for each target using any central catadioptric camera. Using the proposed set for a sphere, a classical control law is proved to be globally asymptotically stable in the presence of modeling errors and locally asymptotically stable in the presence of calibration errors, considering that perspective and paracatadioptric cameras were used. Simulation and experimental results with perspective, paracatadioptric, and fish-eye cameras validate the proposed theoretical results.

We present an image-based visual servoing controller enabling nonholonomic mobile robots with a fixed pinhole camera to reach and follow a continuous path on the ground. The controller utilizes only a small set of features extracted from the image plane, without using the complete geometric representation of the path. A Lyapunov-based stability analysis is carried out. The performance of the controller is validated and compared by simulations and experiments on a car-like robot equipped with a pinhole camera.

This paper focuses on visual servoing from a special target using a vision sensor with an unique projection center. The target is a sphere marked with two points on its surface. Using a spherical projection model, a new minimal set of six features is proposed for this target. Spherical moments are exploited to compute the new set on the image of the sensor. Using the new set, a classical control method is proved to be asymptotically stable even in the presence of modeling errors. Experimental results with a perspective camera and a fish-eye camera validate the proposed theoretical results.

This paper proposes a new way to achieve robotic tasks by visual servoing. Instead of using geometric features (points, straight lines, pose, homography, etc.) as it is usually done, we use directly the luminance of all pixels in the image. Since most of the classical control laws fail in this case, we turn the visual servoing problem into an optimization problem leading to a new control law. Experimental results validate the proposed approach and show its robustness regarding to approximated depths, non Lambertian objects and partial occlusions.

In this paper, we analyze and compare five image-based visual servoing control laws. Three of them are classical while two new ones are proposed. The first new control law is based on a behavior controller to adjust the movement of the camera. It can also be used to switch between the classical methods. The second control law is designed to try to obtain the global stability of the system. An analytical study of all control schemes when translational motion along and rotational motion around the optical axis is also presented. Finally, simulation and experimental results show that the new control law with a behavior controller has a wider range of success than the other control schemes and can be used to avoid local minima and singularities.

A new visual servoing method based on B-mode ultrasound images is proposed to automatically control the motion of a 2D ultrasound probe held by a medical robot in order to reach a desired B-scan image of an object of interest. In this approach, combinations of image moments extracted from the current observed object cross-section are used as feedback visual features. The analytical form of the interaction matrix, relating the time variation of these visual features to the probe velocity, is derived and used in the control law. Simulations performed with a static ultrasound volume containing an egg-shaped object, and in-vitro experiments using a robotized ultrasound probe that interacts with a rabbit heart immersed in water, show the validity of this new approach and its robustness with respect to modeling and measurements errors.

In this paper a new decoupled imaged-based control scheme is proposed from projection onto a unit sphere. This control scheme is based on moment invariants to 3D rotational motion. This allows the control of translational motion independently of the rotational one. First, the analytical form of the interaction matrix related to the spherical moments is derived. It is based on the projection of a set of points onto a unit sphere. From the spherical moment, six features are presented to control the full 6 degrees of freedom. Finally, the results are validated through realistic simulation results.

We present a visual servoing scheme enabling non-holonomic mobile robots with a fixed pinhole camera to reach and follow a continuous path on the ground. The controller utilizes only a small set of features extracted from the image plane, without using the complete geometric representation of the path. The scheme is position-based, and a Lyapunov-based stability analysis is carried out. The performance of our control design is experimentally validated on a car-like robot equipped with a pinhole camera.

This paper is concerned with the use of a spherical projection model to design optimal visual features for visual servoing. Here two special targets are considered: a sphere marked with two points on its surface and a sphere marked with a tangent vector to a point on its surface. For these targets we propose a new minimal set of six visual features which can be computed on classical perspective cameras. Using the new set, a classical control method is proved to be globally stable even in the presence of modeling error. In comparison with the previous set originally proposed for the second target, the new set draws a better camera trajectory. Finally, simulation and experimental results confirm the validity of the proposed theoretical results.

This paper focuses on the way to achieve accurate visual servoing tasks when the shape of the object being observed as well as the desired image are unknown. More precisely, we want to control the camera orientation with respect to the tangent plane at a certain object point corresponding to the center of a region of interest. We also want to observe this point at the principal point to fulfil a fixation task. A 3-D reconstruction phase must, therefore, be performed during the camera motion. Our approach is then close to the structure-from-motion problem. The reconstruction phase is based on the measurement of the 2-D motion in a region of interest and on the measurement of the camera velocity. Since the 2-D motion depends on the shape of the objects being observed, we introduce a unified motion model to cope both with planar and nonplanar objects. However, since this model is only an approximation, we propose two approaches to enlarge its domain of validity. The first is based on active vision, coupled with a 3-D reconstruction based on a continuous approach, and the second is based on statistical techniques of robust estimation, coupled with a 3-D reconstruction based on a discrete approach. Theoretical and experimental results compare both approaches.

In this paper, we consider the problem of controlling a 6 DOF holonomic robot and a nonholonomic mobile robot from the projection of 3-D straight lines in the image plane of central catadioptric systems. A generic central catadioptric interaction matrix for the projection of 3-D straight lines is derived using an unifying imaging model valid for an entire class of cameras. This result is exploited to design an image-based control law that allows us to control the 6 DOF of a robotic arm. Then, the projected lines are exploited to control a nonholonomic robot. We show that as when considering a robotic arm, the control objectives are mainly based on catadioptric image feature and that local asymptotic convergence is guaranteed. Simulation results and real experiments with a 6 DOF eye-to-hand system and a mobile robot illustrate the control strategy.

Autonomous cars will likely play an important role in the future. A vision system designed to support outdoor navigation for such vehicles has to deal with large dynamic environments, changing imaging conditions, and temporary occlusions by other moving objects. This paper presents a novel appearance-based navigation framework relying on a single perspective vision sensor, which is aimed towards resolving of the above issues. The solution is based on a hierarchical environment representation created during a teaching stage, when the robot is controlled by a human operator. At the top level, the representation contains a graph of key-images with extracted 2D features enabling a robust navigation by visual servoing. The information stored at the bottom level enables to efficiently predict the locations of the features which are currently not visible, and eventually (re-)start their tracking. The outstanding property of the proposed framework is that it enables robust and scalable navigation without requiring a globally consistent map, even in interconnected environments. This result has been confirmed by realistic off-line experiments and successful real-time navigation trials in public urban areas.

In this paper, we propose a visual servoing scheme to align an airplane with respect to a runway. A linearized model of the airplane dynamics and decoupled visual features are designed to build the control scheme. Simulation results are obtained with a quite realistic flight simulator which is based on a non linear airplane dynamic model. They show that the airplane realizes correctly the specified task by using visual data.

This paper presents a method to achieve visual servoing tasks when the shape of the object being observed as well as the final image are unknown. Therefore, a reconstruction phase has to be performed during the camera motion. More precisely, the reconstruction phase is based on the measurement of the 2D motion in a region of interest and on the measurement of the camera velocity. Since the 2D motion depends on the shape of objects being observed, we introduce an unified motion model to cope as well with planar as with non-planar objects. This is an improvement of our previous works described by Lhaj et al. (2003), Collewet et al. (2004) where only planar objects had been considered. However, since this model is only an approximation, we propose to use active vision to enlarge its domain of validity. Experimental results validate the proposed approach.

The aim of this article is to achieve accurate visual servoing tasks when the shape of the object being observed as well as the final image are unknown. More precisely, we want to control the orientation of the tangent plane at a certain point on the object corresponding to the center of a region of interest and to move this point to the principal point to fulfill a fixation task. To do that, we perform a 3D reconstruction phase during the servoing. It is based on the measurement of the 2D displacement in the region of interest and on the measurement of the camera velocity. Since the 2D displacement depends on the scene, we introduce an unified motion model to deal with planar as well with non-planar objects. Unfortunately, this model is only an approximation. We proposed to use active vision to enlarge its domain of validity and a 3D reconstruction based on a continuous approach (2007). In this paper, we propose to use robust estimation techniques and a 3D reconstruction based on discrete approach. Experimental results compare both approaches.

In this paper, we investigate the use of a spherical projection model to search for optimal visual features for visual servoing. A new minimal set of three visual features is proposed for visual servoing from spheres using any central catadioptric system. Using this set of features, a classical control method is proved to be globally stable even in the presence of modeling error and locally stable to calibration errors on perspective cameras. Using this type of cameras, experimental results are presented and validate the proposed theoretical results.

In this paper, we apply a general framework for building complex whole-body control for highly redundant robot, and we propose to implement it for visually-guided grasping while walking on a humanoid robot. The key idea is to divide the control into several sensor-based control tasks that are simultaneously executed by a general structure called stack of tasks. This structure enables a very simple access for task sequencing, and can be used for task-level control. This framework was applied for a visual servoing task. The robot walks along a planned path, keeping the specified object in the middle of its field of view and finally, when it is close enough, the robot grasps the object while walking.

Increasing the field of view of camera is an important issue practical in robot vision. One solution is to consider catadioptric camera that allows a 360deg field of view. In this paper we propose a 3D model tracking algorithm that allows a fast and reliable tracking of 3D objects within central catadioptric images. The proposed approach relies on the virtual visual servoing approach. All the modeling aspects have been reconsidered to consider the projection model. Results show the method to be robust and efficient.

Calibration techniques allow the estimation of the intrinsic parameters of a camera. This paper describes an adaptive visual servoing scheme which employs the visual data measured during the task to determine the camera intrinsic parameters. This approach is based on the virtual visual servoing approach. However, in order to increase the robustness of the calibration several aspects have been introduced in this approach with respect to the previous developed virtual visual servoing systems. Furthermore, the system is able to determine the value of the intrinsic parameters when they vary during the task. This approach has been tested using an eye-in-hand robotic system.

In this paper, a visual servoing scheme is proposed to control an airplane during its landing. A linearized model of the airplane dynamics and decoupled visual features are used to build the control scheme. A desired trajectory which takes into account the airplane dynamic is designed. Coupling this trajectory and the control law enables the airplane to join its desired path. Then the airplane is controlled to follow the glide path, realize the flare manoeuvre and finally touchdown. Simulation results are obtained with a quite realistic flight simulator which is based on a non linear airplane dynamic model. They show that the airplane is able to land automatically by using visual data.

In this paper the performance of a topological- metric visual path following framework is investigated in different environments. The framework relies on a monocular camera as the only sensing modality. The path is represented as a series of reference images such that each neighboring pair contains a number of common landmarks. Local 3D geometries are reconstructed between the neighboring reference images in order to achieve fast feature prediction which allows the recovery from tracking failures. During navigation the robot is controlled using image-based visual servoing. The experiments show that the framework is robust against moving objects and moderate illumination changes. It is also shown that the system is capable of on-line path learning.

This document provides a summary to a short video with the same title. The video shows the French intelligent transportation vehicle CyCab performing visual path following using only monocular vision. All phases of the process are shown with a spoken commentary. In the teaching phase, the user drives the robot manually while images from the camera are stored. Key images with corresponding images features are stored as a map together with 2D and 3D local information. In the navigation phase, CyCab follows the learned path by tracking the images features projected from the map and with a simple visual servoing control law.

Classical sensor-based approaches tend to constrain all the degrees of freedom of a robot during the execution of a task. In this paper, a new solution is proposed. The key idea is to divide the global full-constraining task into several subtasks, which can be applied or inactivated to take into account potential constraints of the environment. Far from any constraint, the robot moves according to the full task. When it comes closer to a configuration to avoid, a higher level controller removes one or several subtasks, and activates them again when the constraint is avoided. The last controller ensures the convergence at the global level by introducing some look-ahead capabilities when a local minimum is reached. The robot accomplishes the global task by automatically sequencing sensor-based tasks, obstacle avoidance, and short deliberative phases. In this paper, a complete solution to implement this idea is proposed, along with several experiments that prove the validity of this approach

Positioning a robot with respect to objects by using data provided by a camera is a well known technique called visual servoing. In order to perform a task, the object must exhibit visual features which can be extracted from different points of view. Then, visual servoing is object-dependent as it depends on the object appearance. Therefore, performing the positioning task is not possible in presence of non-textured objects or objects for which extracting visual features is too complex or too costly. This paper proposes a solution to tackle this limitation inherent to the current visual servoing techniques. Our proposal is based on the coded structured light approach as a reliable and fast way to solve the correspondence problem. In this case, a coded light pattern is projected providing robust visual features independently of the object appearance

This paper addresses the design of a control law for vision-based robot navigation. The method proposed is based on a topological representation of the environment. Within this context, a learning stage enables a graph to be built in which nodes represent views acquired by the camera, and edges denote the possibility for the robotic system to move from one image to an other. A path finding algorithm then gives the robot a collection of views describing the environment it has to go through in order to reach its desired position. This article focuses on the control law used for controlling the robot motion's online. The particularity of this control law is that it does not require any reconstruction of the environment, and does not force the robot to converge towards each intermediary position in the path. Landmarks matched between each consecutive views of the path are considered as successive features that the camera has to observe within its field of view. An original visual servoing control law, using specific features, ensures that the robot navigates within the visibility path. Simulation results demonstrate the validity of the proposed approach

Image moments provide an important class of image features used for image-based visual servo control. Spherical image moments have the additional desirable property that they are invariant under rotation of the camera frame. For these features one can study the local and global stability and performance of the position control independently of the rotation control. In this paper we study a range of control algorithms including the classical approximately linearising control, a recently proposed robust control based on Lyapunov function design methodology, and modifications of these designs to improve global and asymptotic performance and robustness of the schemes. The comparison of performance demonstrates that the choice of image feature and control design for image-based visual servo are each equally important and highly coupled. We finally propose a control law using a modified image feature and a Lyapunov control design that ensures global asymptotic stability and good performance equally in image space as in task space

In this paper, the coupling between Jacobian learning and task sequencing through the redundancy approach is studied. It is well known that visual servoing is robust to modeling errors in the Jacobian matrices. This justifies why Jacobian estimation does not usually degrade the system convergence. However, we show that this is not true any more when the redundancy formalism is used. In this case the Jacobian matrix is also necessary to compute projection operators for task decomposition, which is quite sensitive to errors. We show that learning improves the servoing performance, when task sequencing is used. Conversely, sequencing improves the convergence of learning, especially for tasks involving several degrees of freedom. Eye-in-hand and eye-to-hand experiments have been performed on two robots with six degrees of freedom

This paper describes an original control law called qualitative servoing. The particularity of this method is that no specific desired value is specified for the visual features involved in the control scheme. Indeed, visual features are only constrained to belong to a confident interval, which gives more flexibility to the system. While this formalism can be used for several types of visual features, it is used in this paper for improving the on-line control of the visibility of a target. The principle is to make a compromise between the classical positioning task and the visibility constraint. Experimental results obtained with a six degrees of freedom robot arm are presented, demonstrating the performance of the proposed method

A fundamental step toward broadening the use of real-world image-based visual servoing is to deal with the important issue of reliability and robustness. In order to address this issue, a closed-loop control law is proposed that simultaneously accomplishes a visual servoing task and is robust to a general class of image processing errors. This is achieved with the application of widely accepted statistical techniques such as robust M-estimation and LMedS. Experimental results are presented which demonstrate visual servoing tasks that resist severe outlier contamination.

This paper considers the problem of positioning an eye-in-hand system so that it becomes parallel to a planar object. Our approach to this problem is based on linking to the camera a structured light emitter designed to produce a suitable set of visual features. The aim of using structured light is not only for simplifying the image processing and allowing low-textured objects to be considered, but also for producing a control scheme with nice properties like decoupling, convergence, and adequate camera trajectory. This paper focuses on an image-based approach that achieves decoupling in all the workspace, and for which the global convergence is ensured in perfect conditions. The behavior of the image-based approach is shown to be partially equivalent to a 3-D visual servoing scheme, but with a better robustness with respect to image noise. Concerning the robustness of the approach against calibration errors, it is demonstrated both analytically and experimentally

Tracking is a very important research subject in a real-time augmented reality context. The main requirements for trackers are high accuracy and little latency at a reasonable cost. In order to address these issues, a real-time, robust, and efficient 3D model-based tracking algorithm is proposed for a "video see through" monocular vision system. The tracking of objects in the scene amounts to calculating the pose between the camera and the objects. Virtual objects can then be projected into the scene using the pose. Here, nonlinear pose estimation is formulated by means of a virtual visual servoing approach. In this context, the derivation of point-to-curves interaction matrices are given for different 3D geometrical primitives including straight lines, circles, cylinders, and spheres. A local moving edges tracker is used in order to provide real-time tracking of points normal to the object contours. Robustness is obtained by integrating an M-estimator into the visual control law via an iteratively reweighted least squares implementation. This approach is then extended to address the 3D model-free augmented reality problem. The method presented in this paper has been validated on several complex image sequences including outdoor environments. Results show the method to be robust to occlusion, changes in illumination, and mistracking.

In this paper, two real-time pose tracking algorithms for rigid objects are compared. Both methods are 3D-model based and are capable of calculating the pose between the camera and an object with a monocular vision system. Here, special consideration has been put into defining and evaluating different performance criteria such as computational efficiency, accuracy and robustness. Both methods are described and a unifying framework is derived. The main advantage of both algorithms lie in their real-time capabilities (on standard hardware) whilst being robust to miss-tracking, occlusion and changes in illumination.

The paper contributes to presenting both an accurate and robust shape tracking algorithm and a novel visual servoing method. Two steps are involved in the tracking algorithm. Firstly the object shape is assumed to vary under an affine model, and the edge detection is performed along the normal lines to the contour. As a result it is possible to use a Kalman filter to perform efficient tracking. The second step concerns image matching based on perspective model, which is achieved iteratively by searching locally along the normal lines also. As to visual servoing we propose to control the translations of the robot with the normalized zeroth and first order image moments, and to control the orientation with rotation axis and angle extracted from a Homography matrix. Two experiments demonstrate that the tracking algorithm is accurate and robust enough to be used in visual servoing, and the novel visual servoing method is superior to traditional ones.

Classical visual servoing approaches tend to constrain all degrees of freedom (DOF) of the robot during the execution of a task. In this article a new approach is proposed. The key idea is to control the robot with a very under-constrained task when it is far from the desired position, and to incrementally constrain the global task by adding further tasks as the robot moves closer to the goal. As long as they are sufficient, the remaining DOF are used to avoid undesirable configurations, such as joint limits. Closer from the goal, when not enough DOF remain available for avoidance, an execution controller selects a task to be temporary removed from the applied tasks. The released DOF can then be used for the joint limits avoidance. A complete solution to implement this general idea is proposed. Experiments that prove the validity of the approach are also provided.

Moments are generic (and usually intuitive) descriptors that can be computed from several kinds of objects defined either from closed contours or from a set of points. In this paper, image moments are used in two new methods for the pose estimation of a planar object observed through full perspective model. The first method is based on an iterative optimization scheme formulated as virtual visual servoing, while the second is based on an exhaustive but efficient optimization scheme of the two most critical parameters. It allows to avoid local minima. We finally present some experimental results to validate the theoretical developments presented in this paper.

A new visual servoing technique based on 2D ultrasound image is proposed in order to control the motion of an ultrasound probe held by a medical robot. In opposition to a standard camera which provides a projection of the 3D scene to a 2D image, ultrasound information is strictly in the observation plan of the probe and consequently visual servoing techniques have to be adapted. In this paper the coupling between the ultrasound probe and a motionless crossed string phantom used for probe calibration is modeled. Then a robotic task is developed which consists to position the ultrasound image on the intersection point of the crossed string phantom while moving the probe to different orientations. The goal of this task is to optimize the procedure of spatial parameters calibration of 3D ultrasound systems.

The paper presents a new approach to construct a control law that realizes a main task and simultaneously takes supplementary constraints into account. Classically, this is done by using the redundancy formalism. If the main task does not constrain all the motions of the robot, a secondary task can be achieved by using only the remaining degrees of freedom (DOF). We propose a new general method that frees up some of the DOF constrained by the main task in addition of the remaining DOF. The general idea is to enable the motions produced by the secondary control law that help the main task to be completed faster. The main advantage is to enhance the performance of the secondary task by enlarging the number of available DOF. In a formal framework, a projection operator is built which ensures that the secondary control law does not disturb the main task. A control law can be then easily computed from the two tasks considered. Experiments that implement and validate this approach are proposed. The visual servoing framework is used to position a 6-DOF robot while simultaneously avoiding occlusions and joint limits.

This paper focuses on the problem of realizing a plane-to-plane virtual link between a camera attached to the end-effector of a robot and a planar object. In order to do the system independent to the object surface appearance, a structured light emitter is linked to the camera so that 4 laser pointers are projected onto the object. In a previous paper we showed that such a system has good performance and nice characteristics like partial decoupling near the desired state and robustness against misalignment of the emitter and the camera (J. Pages et al., 2004). However, no analytical results concerning the global asymptotic stability of the system were obtained due to the high complexity of the visual features utilized. In this work we present a better set of visual features which improves the properties of the features in (J. Pages et al., 2004) and for which it is possible to prove the global asymptotic stability.

Moments are generic (and usually intuitive) descriptors that can be computed from several kinds of objects, defined either from closed contours or from a set of points. In this paper, we present improvements in image-based visual servo using image moments. First, the analytical form of the interaction matrix related to the moments computed from a set of coplanar points is derived, and we show that it is different from the form obtained previously, using coplanar closed contours. Six visual features are selected to design a decoupled control scheme when the object is parallel to the image plane. This nice property is then generalized to the case where the desired object position is not parallel to the image plane. Finally, experimental results are presented to illustrate the validity of our approach and its robustness, with respect to modeling errors.

We present a way to achieve positioning tasks by model-free visual servoing in the case of planar and motionless objects whose shape is unknown. Emphasize is made on the algorithm of 3D reconstruction which allows to synthesize easily the control law. More precisely, the reconstruction phase is based on the measurement of the 2D displacements in a region of interest and on the measurement of the camera velocity. However, we will show that the proposed algorithm is robust with respect to inaccurate values of this velocity. 2D displacements rather than 2D motions are used to remove the assumption that the acquisition rate has to be high. In addition, a particular attention is paid to the complex case of large displacements to access high camera velocities. Once the parameters of the plane are sufficiently stable, a visual servoing scheme is used to control the orientation of the camera with respect to the object and to ensure that it remains in the camera field of view for any desired orientation. The 3D reconstruction phase is maintained active during the servoing to improve the accuracy of the parameters and, consequently, to obtain a small positioning error. Experimental results validate the proposed approach.

The work presents a method for estimating the inverse Jacobian matrix of a function, without computing the direct Jacobian matrix. The resulting inverse Jacobian matrix is shown to perform much better in modelling a relation /spl theta/ = f/sup -1/ (x) than the classical Moore-Penrose inverse J/sup +//sub f/. Theoretical insight as well as comparisons in the domain of visual servoing are provided to demonstrate this assertion.

A fundamental step towards broadening the use of real world image-based visual servoing is to deal with the important issues of reliability and robustness. In order to address this issue, a closed loop control law is proposed that simultaneously accomplishes a visual servoing task and is robust to a general class of image processing errors. This is achieved with the application of widely accepted statistical techniques of robust M-estimation. Furthermore, improvement have been added in the weight computation process: memory, initialization. Indeed, when the error between current visual features and desired ones are large, which occurs when large robot displacement are required, M-estimator may not detect outliers. To address this point, the method we propose to initialize the confidence in each feature is based on the LMedS estimators. Experimental results are presented which demonstrate visual servoing tasks which resist severe outlier contamination.

This article presents a new approach for robot motion control, using images acquired by an on-board camera. A particularity of this method is that it can avoid reconstructing the entire scene without limiting the displacements possible. To achieve this, an image base of the environment is used to describe the navigation space. We extract from this base a sequence of overlapping images which define the zone that the robot must traverse, in order to reach the desired position. Motions are computed on-line using only points of interest extracted from these images. A method based on potential field theory has been adapted in order to ensure a sufficient visibility of these features during the entire motion of the robot. Experimental results obtained on a six degrees of freedom robotic system are presented and confirm the validity of our approach.

Moments are generic and (usually) intuitive descriptors that can be computed from several kinds of objects defined either from closed contours (continuous object) or a set of points (discrete object). In this paper, we propose to use moments to design a decoupled image-based visual servo. The analytical form of the interaction matrix related to the moments computed for a discrete object is derived, and we show that it is different of the form obtained for a continuous object. Six visual features are selected to design a decoupled control scheme when the object is parallel to the image plane. This nice property is then generalized to the case where the desired object position is not parallel to the image plane. Finally, experimental results are presented to illustrate the validity of our approach and its robustness with respect to modeling errors.

This paper proposes a real-time, robust and efficient 3D model-based tracking algorithm for visual servoing. A virtual visual servoing approach is used for monocular 3D tracking. This method is similar to more classical nonlinear pose computation techniques. A concise method for derivation of efficient distance-to-contour interaction matrices is described. An oriented edge detector is used in order to provide real-time tracking of points normal to the object contours. Robustness is obtained by integrating a M-estimator into the virtual visual control law via an iteratively reweighted least squares implementation. The method presented in this paper has been validated on several 2D 1/2 visual servoing experiments considering various objects. Results show the method to be robust to occlusion, changes in illumination and miss-tracking.

Classical visual servoing approaches tend to constrain all degrees of freedom (DOF) of the robot during a task's execution. In this article a new approach is proposed. The key idea is to control the robot with a very under-constrained task when it is far of the desired position, and to incrementally constrain the global task by adding further tasks as the robot moves closer to the goal. A method is first proposed that stacks elementary tasks until the robot is fully constrained. To insure the continuity of the articular velocities when adding constraints, a new control law is then proposed. Experiments that prove the interest of the approach are also provided.

In this paper we consider the problem of controlling a robotic system using the projection of 3D lines in the image plane of central catadioptric systems. Most of the efforts in visual servoing are devoted to points, only few works have investigated the use of lines in visual servoing with traditional cameras and none has explored the case of omnidirectional cameras. First a generic central catadioptric interaction matrix for the projection of 3D straight lines is derived from the projection model of an entire class of camera. Then an image-based control law is designed and validated through simulation results.

In this paper we face the problem of positioning a camera attached to the end-effector of a robotic manipulator so that it gets parallel to a planar object. Such problem has been treated for a long time in visual servoing. Our approach is based on linking to the camera several laser pointers so that its configuration is aimed to produce a suitable set of visual features. The aim of using structured light is not only for easing the image processing and to allow low-textured objects to be treated, but also for producing a control scheme with nice properties like decoupling, stability, well conditioning and good camera trajectory.

In this paper, we determine the analytical form of the interaction matrix related to any moment that can be computed from segmented images. The derivation method we present is based on Green's theorem. We apply this general result to classical geometrical primitives. We then consider using moments in image-based visual servoing. For that, we select six combinations of moments to control the six degrees of freedom of the system. These features are particularly adequate, if we consider a planar object and the configurations such that the object and camera planes are parallel at the desired position. The experimental results we present show that a correct behavior of the system is obtained if we consider either a simple symmetrical object or a planar object with complex and unknown shape.

This paper deals with the way to achieve positioning tasks by visual servoing in the case of planar and motionless objects whose shape is unknown. In fact, we consider here complex objects like those one can encounter in the agrifood industry. A 3D reconstruction phase is first considered yielding to the parameters of the plane. This computation is based on the measurement of the 2D velocity in a region of interest and on the measurement of the camera velocity. Once the parameters of the plane are sufficiently stable, a visual servoing scheme is used to control the orientation of the camera with respect to the object and to ensure that it remains in the camera field of view for any desired orientation. This 3D reconstruction phase is maintained active during the servoing to improve the accuracy of the parameters and, consequently, to obtain a small positioning error. Finally, experimental results validate the proposed approach as well as for its effectiveness than for the obtained accuracy.

In this paper, we present how moment invariants can be used to design a decoupled 2D visual servoing scheme and to minimise the nonlinearity of the interaction matrix related to the selected visual features. Experimental results using a 6 DOF eye-in-hand system to position a camera parallel to planar objects of complex shape are presented to demonstrate the efficiency of the proposed method.

A fundamental step towards broadening the use of real world image-based visual servoing is to deal with the important issues of reliability and robustness. In order to address this issue, a closed loop control law is proposed that simultaneously accomplishes a visual servoing task and is robust to a general class of external errors. This generality allows concurrent consideration of a wide range of errors including: noise from image feature extraction, small scale errors in the tracking and even large scale errors in the matching between current and desired features. This is achieved with the application of widely accepted statistical techniques of robust M-estimation. The M-estimator is integrated by an iteratively re-weighted method. The median absolute deviation is used as an estimate of the standard deviation of the inlier data and is compared with other methods. This combination is advantageous because of its high efficiency, high breakdown point and desirable influence functions. The robustness and stability of the control law is shown to be dependent on a subsequent measure of position uncertainty. Furthermore the convergence criteria of the control law are investigated. Experimental results are presented which demonstrate visual servoing tasks which resist severe outlier contamination.

Augmented reality has now progressed to the point where real-time applications are required and being considered. At the same time it is important that synthetic elements are rendered and aligned in the scene in an accurate and visually acceptable way. In order to address these issues a real-time, robust and efficient 3D model-based tracking algorithm is proposed for a 'video see through' monocular vision system. The tracking of objects in the scene amounts to calculating the pose between the camera and the objects. Virtual objects can then be projected into the scene using the pose. Here, non-linear pose computation is formulated by means of a virtual visual servoing approach. In this context, the derivation of point-to-curve interaction matrices is given for different features including lines, circles, cylinders and spheres. A local moving edge tracker is used in order to provide real-time tracking of points normal to the object contours. A method is proposed for combining local position uncertainty and global pose uncertainty in an efficient and accurate way by propagating uncertainty. Robustness is obtained by integrating an M-estimator into the visual control law via an iteratively re-weighted least squares implementation. The method presented in this paper has been validated on several complex image sequences including outdoor environments. Results show the method to be robust to occlusion, changes in illumination and mistracking.

An image-based visual servo control is presented for a robotic manipulator. The proposed control design addresses visual servo of 'eye-in-hand' type systems. Using a novel representation of the visual error based on a spherical representation of target centroid information along with a measure of the rotation between the camera and the target, the control of the position and orientation is decoupled. A non-linear gain introduced into the orientation feedback kinematics prevents the target image from leaving the visual field. Semi-global convergence of the closed loop system is proved.

The principal deficiency of image-based visual servoing is that the induced (3D) trajectories are not optimal and sometimes, especially when the displacement to realize is large, these trajectories are not physically valid leading to the failure of the servoing process. Furthermore, visual control needs a matching step between the features extracted from the initial image and the desired one. This step can be problematic (or impossible) when the camera displacement between the acquisitions of the initial and desired images is large and/or for complex scenes. To resolve these deficiencies, we couple an image interpolation process between N relay images extracted from a database and an image-based trajectory tracking. The camera calibration and the model of the observed scene are not assumed to be known. The relay images are interpolated in such a way that the corresponding camera trajectory is minimal. First a closed form collineation path is obtained and then the analytical form of image features trajectories are derived and efficiently tracked using a purely image-based control. Experimental results obtained on a six DOF eye-in-hand robotic system are presented and confirm the validity of the proposed approach.

In this paper, we determine the analytical form of the interaction matrix related to any moments that can be computed from binary or segmented images. We then apply this theoretical result to image-based visual servoing by selecting six combinations of moments able to control the six DOF of the system. The experimental results we present show that a correct behavior of the system is obtained if we consider either a simple symmetrical object or a planar object with complex and unknown shape.

This paper is concerned with choosing image features for image based visual servo control and how this choice influences the closed-loop dynamics of the system. In prior work, image features tend to be chosen on the basis of image processing simplicity and noise sensitivity. In this paper we show that the choice of feature directly influences the closed-loop dynamics in task-space. We focus on the depth axis control of a visual servo system and compare analytically various approaches that have been reported recently in the literature. The theoretical predictions axe verified by experiment.

This paper presents a framework to achieve real-time augmented reality applications. We propose a framework based on the visual servoing approach well known in robotics. We consider pose or viewpoint computation as a similar problem to visual servoing. It allows one to take advantage of all the research that has been carried out in this domain in the past. The proposed method features simplicity, accuracy, efficiency, and scalability wrt. to the camera model as well as wrt. the features extracted from the image. We illustrate the efficiency of our approach on augmented reality applications with various real image sequences.

This paper proposes a way to achieve positioning tasks by 2-D visual servoing when the desired image of the observed object cannot be precisely described. The object is assumed to be planar and motionless, but no knowledge about its shape or pose is required. First, we treat the case of a threadlike object, and then we show how our approach can be generalized to an object with three particular points. The control law is based on the use of 2-D visual servoing, and on an estimation of two 3-D parameters. We show that this control scheme is not sensitive to the calibration of the camera. We conclude this paper by experimental results relative to objects of unknown shape. In addition, an algorithm to estimate the depth between the object and the camera is provided, which finally leads to a 3-D estimation of the object shape.

This paper concerns the stability analysis of a new class of model-free visual servoing methods. These methods are "model-free" since they are based on the estimation of the relative camera orientation between two views of an object without knowing its 3-D model. The visual servoing is decoupled by controlling the rotation of the camera separately from the rest of the system. The way the remaining degrees of freedom are controlled differentiates the methods within the class. For all the methods of the class, the robustness with respect to both camera and hand-eye calibration errors can be analytically studied. In some cases, necessary and sufficient conditions can be found not only for the local asymptotic stability but also for the global asymptotic stability. In the other cases, simple conditions on the calibration errors are sufficient to ensure the global asymptotic stability of the control law. In addition to the theoretical proof of the stability, the experimental results prove the validity of the control strategy proposed in the paper.

Vision feedback control loop techniques are efficient for a large class of applications, but they come up against difficulties when the initial and desired robot positions are distant. Classical approaches are based on the regulation to zero of an error function computed from the current measurement and a constant desired one. By using such an approach, it is not obvious how to introduce any constraint in the realized trajectories or to ensure the convergence for all the initial configurations. In this paper, we propose a new approach to resolve these difficulties by coupling path planning in image space and image-based control. Constraints such that the object remains in the camera field of view or the robot avoids its joint limits can be taken into account at the task planning level. Furthermore, by using this approach, current measurements always remain close to their desired value, and a control by image-based servoing ensures robustness with respect to modeling errors. The proposed method is based on the potential field approach and is applied whether the object shape and dimensions are known or not, and when the calibration parameters of the camera are well or badly estimated. Finally, real-time experimental results using an eye-in-hand robotic system are presented and confirm the validity of our approach.

Since image-based servoing is a local control solution, it requires the definition of intermediate subgoals in the sensor space when the initial robot position is far from the desired one. This paper addresses the problem of generating and tracking realistic and optimal smooth trajectories of complex features in the image space. The model of the observed target and the internal camera parameters are assumed to be unknown. First a closed-form smooth collineation path (related to a reference plane) between given starts and end-points is obtained. This path is generated in order to correspond to an optimal camera path. The trajectories of the image features (corresponding to points belonging to or not belonging to the reference plane) are then derived and tracked using image based control.

This paper proposes a way to achieve positioning tasks by visual servoing, for any orientation of the camera, when the desired image of the observed object cannot be precisely described. The object is assumed to be planar and motionless but no knowledge about its shape or pose is required. To simplify the problem, first, we treat the case of a thread-like object and then we show how our approach can be generalized to an object with three particular points. The control law is based on the use of 2D visual servoing and on an estimation of a 3D parameter. Experimental results relative to objects of unknown shape are given to validate the approach. In addition, an algorithm to estimate the depth between the object and the camera is provided which leads to the dimensions of the object.

This paper introduces a 2D visual servoing technique for the station keeping of an unmanned underwater vehicle (UUV) with respect to planar targets on the sea bed. The underwater vehicle is subject to sea current disturbances which make it drift from its desired position. Feature points from unmarked objects are extracted and tracked with a sparse feature tracker developed in-house. The scene depth is estimated from a planar homography. To validate our approach, we emulate the dynamics of the surge and the sway degrees-of-freedom of an UUV model with a planar Cartesian robot in our water test tank. Successful station keeping experiments obtained with a range of sea current disturbances are presented.

We present a vision-based method to control the displacement of robot arm mounted on an underwater remotely operated vehicle (ROV). A closed-loop system based on an eye-to-hand visual servoing approach has been designed to achieve this task. We show that, using such an approach, the measuring of the manipulator motion with proprioceptive sensors is not required to precisely control the end-effector motion. To maintain the end effector in the field of view, the camera orientation is also controlled. The results presented show the validity of the approach.

Since image-based visual servoing is a local feedback control solution, it requires the definition of intermediate subgoals in the sensor space at the task planning level. We describe a general technique for specifying and tracking trajectories of an unknown object in the camera image space. First, physically valid C/sup 2/ image trajectories which correspond to quasi-optimal 3D camera trajectory (approaching as much as possible a straight line) are performed. Both mechanical (joint limits) and visibility constraints are taken into account at the task planning level. The good behavior of image-based control when desired and current camera positions are closed is then exploited to design an efficient control scheme. Real time experimental results using a camera mounted on the end effector of a 6 DOF robot confirm the validity of our approach.

We present a suitable object knowledge representation, based on a mixture of stochastic and set membership models. We consider that, for a large class of applications, an approximated representation of objects is sufficient to build a preliminary map of the scene. Our approximation mainly results in ellipsoidal calculus by means of a normal assumption for stochastic laws and ellipsoidal over or inner bounding for uniform laws. These approximations allow us to build an efficient estimation process integrating visual data online. Based on this estimation scheme, we perform online and optimal exploratory motions for the camera.

Since image-based servoing is a local control solution, it requires the definition of intermediate subgoals in the sensor space. This paper addresses the problem of generating realistic and optimal smooth trajectories of complex features in the image space. The model of the observed target and the internal camera parameters are assumed to be unknown. First, a closed-form smooth collineation path between given starts and endpoints is obtained. This path is generated in order to correspond to an optimal camera path. The trajectories of the image features are then derived.

We propose new redundancy-based solutions to avoid robot joint limits of a manipulator. We use a control scheme based on the task function approach. We first recall the classical gradient projection approach and then present a far more efficient method that relies on the iterative computation of motion that does not affect the task achievement and ensures the avoidance problem. We apply this new method in a visual servoing application. We demonstrate the validity of the approach on various real experiments as well as on the control of a virtual humanoid.

We describe in this paper potential problems that may appear in image-based visual servoing when the initial camera position is far away from its desired position. We show by concrete examples that local minima or a singularity of the image Jacobian can be reached during the servoing. We then recall recent results obtained to avoid these drawbacks. It consists in combining visual features obtained directly from the image, and position-based features. This approach, called 2 1/2 D visual servoing, also provides supplementary advantages with regard to the mean the features are combined with.

We propose in this paper new redundancy-based solutions to avoid robot joint limits of a manipulator. We use a control scheme based on the task function approach. We first recall the classical gradient projection approach and we then present a far more efficient method that relies on the iterative computation of motion that does not affect the task achievement and ensures the avoidance problem. We apply this new method in a visual servoing application and we demonstrate on various real experiments the validity of the approach.

The use of a camera in a robot control loop can be performed with two types of architecture: for eye-in-hand camera when it rigidly mounted on the robot end-effector; and for eye-to-hand camera when it observes the robot within its work space. These two schemes have technical differences and they can play very complementary parts. Obviously, the eye-in-hand one has a partial but precise sight of the scene whereas the eye-to-hand camera has a less precise but global sight of it. The motivation of our work is to take advantage of both, free-standing and robot-mounted sensors, in a cooperation scheme. The system presented performs two separate tasks: a positioning task to perform in the global image, and a tracking task to perform in the local image. For robustness considerations, the control law stability is proved and several cooperative schemes are studied and compared in experimental results.

In this paper, the classical visual servoing techniques have been extended to the use of several cameras observing different parts of an object. The multi-camera visual servoing has been designed as a part of the task function approach. The particular choice of the task function allows one to simplify the design of the control law and the stability analysis. A positioning task on a cumbersome object has been realized using 2D and 2 1/2 D visual servoings with two cameras, mounted on a manipulator robot, and observing two different parts of the object.

Vision feedback control loop techniques are efficient for a number of applications but they come up against difficulties when the initial and desired positions of the camera are distant. We propose a new approach to resolve these difficulties by planning trajectories in the image. Constraints such that the object remains in the camera field of view can be taken into account. Furthermore, using this process, current measurement always remain close to their desired value and a control by image based servoing ensures the robustness with respect to modeling errors. We apply our method when object dimension are known or not and/or when the calibration parameters of the camera are well or badly estimated. Finally, real time experimental results using a camera mounted on the end effector of a 6-DOF robot are presented.

We describe a method to achieve robotic positioning tasks by image-based visual servoing when the object being observed has a complex and unknown shape. We first focus on the computation of an analytical expression of the interaction matrix according to a polar description of the image contour of the object. Experimental results are presented to validate the proposed algorithm. In particular, the robustness of the control law is tested with respect to a coarse calibrated system, to an approximation of the depth of the object, and to partial occlusion.

Visual servoing based upon geometrical features such as image points coordinates is now well set on. Nevertheless, this approach has the drawback that it usually needs visual marks on the observed object to retrieve geometric features. The idea developed here is that these features can be retrieved by integrating dynamic ones, which can be estimated without any a priori knowledge of the scene. Thus, more realistic scenes can be used to achieve vision-based control such as positioning tasks. We show that an affine model is insufficient to ensure convergence and that a quadratic one is needed. Finally, results are presented related to the positioning with respect to a complex scene.

We present an original method for tracking, in an image sequence, complex objects which can be approximately modeled by a polyhedral shape. The approach relies on the estimation of the 2D object image motion along with the computation of the 3D object pose. The proposed method fulfills real-time constraints along with reliability and robustness requirements. Real tracking experiments and results concerning a visual servoing positioning task are presented.

We present an original method for tracking, in an image sequence, complex objects which can be modeled approximately by a polyhedral shape. The approach relies on the estimation of the object image motion as well as the computation of the object pose. The proposed method fulfills real-time constraints along with reliability and robustness requirements.

Deals with the 3D structure estimation and exploration of static scenes using active vision. Our method is based on the structure from controlled motion approach that constrains camera motions to obtain an optimal estimation of the 3D structure of a geometrical primitive. Since this approach involves gazing on the considered primitive, we have developed perceptual strategies able to perform a succession of robust estimations. This leads to a gaze planning strategy that mainly uses a representation of known and unknown areas as a basis for selecting viewpoints. This approach ensures a reconstruction as complete as possible of the scene.

We propose an approach to vision-based robot control, called 2 1/2 D visual servoing, which avoids the respective drawbacks of classical position-based and image-based visual servoing. Contrary to the position-based visual servoing, our scheme does not need any geometric three-dimensional model of the object. Furthermore and contrary to image-based visual servoing, our approach ensures the convergence of the control law in the whole task space. 2 1/2 D visual servoing is based on the estimation of the partial camera displacement from the current to the desired camera poses at each iteration of the control law. Visual features and data extracted from the partial displacement allow us to design a decoupled control law controlling the six camera DOFs. The robustness of our visual servoing scheme with respect to camera calibration errors is also analyzed: the necessary and sufficient conditions for local asymptotic stability are easily obtained. Then, due to the simple structure of the system, sufficient conditions for global asymptotic stability are established. Finally, experimental results with an eye-in-hand robotic system confirm the improvement in the stability and convergence domain of the 2 1/2 D visual servoing with respect to classical position-based and image-based visual servoing.

Efficient real-time robotic tasks using a monocular vision system were previously developed with simple objects (e.g. white points on a black background), within a visual servoing context. Due to recent developments, it is now possible to design real-time visual tasks exploiting motion information in the image, estimated by robust algorithms. This paper proposes such an approach to track complex objects, such as a pedestrian. It consists in integrating the measured 2D motion of the object to recover its 2D-position in the image. The principle of the tracking task is to control the camera pan and tilt such that the estimated center of the object appears at the center of the image. Real-time experimental results demonstrate the efficiency and the robustness of the method.

Visual servoing based upon geometrical features such as image point coordinates is now well studied. However, this approach has a drawback that it usually needs visual markings on the observed object to retrieve geometric features. The idea developed here is that these features can be retrieved by integrating dynamic ones, which can be estimated without any a priori knowledge of the scene. With this approach, more realistic objects can be used to achieve vision-based control such as tracking and fixation tasks. We detail the control laws concerning these two tasks, first using integration of speed in the image and then by direct regulation of these dynamic parameters. Results are finally presented and comparisons are made between the two types of control methods.

In this paper we propose a new vision-based robot control approach halfway between the classical position-based and image-based visual servoings. It allows to avoid their respective disadvantages. The homography between some planar feature points extracted from two images (corresponding to the current and desired camera poses) is computed at each iteration. Then, an approximate partial-pose, where the translational term is known only up to a scale factor, is deduced, from which can be designed a closed-loop control law controlling the six camera DOF. Contrarily to the position-based visual servoing, our scheme does not need any geometric 3D model of the object. Furthermore and contrarily to the image-based visual servoing, our approach ensures the convergence of the control law in all the task space.

In this paper, the robustness of a new visual servoing scheme with respect to camera calibration errors is analyzed. This scheme, called 2D 1/2 visual servoing, is based on the estimation of the partial camera displacement from the current to the desired camera poses at each iteration of the control law. Visual features and data extracted from the partial displacement allow as to design a decoupled control law controlling the six camera DOF. The necessary and sufficient conditions for local asymptotic stability in presence of camera calibration errors are easily obtained. Then, thanks to the simple structure of the system, sufficient conditions for global asymptotic stability are proposed. Finally, experimental results show the validity of our approach and its robustness not only with respect camera calibration errors but also to robot calibration errors.

This paper deals with the 3D structure estimation and exploration of a scene using active vision. We have used the structure from controlled motion approach to obtain a precise and robust estimation of the 3D structure of geometrical primitives. Since it involves gazing successively on the considered primitives, we have developed perceptual strategies able to perform a succession of robust estimations without any assumption on the number and on the localization of the different objects. An exploration process centered on current visual features and on the structure of the previously studied primitives is presented. This leads to a gaze planning strategy that mainly uses a representation of known and unknown areas as a basis for selecting viewpoints. The proposed strategy ensures the completeness of the reconstruction.

In this paper, we propose a visual servoing of the orientation of a camera mounted on a robot, using human eye movements. The study aims at the remote survey of sensitive sites or the intervention in contaminated nuclear sites. Visual data stemmed from an eye-tracker, providing the direction of the look of an operator, is generally noisy and sometimes inconsistent. Filtering, smoothing and stabilizing these eye tracker data are thus necessary in order to move the camera in the direction specified by the operator's eye. The visual servoing control scheme, as well as experimental results, are also presented.

Visual servoing is generally based upon geometrical features. Recent developments were made in the way of a generalization of this approach to dynamic features. The idea is that velocity in the image can be measured without the constraint of having an a priori knowledge of the scene. In this paper, a control law to position a camera mounted on the end effector of a robot, in such a way the image plane becomes parallel to a planar object, is presented. A dynamic visual servoing approach is used by defining a control loop upon the second order spatial derivatives of the optical flow. A fixating task, which guarantees the object stays in the camera field of view is joined to the first one. Then, results obtained on a 6-DOF robot are laid out for the two studied tasks.

This paper presents an experiment with the synchronous approach to reactive systems programming, and particularly the Signal language, applied to a significant problem in robot vision: active visual reconstruction. This application consists of the specification of a system dealing with various domains such as robot control, computer vision and transitions between different modes of control. It illustrates the adequacy in such domains of Signal, a data flow programming language and environment. The programming environment features tools for formal specification, analysis, consistency checking and code generation. Signal and its language-level extension for task preemption SignalGTi are used at the different levels of the application: data-flow function for the camera motion control (visual servoing), reconstruction method (in parallel to visual servoing, involving the dynamical processes), and reconstruction of complex scenes (with transitions between several robotics tasks). The combination of these levels constitutes a hybrid behavior with (sampled) continuous control and discrete transitions. These techniques are validated experimentally by an implementation on a robotic cell. © 1997 by John Wiley & Sons, Ltd.

This paper deals with the 3D structure estimation and exploration of a scene using active vision. Our method is based on the structure from controlled motion approach which consists in constraining the camera motion in order to obtain a precise and robust estimation of the 3D structure of a geometrical primitive. Since this approach involves to gaze on the considered primitive, we present a method for connecting up many estimations in order to recover the complete spatial structure of scenes composed of cylinders and segments. We have developed perceptual strategies able to perform a succession of robust estimations without any assumption on the number and on the localization of the different objects. Furthermore, the proposed strategy ensures the completeness of the reconstruction. An exploration process centered on current visual features and on the structure of the previously studied primitives is presented. This leads to a gaze planning strategy that mainly uses a representation of known and unknown areas as a basis for selecting viewpoints. Finally, experiments carried out on a robotic cell have proved the validity of our approach.

We propose in this paper solutions to avoid the joint limits and kinematic singularities in visual servoing. We use a control scheme based on the task function approach. It combines the regulation of the selected vision-based task with the minimization of a secondary cost function, which reflects the manipulability of the robot in the vicinity of joint limits and singularities.

We propose in this paper solutions to avoid robot joint limits and kinematic singularities in visual servoing. We use a control scheme based on the task function approach. It combines the regulation of the selected vision based task with the minimization of a secondary cost function, which reflects the manipulability of the robot in the vicinity of internal or external singularities. Several methods are proposed to avoid joint limits and a comparison between them is presented. We have demonstrated on various experiments the validity of our approach.

This paper deals with the recovery of 3D information using a single mobile camera in the context of active vision. First, we propose a general revisited formulation of the structure-from-known-motion issue. Within the same formalism, we handle various kinds of 3D geometrical primitives such as points, lines, cylinders, spheres, etc. We also aim at minimizing effects of the different measurement errors which are involved in such a process. More precisely, we mathematically determine optimal camera configurations and motions which lead to a robust and accurate estimation of the 3D structure parameters. We apply the visual servoing approach to perform these camera motions using a control law in closed-loop with respect to visual data. Real-time experiments dealing with 3D structure estimation of points and cylinders are reported. They demonstrate that this active vision strategy can very significantly improve the estimation accuracy.

The work presented in this paper belongs to the realm of robotics and computer vision. The problem we seek to solve is the accomplishment of robotics tasks using visual features provided by a special sensor, mounted on a robot end effector. This sensor consists of two laser stripes fixed rigidly to a camera, projecting planar light on the scene. First, we briefly describe the classical visual servoing approach. We then generalize this approach to the case of our special sensor by considering its interaction with respect to a sphere. This interaction permits us to establish a kinematics relation between the sensor and the scene. Finally, both in simulation and in our experimental cell, the results are presented. They concern the positioning task with respect to a sphere, and show the robustness and the stability of the control scheme.

This paper describes a real time visual target tracking using the generalized likelihood ratio (GLR) algorithm. The authors' first introduce the visual servoing approach and the application of the task function concept to vision-based tasks. Then, the authors present a complete control scheme which explicitly enables a moving object to be pursued. In order to make the tracking errors as low as possible, the authors use the GLR test, an algorithm able to detect, estimate and compensate abrupt jumps in target motion. Finally, real-time experimental results using a camera mounted on the end effector of a six-DOF robot are presented.

This paper describes experiments in estimation and control of the fixation point with a 10 degree of freedom head-neck-body system. A system has been constructed which uses standard kinematic techniques to estimate the fixation point. Two approaches have been investigated to control the fixation point: 3D kinematic estimation and 2D visual servoing. Kinematics is shown to be efficient provided that there is a sufficiently precise model of the kinematic chain. In particular, large errors in the kinematic model can cause the system to oscillate. In contrast, visual servoing is extremely robust with respect to errors in the kinematic model, but require a much larger (factor of 10) number of cycles to converge. The two approaches are found to be complementary. Kinematics can be used to perform a ballistic saccade to the fixation point, while visual servoing serves to correct small errors with a "micro-saccade". This approach mimics saccadic control found in the human visual system.

In this paper, we apply the synchronous approach to real time active visual reconstruction. It illustrates the adequateness of SIGNAL, a synchronous data flow programming language, for the specification of a system dealing with various domains such as robot control, computer vision and the programming of hierarchical parallel automata. More precisely, our application consists in the 3D structure estimation of a set of geometrical primitives using a camera mounted on the end effector of a six dof robot. At the level of camera motion control, the visual servoing approach is specified and implemented in SIGNAL as a function from sensor inputs to control outputs. The 3D reconstruction method is based on the "structure from controlled motion" approach. Its specification is made in parallel to visual servoing. We also present a perception strategy for connecting up several estimations, using time intervals and hierarchical structures for task preemption in SIGNAL. The integration of these techniques is validated experimentally by their implementation on a robotic cell.

This paper deals with the recovery of 3D information using a single mobile camera in the context of active vision. We propose a general revisited formulation of the structure-from-motion issue, and we determine adequate camera configurations and motions which lead to a robust and accurate estimation of the 3D structure parameters. We apply the visual servoing approach to perform these camera motions. Real-time experiments dealing with the 3D structure estimation of points and cylinders are reported, and demonstrate that this active vision strategy can very significantly improve the estimation accuracy.>

A method for the reconstruction and localization of geometrical primitives using active dynamic vision is presented. The approach is based on the use of the interaction matrix related to the visual data describing a primitive. Next, active vision is considered by computing adequate camera motions with a control law in closed-loop with respect to visual data. Simulation results on the localization of a sphere are presented and show that active vision can to a large extent improve the accuracy of the structure estimation.>

Vision-based control in robotics based on considering a vision system as a specific sensor dedicated to a task and included in a control servo loop is described. Once the necessary modeling stage is performed, the framework becomes one of automatic control, and stability and robustness questions arise. State-of-the-art visual servoing is reviewed, and the basic concepts for modeling the concerned interactions are given. The interaction screw is thus defined in a general way, and the application to images follows. Starting from the concept of task function, the general framework of the control is described, and stability results are recalled. The concept of the hybrid task is presented and then applied to visual sensors. Simulation and experimental results are presented, and guidelines for future work are drawn in the conclusion.>

An application is described of the so-called visual servoing approach to robot positioning with respect to an object and to target tracking. After briefly discussing how the task function approach can be applied to tasks which include the use of visual features, the authors give a simplified control expression which explicitly takes into account the case of moving objects. Estimating the target velocity while performing the tracking control leads to a kind of adaptive control scheme. The authors then consider the specific case of a square target and derive all the components of the control scheme. They present some experimental results obtained at the video rate in an experimental system composed of a camera mounted on the end effector of a six-DOF robot.>