Humans tend to build environments with structure, which consists of mainly planar surfaces. From the intersection of planar surfaces arise straight lines. Lines have more degrees of freedom than points. Thus, line-based structure-from-motion (SfM) provides more information about the environment. In this article, we present solutions for SfM using lines, namely, incremental SfM. These approaches consist of designing state observers for a camera’s dynamical visual system looking at a 3-D line. We start by presenting a model that uses spherical coordinates for representing the line’s moment vector. We show that this parameterization has singularities, and, therefore, we introduce a more suitable model that considers the line’s moment and shortest viewing ray. Concerning the observers, we present two different methodologies. The first uses a memory-less state-of-the-art framework for dynamic visual systems. Since the previous states of the robotic agent are accessible—while performing the 3-D mapping of the environment—the second approach aims at exploiting the use of memory to improve the estimation accuracy and convergence speed. The two models and the two observers are evaluated in simulation and real data, where mobile and manipulator robots are used.

The planning of robotic manipulation and grasping tasks depends on the reconstruction of the 3D object’s shape. Most of the existing 3D object reconstruction methods are based on visual sensing that are limited due to the lack of the object’s occluded side information. The goal of this paper is to overcome these limitations and improve the 3D objects’ reconstruction by adding the tactile sensing to the visual data. In this paper, a novel multi-modal (visual and tactile) semi-supervised generative model is presented to reconstruct the complete 3D object’s shape using a single arbitrary depth-view and a single dexterous-hand’s touch. The presented approach takes the strength of the autoencoder and generative networks to provide an end-to-end trainable model with high generalization ability. The 3D voxel grids of the depth and tactile data are the only requirements of the proposed model to predict a high resolution voxel grids of 643for the incomplete shape. This research generates its tactile dataset based on the kinematic model of the shadow dexterous hand. The developed dataset has aligned depth, tactile and ground truth voxel grids of different resolutions (403, 643and 1283) from different camera views. Experimental results show that the proposed multi-modal model outperforms other state-of-the-art methods.

Using spherical moments to estimate the rotation of camera is an effective method. However, after the camera rotates, the scene in the captured image will change, which will cause a certain error. This paper proposes a method to reduce this error by removing the non-common part between the two images during rotation. Through multiple iterations, the error caused by scene changing will become very small. The experimental results using manual generated image are provided to show the efficiency of this proposed method.

This paper presents a novel permanent magnets based actuator capable of injecting and 2D wireless control the motion of an untethered microrobot. The actuator is obtained with a simple, but an effective arrangement of four permanent magnets. Its novelty is that it creates local maxima of the magnetic field magnitude in a planar workspace. This results in a convergence point for magnetic microrobots that are in its influence zone. Trapping microrobots in the local maxima makes their open-loop guidance possible, even in presence of reasonable perturbations. Actually, open-loop control is the only way to achieve some drug delivery when there are no sensors to provide us a feedback on the microrobot's positions. With the proposed actuator, the workspace is reduced, the movement is simplified and the actuator orientation is not needed anymore. Experimental results are provided in the paper to show the soundness of the proposed actuator design.

This work presents a method to guide microrobots by positioning a magnetic actuator using hybrid vision system. The used actuator mounted at a robot end-effector creates local maxima of the magnetic field magnitude, which results in an attractive point for microrobots in its influence zone. The hybrid vision system serves to control the actuator position through the robotic platform to make the trapped microrobot undergo a planned trajectory. In first validation results, the particle driving is achieved in open loop with no adjustment with respect to the planned trajectory. Such scheme can be used in the case where the microrobot position can not be measured in real time. The second validation results deal with the case of microrobot positioning by visual servoing. This can be used in the case where recovering microrobot position is possible, for example in eye treatment.

This paper presents new schemes to estimate 3D rotation from spherical images. Unlike existing approaches, spherical moment properties are exploited to obtain a closed form solution without iteratively mimimizing a cost function. Actually, three methods using spherical moments are proposed: two of them can be classified as dense approaches, while the third one is hybrid combining geometrical features with dense ones. Experimental results using both synthetic images and acquired images using catadioptric cameras with different scenarios show the effectiveness of our approach.

A reliable estimation of 3D parameters is a must for several applications like planning and control, in which is included Image-Based Visual Servoing. This control scheme depends directly on 3D parameters, e.g. depth of points, and/or depth and direction of 3D straight lines. Recently, a framework for Active Structure-from-Motion was proposed, addressing the former feature type. However, straight lines were not addressed. These are 1D objects, which allow for more robust detection, and tracking. In this work, the problem of Active Structure-from-Motion for 3D straight lines is addressed. An explicit representation of these features is presented, and a change of variables is proposed. The latter allows the dynamics of the line to respect the conditions for observability of the framework. A control law is used with the purpose of keeping the control effort reasonable, while achieving a desired convergence rate. The approach is validated first in simulation for a single line, and second using a real robot setup. The latter set of experiments are conducted first for a single line, and then for three lines.

In this paper, visual servoing based on photometric moments is advocated. A direct approach is chosen by which the extraction of geometric primitives, visual tracking and image matching steps of a conventional visual servoing pipeline can be bypassed. A vital challenge in photometric methods is the change in the image resulting from the appearance and disappearance of portions of the scene from the camera field of view during the servo. To tackle this issue, a general model for the photometric moments enhanced with spatial weighting is proposed. The interaction matrix for these spatially weighted photometric moments is derived in an analytical form. The correctness of the modeling, effectiveness of the proposed strategy in handling the exogenous regions, and improved convergence domain are demonstrated with a combination of simulation and the experimental results.

The magnetic particles actuated by an external magnetic field present a promising solution as drug carrier at the inner ear (cochlea). This technique allows non-invasive drug delivery with minimal hydraulic force and very high precision. In this work, magnetic prototyping platform for intra cochlear drug delivery procedure is proposed. This platform is composed by magnetic actuator mounted on a robot end-effector and an hybrid vision system. The latter consists of a camera to control the position of magnetic actuator, two microscopes to recover the position of the magnetic particles and the 3D structure of the cochlea. The hybrid vision system allows real-time tracking of magnetic particles inside the artificial cochlea. The accuracy of the drug delivery strategy depends strongly on the calibration of the vision system. In this paper a method to calibrate the hybrid vision system is proposed and validated with experimental results.

Conventional active incremental structure from motion (ISfM) schemes require a precise knowledge of the linear and angular velocities of the vision system to compute the three-dimensional structure of the observed scene. Furthermore, they are generally coupled and nonlinear, which makes the reconstruction inaccurate. In this paper, we present a novel active ISfM scheme to overcome these difficulties. It relies on two appropriate transformations: the first one allows efficient decoupling of the nonlinear model, whereas the second one transforms the decoupled model into the Brunovsky's linear form. We then show that a Luenberger's observer can be designed to estimate the unavailable states, making our scheme globally asymptotically convergent and robust to noise on measurements. We present simulated and experimental results to illustrate the performance of the described methodology.

Earth mover's distance is one of the most effective metric for comparing histograms in various image retrieval applications. The main drawback is its computational complexity which hinders its usage in various comparison tasks. We propose fast earth mover's distance computation by providing better initialization to the transportation simplex algorithm. The new approach enables faster EMD computation in Visual Memory (VM) compared to the state of the art methods. The new proposed strategy computes earth mover distance without compromising its accuracy.

Visual servoing based on photometric data is of great interest since it does not necessitate any image processing or visual tracking steps. A vital issue in such methods is the change in the image resulting from the appearance and disappearance of portions of the scene from the camera field-of-view during the visual servo. In this paper, we propose a spatial weighting scheme to counter this problem. A general model for photometric moments that allows for inclusion of spatial weights is presented. A custom weighting function is then chosen and specific properties of its derivative are exploited in order to develop the interaction matrix in analytical form. The resultant effects on the invariance properties are discussed. Experiments have been performed to validate the visual servoing with the proposed weighting scheme on a 4 dof robot arm.

Incremental Structure from Motion (SfM) algorithms require, in general, precise knowledge of the camera linear and angular velocities in the camera frame for estimating the 3D structure of the scene. Since an accurate measurement of the camera own motion may be a non-trivial task in several robotics applications (for instance when the camera is onboard a UAV), we propose in this paper an active SfM scheme fully independent from the camera angular velocity. This is achieved by considering, as visual features, some rotational invariants obtained from the projection of the perceived 3D points onto a virtual unitary sphere (unified camera model). This feature set is then exploited for designing a rotation-free active SfM algorithm able to optimize online the direction of the camera linear velocity for improving the convergence of the structure estimation task. As case study, we apply our framework to the depth estimation of a set of 3D points and discuss several simulations and experimental results for illustrating the approach.

Over the past decade, image moments have been exploited in several visual servoing schemes for their ability to represent object regions, objects defined by contours or a set of discrete points. Moments have also been useful to achieve control decoupling properties and to choose a minimal number of features to control the whole degrees of freedom (DOFs) of a camera. However, the choice of moment-based features to control the rotational motions around the x-axis and y-axis simultaneously with the translational motions along the same axis remains a key issue. In this paper, we introduce new visual features computed from low-order “shifted moments invariant.” Importantly, they allow us 1) to define a unique combination of visual features to control the whole six DOFs of an eye-in-hand camera independently from the object shape and (2) to significantly enlarge the convergence domain of the closed-loop system.

In this paper, we introduce the concept of tunable visual features for moments based visual servoing schemes. The main contribution of this work is the introduction of tunable shift points along with some effective methods to tune them. We propose two different metrics: the first metric ensures optimal response of the control to errors in the image space and the second metric ensures orthogonality between the interaction matrix components (vectors) related to the control of x and y rotational motions. With the proposed method, it is possible to design moment invariants-based visual features whose interaction matrix is always non-singular for any desired pose (parallel or non-parallel). Thus, this work makes a significant contribution to the difficult problem of controlling the rotational motions around the x and y axes, when all the 6dof are involved. Two case studies are presented to demonstrate the validity of the proposed ideas. Results from each case are then used to design a moment invariants-based visual feature. This visual feature is used for visual servoing with a symmetrical object using binary moments and a free-form planar target using photometric moments.

This paper deals with pose estimation using an iterative scheme. We show that using adequate visual information, pose estimation can be performed iteratively with only three independent unknowns, which are the translation parameters. Specifically, an invariant to rotational motion is used to estimate the camera position. In addition, an adequate transformation is applied to the proposed invariant to decrease the nonlinearities between the variations in image space and 3-D space. Once the camera position is estimated, we show that the rotation can be estimated efficiently using two different direct methods. The proposed approach is compared against two other methods from the literature. The results show that using our method, pose tracking in image sequences and the convergence rate for randomly generated poses are improved.

In image-based visual servoing, the choice of visual features has a strong influence on the performance of the control system. In the last decade, image-moments have been exploited in several visual servoing schemes for their ability to represent object region, object defined by contours or a set of discrete points. Despite the many recent advances, the choice of moment-based features to control the most critical Degrees Of Freedom (DOF), i.e those used to control the rotational motions around the x-axis and y-axis of the camera remains a key issue. In this paper, a new feature formula to control these critical DOF that does not depend on the object shape is proposed. The new features are computed from shifted moments and selected in order to provide nice invariant properties. Additionally, low order shifted moments can be exploited to minimize the impact of measurement noise on the control performances.

This paper deals with pose estimation using an iterative scheme. We show that using adequate visual information, pose estimation can be performed in a decoupling the estimation of translation and rotation. More precisely, we show that pose estimation can be achieved iteratively as a function of only three independent unknowns, which are the translation parameters. An invariant to rotational motion is used to estimate the camera position. Once the camera position is estimated, we show that the rotation can be estimated efficiently using a direct method. The proposed approach is compared against two classical methods from the literature. The results show that using our method, pose tracking in image sequences and the convergence rate for randomly generated poses are improved.

Catadioptric cameras combine conventional cameras and mirrors to create omnidirectional sensors providing 360° panoramic views of a scene. Modeling such cameras has been subject of significant research interest in the computer vision community leading to a deeper understanding of the image properties and also to different models for different types of configurations. Visual servoing applications using catadioptric cameras have essentially been using central cameras and the corresponding unified projection model. So far only in very few cases more general models have been used. In this paper we address the problem of visual servoing using the so-called the radial model. The radial model can be applied to many camera configurations and in particular to non-central catadioptric systems with mirror shapes that are symmetric around the optical axis. In this case we show that the radial model can be used with a non-central catadioptric camera to allow effective image-based visual servoing (IBVS) of a mobile robot. Using this model, which is valid for a large set of catadioptric cameras, new visual features are proposed to control the degrees of freedom of a mobile robot moving on a plane. Several simulation results are provided to validate the effectiveness of such features.

This paper is concerned with the use of a spherical-projection model for visual servoing from three points. We propose a new set of six features to control a 6-degree-of-freedom (DOF) robotic system with good decoupling properties. The first part of the set consists of three invariants to camera rotations. These invariants are built using the Cartesian distances between the spherical projections of the three points. The second part of the set corresponds to the angle-axis representation of a rotation matrix measured from the image of two points. Regarding the theoretical comparison with the classical perspective coordinates of points, the new set does not present more singularities. In addition, using the new set inside its nonsingular domain, a classical control law is proven to be optimal for pure rotational motions. The theoretical results and the robustness to points range errors of the new control scheme are validated through simulations and experiments on a 6-DOF robot arm.

This paper deals with visual servoing from three points. Using the geometric properties of the spherical projection of points, a new decoupled set of six visual features is proposed. The main originality lies in the use of the distances between spherical projection of points to define three features that are invariant to camera rotations. The three other features present a linear link with respect to camera rotations. In comparison with the classical perspective coordinates of points, the new decoupled set does not present more singularities. In addition, using the new set in its non-singular domain, a classical control law is proven to be ideal for rotational motions. These theoretical results as well as the robustness to errors of the new decoupled control scheme are illustrated through simulation results.

This paper proposes a generic decoupled image-based control scheme for cameras obeying the unified projection model. The scheme is based on the spherical projection model. Invariants to rotational motion are computed from this projection and used to control the translational degrees of freedom (DOFs). Importantly, we form invariants that decrease the sensitivity of the interaction matrix to object-depth variation. Finally, the proposed results are validated with experiments using a classical perspective camera as well as a fisheye camera mounted on a 6-DOF robotic platform.

In this paper a generic decoupled imaged-based control scheme for calibrated cameras obeying the unified projection model is proposed. The proposed decoupled scheme is based on the surface of object projections onto the unit sphere. Such features are invariant to rotational motions. This allows the control of translational motion independently from the rotational motion. Finally, the proposed results are validated with experiments using a classical perspective camera as well as a fisheye camera mounted on a 6 dofs robot platform.

This paper deals with the visual control of the Gough-Stewart platform using a central catadioptric camera observing the platform's legs. This allows a large field of view to be obtained and avoids the occlusion problems observed when a classical perspective camera is used. An automatic and simple method to detect the projections of the leg in the image is also proposed. The control scheme presented here is shown to encompass the classical perspective camera case, as well as catadioptric ones. Finally, experimental results comparing two kinds of visual features (leg directions and leg edges) are described.

In this paper a new decoupled imaged-based control scheme is proposed from projection onto a unit sphere. This control scheme is based on moment invariants to 3D rotational motion. This allows the control of translational motion independently of the rotational one. First, the analytical form of the interaction matrix related to the spherical moments is derived. It is based on the projection of a set of points onto a unit sphere. From the spherical moment, six features are presented to control the full 6 degrees of freedom. Finally, the results are validated through realistic simulation results.

This paper deals with the efficient second order minimization (ESM) and the image-based visual servoing schemes. In other word, it deals with the minimization based on the pseudo-inverses of the mean of the Jacobians or on the mean of Jacobian Pseudo-inverses. Chronologically, it has been noted by Tahri and Chaumette (2003) that the (ESM) improves generally the system behavior compared to the case where only the simple Jacobian Pseudo- inverses is used. Subsequently, a mathematical explanation has been given by Malis (2004). In this paper, the proofs given by Malis are considered to deal with their validity. It will be shown that there is a limitation to the the validity of this method and some precautions should be taken, for adequate application of it. In other words, we will show that the use of ESM does not necessary ensures a better system behavior, especially in the cases where large rotational motions are considered.

This work deals with the control by vision of the Gough-Stewart platform. For that, a central catadioptric camera is used to observe the platform legs. This allows to obtain a large field of view, and then avoids the occlusion problems observed when a classical perspective camera is used. The leg projections onto the catadioptric plane are used to determine their orientation in the camera frame. Finally, the computed orientations will be used in a visual servoing scheme of the platform effector.

Pose estimation has many applications in robotics, such as robot localization using a vision sensor, or position-based visual servoing. In this paper, a simple but an efficient pose estimation method from minimal point pairs is given. The method we propose is based on exhaustive optimization of only two most critical parameters. It allows to solve the local minima problem. The both model based and free model pose estimation are treated. Finally, simulation results are given to validate the proposed method

The paper contributes to presenting both an accurate and robust shape tracking algorithm and a novel visual servoing method. Two steps are involved in the tracking algorithm. Firstly the object shape is assumed to vary under an affine model, and the edge detection is performed along the normal lines to the contour. As a result it is possible to use a Kalman filter to perform efficient tracking. The second step concerns image matching based on perspective model, which is achieved iteratively by searching locally along the normal lines also. As to visual servoing we propose to control the translations of the robot with the normalized zeroth and first order image moments, and to control the orientation with rotation axis and angle extracted from a Homography matrix. Two experiments demonstrate that the tracking algorithm is accurate and robust enough to be used in visual servoing, and the novel visual servoing method is superior to traditional ones.

Moments are generic (and usually intuitive) descriptors that can be computed from several kinds of objects defined either from closed contours or from a set of points. In this paper, image moments are used in two new methods for the pose estimation of a planar object observed through full perspective model. The first method is based on an iterative optimization scheme formulated as virtual visual servoing, while the second is based on an exhaustive but efficient optimization scheme of the two most critical parameters. It allows to avoid local minima. We finally present some experimental results to validate the theoretical developments presented in this paper.

Moments are generic (and usually intuitive) descriptors that can be computed from several kinds of objects, defined either from closed contours or from a set of points. In this paper, we present improvements in image-based visual servo using image moments. First, the analytical form of the interaction matrix related to the moments computed from a set of coplanar points is derived, and we show that it is different from the form obtained previously, using coplanar closed contours. Six visual features are selected to design a decoupled control scheme when the object is parallel to the image plane. This nice property is then generalized to the case where the desired object position is not parallel to the image plane. Finally, experimental results are presented to illustrate the validity of our approach and its robustness, with respect to modeling errors.

Moments are generic and (usually) intuitive descriptors that can be computed from several kinds of objects defined either from closed contours (continuous object) or a set of points (discrete object). In this paper, we propose to use moments to design a decoupled image-based visual servo. The analytical form of the interaction matrix related to the moments computed for a discrete object is derived, and we show that it is different of the form obtained for a continuous object. Six visual features are selected to design a decoupled control scheme when the object is parallel to the image plane. This nice property is then generalized to the case where the desired object position is not parallel to the image plane. Finally, experimental results are presented to illustrate the validity of our approach and its robustness with respect to modeling errors.

In this paper, we present how moment invariants can be used to design a decoupled 2D visual servoing scheme and to minimise the nonlinearity of the interaction matrix related to the selected visual features. Experimental results using a 6 DOF eye-in-hand system to position a camera parallel to planar objects of complex shape are presented to demonstrate the efficiency of the proposed method.