We present a new quasi-affine reconstruction of a scene and its application to camera self-calibration. We refer to this reconstruction as QUARCH (QUasi-Affine Reconstruction with respect to Camera centers and the Hodographs of horopters). A QUARCH can be obtained by solving a semidefinite programming problem when, (i) the images have been captured by a moving camera with constant intrinsic parameters, and (ii) a vague knowledge of the relative orientation (under or over 120 degrees) between camera pairs is available. The resulting reconstruction comes close enough to an affine one allowing thus an easy upgrade of the QUARCH to its affine and metric counterparts. We also present a constrained Levenberg-Marquardt method for nonlinear optimization subject to Linear Matrix Inequality (LMI) constraints so as to ensure that the QUARCH LMIs are satisfied during optimization. Experiments with synthetic and real data show the benefits of QUARCH in reliably obtaining a metric reconstruction.

This paper addresses the problem of estimating the Euclidean transformation relating two rigidly attached reference frames from motion measurements: a problem that is commonly referred to as the Hand-Eye calibration. The motion measurements are often affected by synchronization issues and hardware malfunctions. When using pose sensors, such as Electromagnetic or Inertial Measurement Units, the measurements are often less reliable than those obtained by a traditional robot-link and camera pair. Corrupt measurements, whether due to disturbances, large synchronization mismatches or malfunctions, are considered outliers that ought to be filtered out of the estimation process. While this may be achieved non-deterministically by using Random Sample Consensus, we propose a deterministic, robust and accurate method for solving the Hand-Eye calibration problem despite the presence of large amounts of outliers and high levels of measurement noise. The proposed method is based on a reformulation of this estimation problem as a rank-constrained semi-definite programming problem allowing for robustness to be enforced via an iteratively re-weighted optimization approach.

This paper addresses the problem of estimating the geometric transformation relating two distinct visual modalities (e.g. an image and a map, or a projective structure and a Euclidean 3D model) while relying only on semantic cues, such as semantically segmented regions or object bounding boxes. The proposed approach differs from the traditional feature-to-feature correspondence reasoning: starting from semantic regions on one side, we seek their possible corresponding regions on the other, thus constraining the sought geometric transformation. This entails a simultaneous search for the transformation and for the region-to-region correspondences. This paper is the first to derive the conditions that must be satisfied for a convex region, defined by control points, to be transformed inside an ellipsoid. These conditions are formulated as Linear Matrix Inequalities and used within a Branch-and-Prune search to obtain the globally optimal transformation. We tested our approach, under mild initial bound conditions, on two challenging registration problems for aligning: (i) a semantically segmented image and a map via a 2D homography; (ii) a projective 3D structure and its Euclidean counterpart.

This paper investigates the problem of registering a scanned scene, represented by 3D Euclidean point coordinates, and two or more uncalibrated cameras. An unknown subset of the scanned points have their image projections detected and matched across images. The proposed approach assumes the cameras only known in some arbitrary projective frame and no calibration or autocalibration is required. The devised solution is based on a Linear Matrix Inequality (LMI) framework that allows simultaneously estimating the projective transformation relating the cameras to the scene and establishing 2D-3D correspondences without triangulating image points. The proposed LMI framework allows both deriving triangulation-free LMI cheirality conditions and establishing putative correspondences between 3D volumes (boxes) and 2D pixel coordinates. Two registration algorithms, one exploiting the scene's structure and the other concerned with robustness, are presented. Both algorithms employ the Branch-and-Prune paradigm and guarantee convergence to a global solution under mild initial bound conditions. The results of our experiments are presented and compared against other approaches.

This paper deals with the problem of registering a known structured 3D scene and its metric Structure-from-Motion (SfM) counterpart. The proposed work relies on a prior plane segmentation of the 3D scene and aligns the data obtained from both modalities by solving the point-to-plane assignment problem. An inliers-maximization approach within a Branch-and-Bound (BnB) search scheme is adopted. For the first time in this paper, a Sum-of-Squares optimization theory framework is employed for identifying point-to-plane mismatches (i.e. outliers) with certainty. This allows us to iteratively build potential inliers sets and converge to the solution satisfied by the largest number of point-to-plane assignments. Furthermore, our approach is boosted by new plane visibility conditions which are also introduced in this paper. Using this framework, we solve the registration problem in two cases: (i) a set of putative point-to-plane correspondences (with possibly overwhelmingly many outliers) is given as input and (ii) no initial correspondences are given. In both cases, our approach yields outstanding results in terms of robustness and optimality.

We present a new globally optimal algorithm for self-calibrating a moving camera with constant parameters. Our method aims at estimating the Dual Absolute Quadric (DAQ) under the rank-3 and, optionally, camera centers chirality constraints. We employ the Branch-and-Prune paradigm and explore the space of only 5 parameters. Pruning in our method relies on solving Linear Matrix Inequality (LMI) feasibility and Generalized Eigenvalue (GEV) problems that solely depend upon the entries of the DAQ. These LMI and GEV problems are used to rule out branches in the search tree in which a quadric not satisfying the rank and chirality conditions on camera centers is guaranteed not to exist. The chirality LMI conditions are obtained by relying on the mild assumption that the camera undergoes a rotation of no more than 90 between consecutive views. Furthermore, our method does not rely on calculating bounds on any particular cost function and hence can virtually optimize any objective while achieving global optimality in a very competitive running-time.

In this paper we propose a robust and direct 2D-to-3D registration method for localizing 2D cameras in a known 3D environment. Although the 3D environment is known, localizing the cameras remains a challenging problem that is particularly undermined by the unknown 2D-3D correspondences, outliers, scale ambiguities and occlusions. Once the cameras are localized, the Structure-from-Motion reconstruction obtained from image correspondences is refined by means of a constrained nonlinear optimization that benefits from the knowledge of the scene. We also propose a common optimization framework for both localization and refinement steps in which projection errors in one view are minimized while preserving the existing relationships between images. The problem of occlusion and that of missing scene parts are handled by employing a scale histogram while the effect of data inaccuracies is minimized using an M-estimator-based technique.

Accurate estimation of camera motion is very important for many robotics applications involving SfM and visual SLAM. Such accuracy is attempted by refining the estimated motion through nonlinear optimization. As many modern robots are equipped with both 2D and 3D cameras, it is both highly desirable and challenging to exploit data acquired from both modalities to achieve a better localization. Existing refinement methods, such as Bundle adjustment and loop closing, may be employed only when precise 2D-to-3D correspondences across frames are available. In this paper, we propose a framework for robot localization that benefits from both 2D and 3D information without requiring such accurate correspondences to be established. This is carried out through a 2D-3D based initial motion estimation followed by a constrained nonlinear optimization for motion refinement. The initial motion estimation finds the best possible 2D-to-3D correspondences and localizes the cameras with respect the 3D scene. The refinement step minimizes the projection errors of 3D points while preserving the existing relationships between images. The problems of occlusion and that of missing scene parts are handled by comparing the image-based reconstruction and 3D sensor measurements. The effect of data inaccuracies is minimized using an M-estimator based technique. Our experiments have demonstrated that the proposed framework allows to obtain a good initial motion estimate and a significant improvement through refinement.

This paper presents a simple, yet effective, method to recover the affine structure of a scene from a (stereo) pair of stationary zooming cameras. The proposed method solely relies on point correspondences across images and no knowledge about the scene whatsoever is required. Our method exploits implicit properties of the projective camera matrices of zooming cameras and allows to estimate the affine structure of a scene by solving a linear system of equations. The 3D reconstruction results obtained by using our method, on both real and simulated data, have remarkably validated its feasibility.

This paper deals with the problem of retrieving the affine structure of a scene from two or more images of parallel planes. We propose a new approach that is solely based on plane homographies, calculated from point correspondences, and that does not require the recovery of the 3D structure of the scene. Neither vanishing points nor lines need to be extracted from the images. The case of a moving camera with constant intrinsic parameters and the one of cameras with possibly different parameters are both addressed. Extensive experiments with both synthetic and real images have validated our approach.

This paper proposes a new method for identifying parallel planes in a scene from three or more uncalibrated images. By using the fact that parallel planes intersect at infinity, we were able to devise a linear relationship between the inter-image homographies of the parallel planes and the plane at infinity. This relationship is combined with the so-called modulus constraint for identifying pairs of parallel planes solely from point correspondences. Experiments with both synthetic and real images have validated our method.

In this paper, new equations for the self-calibration of a moving camera with unchanged intrinsic parameters are proposed. Unlike most existing methods that require solving equations in three or more unknowns, our equations are only bivariate. The two unknowns, in our equations, are the scale factors that are responsible for the nonlinearity of Kruppa's equations due to a triplet of images. Once the scale factors are calculated, Kruppa's coefficients are linearly retrieved. The results of our experiments, conducted on simulated and real data, are also presented.

In the computation of the 3D structure, it has been demonstrated that by introducing a third camera provides improvement over the use of two views only. In this paper, we propose a new algebraic derivation for the projection matrix of the third camera. This derivation is obtained by using the relationships between the projection matrices, the epipolar geometry and the trilinear tensors. Compared to previous methods, our method has the advantage to work in the special case where the camera centers are collinear. In addition, it has the advantage of simplicity and it is very straightforward to implement since it is a linear method. In our experiments, we used the view synthesis to high-light the quality of the computed projective structure. Tests have been performed on both synthetic and real data.

We claim that time has come in graphics recognition for choosing stable and robust methods, even (or especially) when this means implementing methods proposed by others, instead of inventing a new algorithm which ends up being a minor variation on an old idea. In this spirit, we present some of the choices we have made. We do not cover the whole range of methods and steps used in graphics recognition, but rather concentrate on those which we deem the most crucial and time consuming, in terms of development time, in the design of a graphics recognition system. They include: binarization, text-graphics segmentation, thin-thick separation, vectorisation and arc recognition, and tiling.