In this paper, we speed up robust two-view relative pose from dense correspondences. Previous work has shown that dense matchers can significantly improve both accuracy and robustness in the resulting pose. However, the large number of matches comes with a significantly increased runtime during robust estimation in RANSAC. To avoid this, we propose an efficient match summarization scheme which provides comparable accuracy to using the full set of dense matches, while having 10-100x faster runtime. We validate our approach on standard benchmark datasets together with multiple state-of-the-art dense matchers.

In this paper, we revisit the problem of estimating the relative pose from a sparse set of point-correspondences. For each point-correspondence we also estimate the relative depth, i.e. the relative distance to the scene point in the two images. This yields an additional constraint, allowing us to use fewer matches in RANSAC to generate the pose candidates. In the paper we propose two novel minimal solvers: one for general motion and one for the case of known vertical direction. To obtain the relative depth estimates, we explore using scale estimates obtained from a keypoint detector as well as a neural network that directly predicts the relative depth for a pair of patches. We show in experiments that while our estimates are more noisy compared to the purely point-based solvers, the smaller sample size leads to a significantly reduced runtime in settings with high outlier ratios.

Abstract The task of reconstructing detailed 3D human body models from images is interesting but challenging in computer vision due to the high freedom of human bodies. This work proposes a coarse-to-fine method to reconstruct detailed 3D human body from multi-view images combining Voxel Super-Resolution (VSR) based on learning the implicit representation. Firstly, the coarse 3D models are estimated by learning an Pixel-aligned Implicit Function based on Multi-scale Features (MF-PIFu) which are extracted by multi-stage hourglass networks from the multi-view images. Then, taking the low resolution voxel grids which are generated by the coarse 3D models as input, the VSR is implemented by learning an implicit function through a multi-stage 3D convolutional neural network. Finally, the refined detailed 3D human body models can be produced by VSR which can preserve the details and reduce the false reconstruction of the coarse 3D models. Benefiting from the implicit representation, the training process in our method is memory efficient and the detailed 3D human body produced by our method from multi-view images is the continuous decision boundary with high-resolution geometry. In addition, the coarse-to-fine method based on MF-PIFu and VSR can remove false reconstructions and preserve the appearance details in the final reconstruction, simultaneously. In the experiments, our method quantitatively and qualitatively achieves the competitive 3D human body models from images with various poses and shapes on both the real and synthetic datasets.

3D human pose and shape estimation plays a vital role in many computer vision applications. There are many deep learning based methods attempting to solve the problem only relying on single-view RGB images for training the network. However, since some public datasets are captured from multi-view cameras system, we propose a novel method to tackle the problem by putting optimization-based multi-view model-fitting into a regression-based learning loop from multi-view images. Firstly, a convolutional neural network (CNN) regresses the pose and shape of a parametric human body model (SMPL) from multi-view images. Then, utilizing the regressed pose and shape as initialization, we propose an improved multi-view optimization method based on the SMPLify method (MV-SMPLify) to fit the SMPL model to the multi-view images simultaneously. Subsequently, the optimized parameters can be adopted to supervise the training of the CNN model. This whole process forms a self-supervising framework which can combine the advantages of the CNN approach and the optimization- based approach through a collaborative process. In addition, the multi-view images can provide more comprehensive supervision for the training. Experiments on public datasets qualitatively and quantitatively demonstrate that our method outperforms previous approaches in a number of ways.

In this paper we present a novel algorithm for onboard radial distortion correction for unmanned aerial vehicles (UAVs) equipped with an inertial measurement unit (IMU), that runs in real-time. This approach makes calibration procedures redundant, thus allowing for exchange of optics extemporaneously. By utilizing the IMU data, the cameras can be aligned with the gravity direction. This allows us to work with fewer degrees of freedom, and opens up for further intrinsic calibration. We propose a fast and robust minimal solver for simultaneously estimating the focal length, radial distortion profile and motion parameters from homographies. The proposed solver is tested on both synthetic and real data, and perform better or on par with state-of-the-art methods relying on pre-calibration procedures. Code available at: https://github.com/marcusvaltonen/HomLib.1

For diagnosis and risk assessment in patients with stable ischemic heart disease, myocardial perfusion scintigraphy is one of the most common cardiological examinations performed today. There are however many motivations for why an artificial intelligence algorithm would provide useful input to this task. For example to reduce the subjectiveness and save time for the nuclear medicine physicians working with this time consuming task. In this work we have developed a deep learning algorithm for multi-label classification based on a convolutional neural network to estimate the probability of obstructive coronary artery disease in the left anterior artery, left circumflex artery and right coronary artery. The prediction is based on data from myocardial perfusion scintigraphy studies conducted in a dedicated Cadmium-Zinc-Telluride cardio camera (D-SPECT Spectrum Dynamics). Data from 588 patients was available, with stress images in both upright and supine position, as well as a number of auxiliary parameters such as angina symptoms and age. The data was used to train and evaluate the algorithm using 5-fold cross-validation. We achieve state-of-the-art results for this task with an area under the receiver operating characteristics curve of 0.89 as average on per-vessel level and 0.95 on per-patient level.

With the development of cheap image sensors, the amount of available image data have increased enormously, and the possibility of using crowdsourced collection methods has emerged. This calls for development of ways to handle all these data. In this paper, we present new tools that will enable efficient, flexible and robust map merging. Assuming that separate optimisations have been performed for the individual maps, we show how only relevant data can be stored in a low memory footprint representation. We use these representations to perform map merging so that the algorithm is invariant to the merging order and independent of the choice of coordinate system. The result is a robust algorithm that can be applied to several maps simultaneously. The result of a merge can also be represented with the same type of low-memory footprint format, which enables further merging and updating of the map in a hierarchical way. Furthermore, the method can perform loop closing and also detect changes in the scene between the capture of the different image sequences. Using both simulated and real data - from both a hand held mobile phone and from a drone - we verify the performance of the proposed method.

Reconstruction of 3D human bodies, from images, faces many challenges, due to it generally being an ill-posed problem. In this paper we present a method to reconstruct 3D human bodies from multi-view images, through learning an implicit function to represent 3D shape, based on multi-scale features extracted by multi-stage end-to-end neural networks. Our model consists of several end-to-end hourglass networks for extracting multi-scale features from multi-view images, and a fully connected network for implicit function classification from these features. Given a 3D point, it is projected to multi-view images and these images are fed into our model to extract multiscale features. The scales of features extracted by the hourglass networks decrease with the depth of our model, which represents the information from local to global scale. Then, the multi-scale features as well as the depth of the 3D point are combined to a new feature vector and the fully connected network classifies the feature vector, in order to predict if the point lies inside or outside of the 3D mesh. The advantage of our method is that we use both local and global features in the fully connected network and represent the 3D mesh by an implicit function, which is more memory-efficient. Experiments on public datasets demonstrate that our method surpasses previous approaches in terms of the accuracy of 3D reconstruction of human bodies from images.

In this paper we consider a collection of relative pose problems which arise naturally in applications for visual indoor UAV navigation. We focus on cases where additional information from an onboard IMU is available and thus provides a partial extrinsic calibration through the gravitational vector. The solvers are designed for a partially calibrated camera, for a variety of realistic indoor scenarios, which makes it possible to navigate using images of the ground floor. Current state-of-the-art solvers use more general assumptions, such as using arbitrary planar structures; however, these solvers do not yield adequate reconstructions for real scenes, nor do they perform fast enough to be incorporated in real-time systems. We show that the proposed solvers enjoy better numerical stability, are faster, and require fewer point correspondences, compared to state-of-the-art solvers. These properties are vital components for robust navigation in real-time systems, and we demonstrate on both synthetic and real data that our method outperforms other methods, and yields superior motion estimation.

Estimating the 3D model of the human body is needed for many applications. However, this is a challenging problem since the human body inherently has a high complexity due to self-occlusions and articulation. We present a method to reconstruct the 3D human body model from a single RGB-D image. 2D joint points are firstly predicted by a CNN-based model called convolutional pose machine, and the 3D joint points are calculated using the depth image. Then, we propose to utilize both 2D and 3D joint points, which provide more information, to fit a parametric body model (SMPL). This is implemented through minimizing an objective function, which measures the difference of the joint points between the observed model and the parametric model. The pose and shape parameters of the body are obtained through optimization and the final 3D model is estimated. The experiments on synthetic data and real data demonstrate that our method can estimate the 3D human body model correctly.

In this paper, we consider a mobile platform with two cameras directed towards the floor. In earlier work, this specific problem geometry has been considered under the assumption that the cameras have been mounted at the same height. This paper extends the previous work by removing the height constraint, as it is hard to realize in real-life applications. We develop a method based on an equivalent problem geometry, and show that much of previous work can be reused with small modification to account for the height difference. A fast solver for the resulting nonconvex optimization problem is devised. Furthermore, we propose a second method for estimating the height difference by constraining the mobile platform to pure translations. This is intended to simulate a calibration sequence, which is not uncommon to impose. Experiments are conducted using synthetic data, and the results demonstrate a robust method for determining the relative parameters comparable to previous work.

Sound and radio can be used to estimate the distance between a transmitter and a sender by correlating the emitted and received signal. Alternatively by correlating two received signals it is possible to estimate distance difference. Such methods can be divided into methods that are robust to noise and reverberation, but give limited precision and sub-sample refinements that are sensitive to noise, but give higher precision when initialized close to the real translation. In this paper we develop stochastic models that can explain the limits in the precision of such sub-sample time-difference estimates. Using such models we provide new methods for precise estimates of time-differences as well as Doppler effects. The method is verified on both synthetic and real data.

In this paper we consider a mobile platform with two cameras directed towards the floor mounted the same distance from the ground, assuming planar motion and constant internal parameters. Earlier work related to this specific problem geometry has been carried out for monocular systems, and the main contribution of this paper is the generalization to a binocular system and the recovery of the relative translation and orientation between the cameras. The method is based on previous work on monocular systems, using sequences of inter-image homographies. Experiments are conducted using synthetic data, and the results demonstrate a robust method for determining the relative parameters.

There is a need for an automatic Gleason scoring system that can be used for prostate cancer diagnosis. Today the diagnoses are determined by pathologists manually, which is both a complex and a time-consuming task. To reduce the pathologists' workload, but also to reduce variations between different pathologists, an automatic classification system would be of great use. Some previous works have aimed for this, but still more work needs to be done. It is probable that such a tool would benefit from having access to individually segmented, pathologically relevant objects from the images. Therefore, we have developed an algorithm for semantic segmentation of the microscopic images of H&E stained prostate tissue into Background, Stroma, Epithelial Cytoplasm and Nuclei. This algorithm is based on deep learning, or more specifically a convolutional neural network. The network design is inspired by architectures that previously have been proved successful in different applications. It consists of a contracting and an expanding part, which are symmetrical. We have reached an accuracy of 80 %, as measured by the mean of the intersection over union, for segmentation into four classes. Previous works have only investigated nuclei segmentation, and our network performed similar but for the more challenging task of four class segmentation.

We present a minimal solver for a special kind of homography arising in applications with planar camera motion (e.g. mobile robotics applications). Since the camera motion we consider only has five degrees of freedom, an explicit parametrisation allows us to reduce the required number of point correspondences to 2.5. Using fewer point correspondences is beneficial when used together with RANSAC, but more importantly, the proposed special solver ensures that the estimated homography is of the correct type (in contrast to the DLT, which estimates a general homography). Our method works by enforcing eleven independent polynomial constraints on the elements of this kind of homography matrix, through the framework of the action matrix method for solving polynomial equations. Some analytical investigation using symbolic software has been conducted in order to understand the properties of the polynomial system, and these results have been used to help guide our design of the solver. Additionally, we provide a direct method to recover the sought motion parameters from the homography matrix. We demonstrate that it is possible to recover both the homography and its generating parameters efficiently and accurately.

Patients waiting for heart transplantation due to a failing heart can get a left ventricular assist device (LVAD) implanted through open chest surgery. The device consists of a pump that pumps blood from the left ventricle into the aorta. To get the correct rotation speed of the pump, the physicians consider a number of measurements as well as a sequence of echocardiographic images. The important information obtained from the images is the shape of the inter-ventricular septum. For instance, if the septum bulges towards the left ventricle the speed is too high and it might harm the right ventricular function. To get a measure of the shape of the septum, which can be incorporated in a decision support system, we perform a segmentation of the septum using a shortest path method. To reduce user interaction, the user only needs to annotate two anchor points in the first frame. They mark the endpoints of the septum and they are tracked through the sequence with our tracking algorithm. After the segmentation the septum is divided into two regions, the one closest to the right ventricle and the one closest to the left ventricle, and the desired measure is the difference between the areas of these regions divided by the total septum area. The performance of the segmentation algorithm is acceptable and the obtained septum measure corresponds in most cases to the assessments from a physician.

The Sparse Bundle Adjustment (SBA) algorithm is a widely used method to solve multi-view reconstruction problems in vision. The critical cost of SBA depends on the fill in of the reduced camera matrix whose pattern is known as the Secondary structure of the problem. In centered object applications where a large number of images are taken in a small area the camera matrix obtained when points are eliminated is dense. On the contrary, visual mapping systems where long trajectories are traversed yield sparse matrices. In this paper, we propose a Decomposable Bundle Adjustment (DBA) method which naturally adapts to the fill in pattern of the camera matrix improving the performance on visual mapping systems. The proposed algorithm is able to decompose the normal equations into small subsystems which are ordered in a junction tree structure. To solve the original system, local factorizations of the small dense matrices are passed between clusters in the tree. The DBA algorithm has been tested for simulated and real data experiments for different environment configurations showing good performance.

The top layer of a road is made up of a mixture of stones and bitumen and the durability is dependent on how well the bitumen adheres to the stones. The standard way of determining the bitumen coverage in the industry is the so called rolling bottle method, where a number of stones covered with bitumen are put in a rolling bottle and the bitumen coverage is estimated after different times. This paper describes a novel method for measuring the bitumen coverage of the stones by using advanced segmentation methods instead of manual inspection. The stones are put on a table and a number of images with different exposure times are taken. The images are normalized and the stones are segmented from the background based on a threshold obtained from an optimality criterion. Then the bitumen covered parts of the stones are segmented based on a graph-cut method. The results are compared to manual inspection and are well in agreement with these.

Estimation of structure and motion in computer vision systems can be performed using a dynamic systems approach, where states and parameters in a perspective system are estimated. We present a novel on-line method for structure and motion estimation in densely sampled image sequences. The proposed method is based on an extended Kalman filter and a novel parameterization. We assume calibrated cameras and derive a dynamic system describing the motion of the camera and the image formation. By a change of coordinates, we represent this system by normalized image coordinates and the inverse depths. Then we apply an extended Kalman filter for estimation of both structure and motion. The performance of the proposed method is demonstrated in both simulated and real experiments. We furthermore compare our method to the unified inverse depth parameterization and show that we achieve superior results.

We present a novel calibration method for multi-camera platforms, based on multi-linear constraints. The calibration method can recover the relative orientation between the different cameras on the platform, even when there are no corresponding feature points between the cameras, i.e. there are no overlaps between the cameras. It is shown that two translational motions in different directions are sufficient to linearly recover the rotational part of the relative orientation. Then two general motions, including both translation and rotation, are sufficient to linearly recover the translational part of the relative orientation. However, as a consequence of the speed-scale ambiguity the absolute scale of the translational part can not be determined if no prior information about the motions are known, e.g. from dead reckoning. It is shown that in case of planar motion, the vertical component of the translational part can not be determined. However, if at least one feature point can be seen in two different cameras, this vertical component can also be estimated. Finally, the performance of the proposed method is shown in simulated experiments.

In this paper, we address the problem of segmenting data defined on a manifold into a set of regions with uniform properties. In particular, we propose a numerical method when the manifold is represented by a triangular mesh. Based on recent image segmentation models, our method minimizes a convex energy and then enjoys significant favorable properties: it is robust to initialization and avoid the problem of the existence of local minima present in many variational models. The contributions of this paper are threefold: firstly we adapt the convex image labeling model to manifolds; in particular the total variation formulation. Secondly we show how to implement the proposed method on triangular meshes, and finally we show how to use and combine the method in other computer vision problems, such as 3D reconstruction. We demonstrate the efficiency of our method by testing it on various data.

Structure and motion estimation from long image sequences is a an important and difficult problem in computer vision. We propose a novel approach based on nonlinear and adaptive observers based on a dynamic model of the motion. The estimation of the three-dimensional position and velocity of the camera as well as the three-dimensional structure of the scene is done by observing states and parameters of a nonlinear dynamic system, containing a perspective transformation in the output equation, often referred to as a perspective dynamic system. An advantage of the proposed method is that it is filter-based, i.e. it provides an estimate of structure and motion at each time instance, which is then updated based on a novel image in the sequence. The observer demonstrates a trade-off compared to a more computer vision oriented approach, where no specific assumptions regarding the motion dynamics are required, but instead additional feature points are needed. Finally, the performance of the proposed method is shown in simulated experiments.

In this paper we propose a novel prior-based variational object segmentation method in a global minimization framework which unifies image segmentation and image denoising. The idea of the proposed method is to convexify the energy functional of the Chan-Vese method in order to find a global minimizer, so called continuous graph cuts. The method is extended by adding an additional shape constraint into the convex energy functional in order to segment an object using prior information. We show that the energy functional including a shape prior term can be relaxed from optimization over characteristic functions to optimization over arbitrary functions followed by a thresholding at an arbitrarily chosen level between 0 and 1. Experimental results demonstrate the performance and robustness of the method to segment objects in real images.

In this paper we present the Coupled Active Contours (CAC) model, which is applied to segmentation of the endocardium in ultrasonic images assuming Rayleigh distributed intensities. Comparative experiments, both real and synthetic, with a standard prior model are presented. In the CAC model the prior acts, by affine transformation, on the same image information as the active contour, in addition to the traditional interaction between prior and active contour. By this higher convergence rate and robustness, w.r.t artifacts and poor initialization, is achieved.

This paper builds on a novel framework of hybrid matching constraints for estimation of structure and recovery of camera focal length and motion, combining the advantages of both discrete and continuous methods. Our recursive method can deal with both image noise and outliers. The system is an extension of the epipolar hybrid matching constraints in conjunction with a simple structure estimation scheme using standard triangulation. The extension enables the system to deal with varying focal length of the camera. The structure obtained from some previous image frames is used to improve estimates of the camera focal length and motion for the current image frame. These are, in turn, used to refine the structure. Finally, a RANSAC outlier rejection scheme is employed to reject outlier tracks, inevitably obtained from any tracker. The performance of the proposed system is demonstrated on simulated experiments.

In this paper we propose a new method for variational segmentation of image sequences containing nonrigid, moving objects. The method is based on the Chan-Vese model augmented with a novel frame-to-frame interaction term, which allow us to update the segmentation result from one image frame to the next using the previous segmentation result as a shape prior. The interaction term is constructed to be pose-invariant and to allow moderate deformations in shape. It can handle the appearance of occlusions which otherwise can make segmentation fail. The performance of the model is illustrated with experiments on synthetic and real image sequences.

Many visual cues for surface reconstruction from known views are sparse in nature, e.g., specularities, surface silhouettes, and salient features in an otherwise textureless region. Often, these cues are the only information available to an observer. To allow these constraints to be used either in conjunction with dense constraints such as pixel-wise similarity, or alone, we formulate such constraints in a variational framework. We propose a sparse variational constraint in the level set framework, enforcing a surface to pass through a specific point, and a sparse variational constraint on the surface normal along the observed viewing direction, as is the nature of, e.g., specularities. These constraints are capable of reconstructing surfaces from extremely sparse data. The approach has been applied and validated on the shape from specularities problem.

In this paper we consider region-based variational segmentation of two- and three-dimensional images by the minimization of functionals whose fidelity term is the quotient of two integrals. Users often refrain from quotient functionals, even when they seem to be the most natural choice, probably because the corresponding gradient descent PDEs are nonlocal and hence require the computation of global properties. Here it is shown how this problem may be overcome by employing the structure of the Euler-Lagrange equation of the fidelity term to construct a good initialization for the gradient descent PDE, which will then converge rapidly to the desired (local) minimum. The initializer is found by making a one-dimensional search among the level sets of a function related to the fidelity term, picking the level set which minimizes the segmentation functional. This partial extremal initialization is tested on a medical segmentation problem with velocity- and intensity data from MR images. In this particular application, the partial extremal initialization speeds up the segmentation by two orders of magnitude compared to straight forward gradient descent.

In this paper, a technique for calibrating a camera using a planar calibration object with known metric structure, when the camera (or the calibration plane) undergoes pure translational motion, is presented. The study is an extension of the standard formulation of plane-based camera calibration where the translational case is considered as degenerate. We derive a flexible and straightforward way of using different amounts of knowledge of the translational motion for the calibration task. The theory is mainly applicable in a robot vision setting, and the calculation of the hand-eye orientation and the special case of stereo head calibration are also being addressed. Results of experiments on both computer-generated and real image data are presented. The paper covers the most useful instances of applying the technique to a real system and discusses the degenerate cases that needs to be considered. The paper also presents a method for calculating the infinite homography between the two image planes in a stereo head, using the homographies estimated between the calibration plane and the image planes. Its possible usage and usefulness for simultaneous calibration of the two cameras in the stereo head are discussed and illustrated using experiments

The problem of feature-based surface reconstruction is considered in this paper. Our main contribution is the ability to handle visibility constraints, obtained from the projections of points, curves and silhouettes, in the surface fitting process. While traditional methods often ignore such information, we show that visibility constraints not only give better initial surface estimates and faster convergence, but also provide an important cue for determining surface topology. The problem is cast as a variational problem with constraints within the level set framework. It is shown how to evolve the surface without violating the visibility constraints using methods from variational calculus. Applications of the theory are detailed for a number of important cases of geometric primitives: points, curves and visual hulls. Several experiments on real image sequences are given to demonstrate the performance of the approach.

Camera calibration with one-dimensional objects is based on an algebraic constraint on the image of the absolute conic. We give an alternative derivation to this constraint, allowing a geometrical interpretation. From this, we derive the degenerate cases, or critical motions, where the calibration algorithm fails. We also show that constraints on the intrinsic parameters lead to simplified closed-form solutions and a reduced set of critical motions. A simulation and a real data experiment is performed to evaluate the accuracy of the calibration result for motions close to being critical.

Estimation of three-dimensional information from two-dimensional images is an important requirement in many computer vision applications. The estimation task can often be formulated as a problem of estimating states and/or parameters in nonlinear dynamic systems. This paper presents an algorithm for recursive state estimation in nonlinear dynamic systems, where the estimated states correspond to three-dimensional positions of feature points on an observed object. The algorithm is designed as a nonlinear observer, with a gain matrix that can be determined using methods from linear control theory. A stability criterion for the resulting nonlinear system is derived, and simulations are presented in order to illustrate the estimation performance.

It is widely known that, for the affine camera model, both shape and motion can be factorized directly from the so-called image measurement matrix constructed from image point coordinates. The ability to extract both shape and motion from this matrix by a single SVD operation makes this shape-from-motion approach attractive; however, it can not deal with missing feature points and, in the presence of outliers, a direct SVD to the matrix would yield highly unreliable shape and motion components. Here, we present an outlier correction scheme that iteratively updates the elements of the image measurement matrix. The magnitude and sign of the update to each element is dependent upon the residual robustly estimated in each iteration. The result is that outliers are corrected and retained, giving improved reconstruction and smaller reprojection errors. Our iterative outlier correction scheme has been applied to both synthesized and real video sequences. The results obtained are remarkably good.

Traditionally, building 3D reconstructions of large scenarios such as a museum or historical site has been costly, time consuming and required the contribution of expert personnel. Usually the results showed an artificial look and had little interactivity. However, newly developed technologies in the areas of video analysis, camera calibration and texture fusion allow us to think in a much more satisfying scenario where the user with the only aid of a domestic video camera is able to acquire all the information it is required to construct the 3D model of the desired environment in an easy and comfortable manner. In this paper, the results obtained in the EC funded project VISIRE are presented. VISIRE attempts to construct photorealistic 3D models of large scenarios using as input multiple freehand video sequences. Once acquired, the computer vision software processes the video information off-line in order to obtain the 3D mesh together with the textures required to obtain a 3D model highly resembling the original.

In this paper we investigate how intrinsic camera calibration and hand-eye calibration can be performed on a robot vision system using the simplest possible motions and a planar calibration object. The standard methods on plane-based camera calibration are extended with theory on how to use pure translational motions for the intrinsic calibration and we see how hand-eye calibration can be performed within the same framework. The calibration of two cameras in a stereo head configuration is shown to be an interesting application of the developed theory. Results of experiments on a real robot vision system are presented.

In this paper we examine the ambiguities between extrinsic and intrinsic parameters in the uncalibrated structure and motion problem. The Jacobian J of the reprojection error is considered, treating each camera separately. Ambiguities correspond to linear dependencies in the column space of J and we thus detect and quantify these by using the condition number of selected combinations of columns of J. When the presence of an ambiguity has been detected, we automatically select a constraint (e.g. constant principal point, or a regularity constraint on the camera motion) that resolves this ambiguity. As a by-product we also obtain a better separation between intrinsic and extrinsic parameters. The proposed method is demonstrated on both synthetic and real data, with good performance.

Based on our previous work on the use of subspace dis-tances for the outlier deiection problem in video sequences under afFne projection. this paper reports ourfurther anal-ysis of the problem and presents fwo algorithms for com-puting the reprojection errors of imagefeatures in the out-lier detection process. Extensive experiments on real video sequences have been conducted to verrfi the performance ofthe algorithms. The key contributions ofthe paper are presentation offhe relationship befween subspace distances and reprojection errors and demonstration thar repmjec-tion errors can be estimated without erplicitly computing the projective structure. 1.

In this paper we show how to calibrate a camera, mounted on a robot, with respect to the intrinsic camera parameters when the so-called hand-eye transformation between the robot hand and the camera is unknown. The calibration is based directly on the spatial and temporal derivatives in an image sequence and do not need any matching and tracking of features or a reference object. The calibration is to be performed on an active robot vision system where the motion of the robot hand can be controlled. A minimum of 3 non-coplanar translations of the robot hand are needed for the calculation. In conjunction with the intrinsic camera calibration the orientation of the camera, with respect to the robot hand, is calculated. The position of the camera can then also be obtained. At each stage only the image derivatives and the known motion of the robot hand are used. For the full, intrinsic and extrinsic, calibration a total of 5 distinct motions are used. The algorithm has been tested in extensive experiments with respect to e.g. noise sensitivity.

A novel robust method for outlier detection in structure and motion recovery for affine cameras is presented. It is an extension of the well-known Tomasi-Kanade factorization technique (C. Tomasi T. Kanade, 1992) designed to handle outliers. It can also be seen as an importation of the LMedS technique or RANSAC into the factorization framework. Based on the computation of distances between subspaces, it relates closely with the subspace-based factorization methods for the perspective case presented by G. Sparr (1996) and others and the subspace-based-factorization for affine cameras with missing data by D. Jacobs (1997). Key features of the method presented are its ability to compare different subspaces and the complete automation of the detection and elimination of outliers. Its performance and effectiveness are demonstrated by experiments involving simulated and real video sequences.

A technique for stereo camera calibration from a known planar calibration object is proposed. Both the intrinsic parameters and the relative orientation are calculated. The proposed algorithm uses the two homogeneous linear constraints on the image of the absolute conic arising from the plane homographies for each camera and position. In addition to these, linear constraints on the relation between the images of the absolute conic for the two cameras in the stereo pair are calculated. The derivation of these constraints is made possible by considering the rigidity of the stereo pair. Using all these constraints simultaneously in a linear system gives a stereo calibration technique that is less sensitive to noise, compared to just using the single camera constraints. As a part of the algorithm, a simple way to calculate the infinite homography from two stereo views of a plane is presented.The performance of the algorithm is shown in experiments on both simulated and real data.

In this paper, an iterative algorithm for auto-calibration is presented. The proposed algorithm switches between linearly estimating the dual of the absolute conic and the intrinsic parameters, while also incorporating the rank-3 constraint on the intrinsic parameters. The most important property of the algorithm is that it is completely general in the sense that any type of constraint on the intrinsic parameters might be used. The proposed algorithm locates in-between of a non-linear optimization and initial linear computation, and provides robust and sufficiently accurate initial values for a bundle adjustment routine. The performance of the algorithm is shown for both simulated and real data, especially in the important case of natural (zero skew and unit aspect ratio) cameras.

This paper deals with the problem of incorporating natural regularity conditions on the motion in an MAP estimator for structure and motion recovery from uncalibrated image sequences. The purpose of incorporating these constraints is to increase performance and robustness. Auto-calibration and structure and motion algorithms are known to have problems with (i) the frequently occurring critical camera motions, (ii) local minima in the non-linear optimization and (iii) the high correlation between different intrinsic and extrinsic parameters of the camera, e.g. the coupling between focal length and camera position. The camera motion (both intrinsic and extrinsic parameters) is modelled as a random walk process, where the inter-frame motions are assumed to be independently normally distributed. The proposed scheme is demonstrated on both simulated and real data showing the increased performance.

The minimal data necessary for projective reconstruction from image points is well-known when each object point is visible in all images. We formulate and propose solutions to a family of reconstruction problems for multiple images from minimal data, where there are missing points in some of the images. The ability to handle the minimal cases with missing data is of great theoretical and practical importance. It is unavoidable to use them to bootstrap robust estimation such as RANSAC and LMS algorithms and optimal estimation such as bundle adjustment. First, we develop a framework to parameterize the multiple view geometry needed to handle the missing data cases. Then, we present a solution to the minimal case of eight points in three images, where one different point is missing in each of the three images. We prove that there are, in general, as many as 11 solutions for this minimal case. Furthermore all minimal cases with missing data for three and four images are catalogued. Finally, we demonstrate the method on both simulated and real images and show that the algorithms presented in the paper can be used for practical problems.

This paper presents a novel method for structure and motion estimation from affine cameras, called direct affine reconstruction (DAR). The main contribution is a deeper theoretical understanding of multiple view geometry for affine cameras. This is accomplished by carefully selecting a specific coordinate system, based on relative affine coordinates. One consequence of the choice of coordinates is that it is possible to directly read out the camera matrices as well as the object coordinates from the image measurements. The proposed method is well suited for tracking purposes as well as robust estimation schemes, like RANSAC, since only four basis points are needed. Furthermore, since no costly calculations are involved, the method is very fast. Experiments are carried out on both simulated and real data, showing its relative performance against factorization.

Traditionally, hand-eye calibration has been done using point correspondences, reducing the problem to a matrix equation. This approach requires reliably detected and tracked points between images taken from fairly widespread locations. We present a new approach to performing hand-eye calibration. The novelty of the proposed method lies in the fact that instead of point correspondences, normal derivatives of the image flow field are used. First, two different small translational motions are made, enabling the direction of the optical axis to be computed from image derivatives only. Next, at least two different rotational motions are made, enabling also the translational part of the hand-eye transformation to be estimated. It is also shown how to compute a depth reconstruction from the information obtained in the hand-eye calibration algorithm. Finally, we discuss how to calculate the derivatives and present some experiments on synthetic data.

This paper describes an iterative algorithm for making Euclidean reconstruction of a scene from an image sequence captured by a camera with zero skew. The output consists of both the Euclidean reconstruction and the intrinsic parameters of the camera at the different imaging instants, i.e. it also provides a camera calibration. The problem is solved in two different steps. Firstly, the projective structure is obtained from a factorization method followed by a bundle adjustment method. Secondly, the Euclidean reconstruction is obtained from an iterative method that estimates the location of the absolute conic and the intrinsic parameters iteratively, using linear operations in each iteration. In this method a new constraint, called the orthogonality constraint, is used to constrain the absolute conic. Results are shown on experiments on both synthetic and real data.

The minimal data necessary for projective reconstruction from point correspondences is well-known when the points are visible in all images. In this paper, we formulate and propose solutions to a new family of reconstruction problems from multiple images with minimal data, where there are missing points in some of the images. The ability to handle the minimal cases with missing data is of great theoretical and practical importance. It is unavoidable to use them to bootstrap robust estimation such as RANSAC and LMS algorithms and optimal estimation such as bundle adjustment. First, we develop a framework to parametrize the multiple view geometry, needed to handle the missing data cases. Then we present a solution to the minimal case of 8 points in 3 images, where one of the points is missing in one of the three images. We prove that there are in general as many as 11 solutions for this minimal case. Furthermore, all minimal cases with missing data for 3 and 4 in images are catalogued. Finally we demonstrate the method on both simulated and real images and show that the algorithms presented in this paper can be used for practical problems.

In this paper we extend the notion of affine shape, introduced by Sparr (1995, 1996), from finite point sets to curves. The extension makes it possible to reconstruct 3D-curves up to projective transformations, from a number of their 2D-projections. We also extend the bundle adjustment technique from point features to curves. The first step of the curve reconstruction algorithm is based on affine shape, is independent of choice of coordinates, robust, does not rely on any preselected parameters and works for an arbitrary number of images. In particular this means that a solution is given to the aperture problem of finding point correspondences between curves. The second step takes advantage of any knowledge of measurement errors in the images. This is possible by extending the bundle adjustment technique to curves. Finally, experiments are performed on both synthetic and real data to show the performance and applicability of the algorithm.

This paper deals with the concept of auto-calibration, i.e. methods to calibrate a camera on-line. In particular we deal with minimal conditions on the intrinsic parameters needed to make a Euclidean reconstruction, called flexible calibration. The main theoretical results are that it is only needed to know that one intrinsic parameter is constant. The method is based on an initial projective reconstruction, which is upgraded to a Euclidean one. The number of images needed increases with the complexity of the constraints, but the number of points needed is only the number needed in order to obtain a projective reconstruction. The theoretical results are exemplified in a number of experiments. An algorithm, based on bundle adjustments and a linear initialization method are presented and experiments are performed on both synthetic and real data.

In this paper it is shown hour corresponding conics in two images can be used to estimate the epipolar geometry in terms of the fundamental/essential matrix. The corresponding conics can, be images of either planar celtics or silhouettes of quadrics. It is shown that one conic correspondence gives two independent constraints on the fundamental matrix and a method to estimate the fundamental matrix from at least four corresponding conics is presented. Furthermore, a new type of fundamental matrix for describing conic correspondences is introduced. Finally, it is shown that the problem of estimating the fundamental matrix from 5 point correspondences and 1 conic correspondence in general has 10 different solutions. A method to calculate these solutions is also given together with an experimental validation.

This paper outlines a new method that makes reconstruction from an image sequence taken by affine cameras. The method is based on the so called closure constraints that link the camera matrices to the different affine quasi-tensors. This method can easily handle missing data and not only points, but also lines and conics are used to constrain the reconstruction. The method works in three steps: 1) the second or third order affine quasi-tensors are estimated from corresponding points, lines and conics in two or three images; 2) all available quasi-tensor components are used to calculate the camera matrices using the closure constraints; and 3) the reconstruction is obtained by intersection. When using the second order quasi-tensors, it is sufficient to estimate the quasi-tensors between images i and i+1 and between images i and i+2. In the case of the third order quasi-tensors, it is sufficient to use every successive triplets of images. Finally, the method is illustrated on real data.

A new approach to self-calibration and Euclidean reconstruction from image sequences is presented. The key idea is to start with the affine camera model as a first approximation to obtain the affine 3D structure. It is then upgraded to an Euclidean structure and finally, refined by applying the full perspective camera model and bundle adjustment. The proposed scheme makes no assumption about the scene nor the camera motion. The only assumption required is that the camera has zero skew, which is a minimal condition in order to self-calibrate the camera. However, if other information is available about the camera, it can and should be incorporated. The method is robust and it also provides an estimate of the accuracy of the estimated parameters. Experiments are presented to illustrate the performance of the approach.

A novel recursive method for estimating structure and motion from image sequences is presented. The novelty lies in the fact that the output of the algorithm is independent of the chosen coordinate systems in the images as well as the ordering of the points. It relies on subspace methods and is derived from both ordinary coordinate representations and camera matrices and from a so called depth and shape analysis. Furthermore, no initial phase is needed to start up the algorithm. It starts directly with the first two images and incorporates new images as soon as new corresponding points are obtained. The performance of the algorithm is shown on simulated data. Moreover, the two different approaches, one using camera matrices and the other using the concepts of affine shape and depth, are unified into a general theory of structure and motion from image sequences.

The special case of reconstruction from image sequences taken by cameras with skew equal to 0 and aspect ratio equal to 1 has been treated. These type of cameras, here called cameras with Euclidean image planes, represent rigid projections where neither the principal point nor the focal length is known, it is shown that it is possible to reconstruct an unknown object from images taken by a camera with Euclidean image plane up to similarity transformations, i.e., Euclidean transformations plus changes in the global scale. An algorithm, using bundle adjustment techniques, has been implemented. The performance of the algorithm is shown on simulated data.

In this paper we study the infinitesimal-time case of the so called multilinear constraints that exist for each subsequence in a sequence of images. These constraints link the infinitesimal motion of the image points with the infinitesimal viewer motion. The analysis is done both for calibrated and uncalibrated cameras. Two simplifications are also presented for the uncalibrated camera case. One simplification is made using affine reduction and kinetic depth. The second simplification is based upon a projective reduction with respect to the image of a planar patch.

Stochastic analysis of edge detectors can be made either by theoretical modeling of the image formation process and the edge detectors or by empirical stochastic analysis of the edge locations. In this paper we study and model the image formation process in detail. In particular the much neglected discretisation process is modelled and taken into account. This makes it possible to define and analyse sub-pixel edge detection. The theoretical results are verified through stochastic analysis of both simulated and real image data.

In the high-level operations of computer vision it is taken for granted that image features have been reliably detected. This paper addresses the problem of feature extraction by scale-space methods. This paper is based on two key ideas: to investigate the stochastic properties of scale-space representations, and to investigate the interplay between discrete and continuous images. These investigations are then used to predict the stochastic properties of sub-pixel feature detectors.

A new method for Euclidean reconstruction from sequences of images taken by uncalibrated cameras, with constant intrinsic parameters, is described. Our approach leads to a variant of the so called Kruppa equations. It is shown that it is possible to calculate the intrinsic parameters as well as the Euclidean reconstruction from at least three images. The novelty of our approach is that we build our calculation on a projective reconstruction obtained without the assumption on constant intrinsic parameters. This assumption simplifies the analysis, because a projective reconstruction is already obtained and we need "only" to find the correct Euclidean reconstruction among all possible projective reconstructions.

We describe a new method to evaluate corner extraction schemes using invariance methods. Since the locations of centers in an image depend both on the intrinsic parameters of the camera and the relative position and orientation of the object with respect to the camera, the exact positions of corners in an image are generally not known. To circumvent the need for this knowledge, we use sets of points (instead of individual points) extracted from images of polyhedral objects and projective invariants to calculate a manifold of constraints on the coordinates of the corners. We then estimate the variance of the detected corners from the distance of the coordinate vector to this manifold. This is independent of the camera parameters and the relative position and orientation between the camera and the object. Five different kinds of corner extraction schemes are investigated. The purpose of the paper is to show that invariance methods can effectively be used to make this comparison rather than to make a thorough comparison of different corner extraction schemes.

The paper deals with the problem of reconstructing the locations of n points in space from m different images without camera calibration. It shows how these problems can be put into a similar theoretical framework. A new concept, the reduced fundamental matrix, is introduced. It contains just 4 parameters and can be used to predict locations of points in the images and to make reconstruction. We also introduce the concept of reduced fundamental tensor which describes the relations between points in 3 images. It has 15 components and depends on 9 parameters. Necessary and sufficient conditions for a tensor to be a reduced fundamental tensor are derived. This framework can be generalised to a sequence of images. The dependencies between the different representations are investigated. Furthermore a canonical form of the camera matrices in a sequence are presented.>