We present a novel view on the indoor visual localization problem, where we avoid the use of interest points and associated descriptors, which are the basic building blocks of most standard methods. Instead, localization is cast as an alignment problem of the edges of the query image to a 3D model consisting of line segments. The proposed strategy is effective in low-textured indoor environments and in very wide baseline setups as it overcomes the dependency of image descriptors on textures, as well as their limited invariance to view point changes. The basic features of our method, which are prevalent indoors, are line segments. As we will show, they allow for defining an efficient Chamfer distance-based aligning cost, computed through integral contour images, incorporated into a first-best-search strategy. Experiments confirm the effectiveness of the method in terms of both, accuracy and computational complexity.

We present a novel SfM pipeline which estimates motion and wiry 3D structure from imaged line segments across multiple views. Though the position of lines can be determined more accurately than point features, the issue of detecting stable endpoints diverted most research focus away to point-based methods. In our approach, we tackle this problem by utilizing relaxed constraints on the endpoint positions both during matching as well as in the Bundle Adjustment stage. Furthermore, we gain efficiency in estimating trifocal image relations by decoupling rotation and translation. To this end, a novel linear solver for relative translation estimation given rotations from five line correspondences in three views is introduced. Experiments on long sequences show that our line-based SfM pipeline advantageously complements point-based methods, giving more meaningful 3D representation for indoor scenarios.

This paper presents a novel method for dense 3D reconstruction of man-made environments. Such environments suffer from texture less and non-Lambertian surfaces, where conventional, feature-Based 3D reconstruction pipelines fail to obtain good feature matches. To compensate this lack of feature matches, we exploit the semantic information available in 2D images to estimate both a corresponding 3D position and a 3D surface normal for each pixel. A semantic classifier is therefore applied on a single segmented image in order to get a likelihood for a segment providing one of the surface normals within a discrete set of them. To improve the accuracy of this labeling step, we exploit multiple segmentation methods. The global best surface normal configuration over all pixels of an image is then obtained by using a Markov Random Field. In the last step, the 3D model of a single 2D input image is reconstructed by combining the semantic surface normal estimation with the sparse point cloud coming from feature Based matching. It is shown experimentally, that our proposed method clearly outperforms state-of-the-art dense 3D reconstruction pipelines and surface layout estimation approaches.

This paper presents a computational effective framework for tracking and pose estimation of vehicles in videos reaching comparable performance to state-of-the-art methods. We cast the problem of vehicle tracking as ranking possible poses for each frame and connecting subsequent poses by exploiting a feasible motion model over time. As a novelty, we use random forests trained on a set of existing 3D models for estimating the pose. We discretize the viewpoint space for training, where a synthetic camera is orbiting around the models. To compare projections of 3D models to real world 2D input frames, we introduce simple but discriminative principle gradient features to describe both images. A Markov Random Field ensures to pick the perfect pose over time and the vehicle to follow a feasible motion. As can be seen from our experiments performed on a variety of videos with vast variation of vehicle types, the proposed framework achieves similar results in less computational time compared to state-of-the-art methods.

We present a novel minimal solution for the uncalibrated absolute pose problem given one vanishing point. The proposed method complements state-of-the-art minimal absolute pose solvers and contributes to the handling camera localization problem in poor textured indoor environments. Those environments almost always exhibit edges aligned with the gravity direction which can be detected in images as a vanishing point. The vanishing point with suitable parametrization of the unknowns allows to cast the problem as a Quartic Eigenvalue Problem, a type of nonlinear problem for which very effective solvers exist. The proposed solution yields a reduction in one of the point correspondences which are required by PnP algorithms. In comparison to the general four point PnP algorithms, our method is numerically more stable and more robust to noise. This is demonstrated on a variety of synthetic and real data in scope of calibrating surveillance and mobile phone cameras.

Radial lens distortion found in real, most notably off-the-shelf medium to wide angle optics can be quite severe. A-priori calibration remedies the problem, but requires access to the camera. Alternative approaches make use of correspondences in multiple images of a moving camera [3], relying on sufficiently overlapping views. Our paper fits in the category of techniques, where straight scene lines are used to determine distortion parameters from a single image. These plumbline methods [2] critically depend on the existence of long edges and get thrown of easily if a significant portion originates from non-linear structures. Here, the use of vanishing points provides an advantage: When a large number of image edges groups w.r.t. to a common vanishing point, it is very likely that they stem from parallel scene lines. Recent methods for vanishing point detection either ignore radial distortion completely, or assume only weak distortion, which is accounted for after vanishing point estimation [1, 4]. If strong distortion is present the effects are detrimental to the extraction of vanishing points and such algorithms are bound to fail. Furthermore, when dealing with radial distortion effects in refinement stages, the error is assessed in the undistorted image. This creates significant bias towards estimates shrinking the undistorted image [5] Contributions: To overcome this limitation, we suggest a simple RANSAC procedure for vanishing point detection by clustering of distorted lines. The proposed technique builds upon: a) Our newly devised closed-form solution for simultaneously estimating a vanishing point and radial distortion from three distorted image lines. b) A consistency measure which avoid bias by quantifying the error of lines w.r.t. a vanishing point in the distorted image. Specifically, our work bases on the division model [3], mapping a distorted image point x = (x,y) to the undistorted point x by

We present a method for estimating the relative pose of two calibrated or uncalibrated non-overlapping surveillance cameras from observing a moving object. We show how to tackle the problem of missing point correspondences heavily required by SfM pipelines and how to go beyond this basic paradigm. We relax the non-linear nature of the problem by accepting two assumptions which surveillance scenarios offer, i.e. the presence of a moving object and easily estimable gravity vector. By those assumptions we cast the problem as a Quadratic Eigenvalue Problem offering an elegant way of treating nonlinear monomials and delivering a quasi closed-form solution as a reliable starting point for a further bundle adjustment. We are the first to bring the closed form solution to such a very practical problem arising in video surveillance. Results in different camera setups demonstrate the feasibility of the approach.

We present a method for reconstructing a trajectory of an object moving in front of non-overlapping fully or partially calibrated cameras. The non-overlapping setup turns that problem ill-posed as no point correspondences can be established which are necessary for the well known point triangulation. The proposed solution instead builds on the assumption of trajectory smoothness and depth ordering prior information. We propose a novel formulation with a consistent minimization criterion and a way to utilize the depth ordering prior reflected by the size change of a bounding box associated to an image point being tracked. Reconstructing trajectory minimizing the trajectory smoothness, its re-projection error and employing the depth priors is casted as the Second Order Cone Program yielding a global optimum. The new formulation together with the proposed depth prior significantly improves the trajectory reconstruction in sense of accuracy and topology, and speeds up the solver. Synthetic and real experiments validate the feasibility of the proposed approach.

In this paper we present a novel object classification and pose recovery algorithm which takes advantage of existing 3D models and multiple synchronized and calibrated views. Having a calibrated scenario provides redundant data which can be exploited for gathering spatial consistency of an object's 3D pose and its class. In a first step, the cameras need to be calibrated and aligned to one common coordinate system. A training set of 3D models, a calibrated setup and Harris corner features are used to find the best fitting 2D projection for an object within the scene. The results are improved by aligning multiple synchronized views to gain spatial consistency. Our experiments using real data show the enhanced results using a calibrated setup over analyzing each camera separately.

We propose a novel method for automatic camera calibration and foot-head homology estimation by observing persons standing at several positions in the camera field of view. We demonstrate that human body can be considered as a calibration target thus avoiding special calibration objects or manually established fiducial points. First, by assuming roughly parallel human poses we derive a new constraint which allows to formulate the calibration of internal and external camera parameters as a Quadratic Eigenvalue Problem. Secondly, we couple the calibration with an improved effective integral contour based human detector and use 3D projected models to capture a large variety of person and camera mutual positions. The resulting camera auto-calibration method is very robust and efficient, and thus well suited for surveillance applications where the camera calibration process cannot use special calibration targets and must be simple.

This paper presents a new approach to the problem of camera localization with non-overlapping camera views, particularly relevant for video surveillance systems. We show how to recast localization as quasi-convex optimization under the L-Infinity norm. Thereby we add the problem of reconstructing camera centers and 3D points for non-overlapping cameras with known internal parameters and known rotations to the class of known geometric problems solvable with Second Order Cone Programming. The 3D points are never seen by more than one camera, which makes the localization problem ill-posed. Therefore, the proposed approach employs temporal consistency of the 3D points to supply the missing constraints. Our formulation allows a global optimal solution to be found with a clear physical meaning of the cost function being minimized.

City environments often lack textured areas, contain repetitive structures, strong lighting changes and therefore are very difficult for standard 3D modeling pipelines. We present a novel unified framework for creating 3D city models which overcomes these difficulties by exploiting image segmentation cues as well as presence of dominant scene orientations and piecewise planar structures. Given panoramic street view sequences, we first demonstrate how to robustly estimate camera poses without a need for bundle adjustment and propose a multi-view stereo method which operates directly on panoramas, while enforcing the piecewise planarity constraints in the sweeping stage. At last, we propose a new depth fusion method which exploits the constraints of urban environments and combines advantages of volumetric and viewpoint based fusion methods. Our technique avoids expensive voxelization of space, operates directly on 3D reconstructed points through effective kd-tree representation, and obtains a final surface by tessellation of backprojections of those points into the reference image.

Indoor and outdoor urban environments posses many regularities which can be efficiently exploited and used for general image parsing tasks. We present a novel approach for detecting rectilinear structures and demonstrate their use for wide baseline stereo matching, planar 3D reconstruction, and computation of geometric context. Assuming a presence of dominant orthogonal vanishing directions, we proceed by formulating the detection of the rectilinear structures as a labeling problem on detected line segments. The line segment labels, respecting the proposed grammar rules, are established as the MAP assignment of the corresponding MRF. The proposed framework allows to detect both full as well as partial rectangles, rectangle-in-rectangle structures, and rectangles sharing edges. The use of detected rectangles is demonstrated in the context of difficult wide baseline matching tasks in the presence of repetitive structures and large appearance changes.

In this paper, we describe the components of a novel algorithm for the extraction of dominant orthogonal planar structures from monocular images taken in indoor environments. The basic building block of our approach is the use of vanishing points and vanishing lines imposed by the frequently observed dominance of three mutually orthogonal vanishing directions in man-made world. Vanishing points are found by an improved approach, taking no assumptions on known internal or external camera parameters. The problem of detecting planar patches is attacked using a probabilistic framework, searching for the maximum a posteriori probability (MAP) in a Markov Random Field (MRF). For this, we propose a novel formulation fusing geometric information obtained from vanishing points and features, such as rectangles and partial rectangles, together with a color-homogeneity criteria imposed by an image over-segmentation. The method was evaluated on a set of images exhibiting largely varying characteristics concerning image quality and scene complexity. Experiments show that the method, despite the variations, works in a stable manner and that its performance compares favorably to the state-of-the-art.

Image segmentation methods like active shape models, active appearance models or snakes require an initialisation that guarantees a considerable overlap with the object to be segmented. In this paper we present an approach that localises anatomical structures in a global manner by means of Markov Random Fields (MRF). It does not need initialisation, but finds the most plausible match of the query structure in the image. It provides for precise, reliable and fast detection of the structure and can serve as initialisation for more detailed segmentation steps. Sparse MRF Appearance Models (SAMs) encode a priori information about the geometric configurations of interest points, local features at these points and local features along the edges of adjacent points. This information is used to formulate a Markov Random Field and the mapping of the modeled object (e.g. a sequence of vertebrae) to the query image interest points is performed by the MAX-SUM algorithm. The local image information is captured by novel symmetry-based interest points and local descriptors derived from Gradient Vector Flow. Experimental results are reported for two data-sets showing the applicability to complex medical data. 1

We formulate single-image multi-label segmentation into regions coherent in texture and color as a MAX-SUM problem for which efficient linear programming based solvers have recently appeared. By handling more than two labels, we go beyond widespread binary segmentation methods, e.g., MIN-CUT or normalized cut based approaches. We show that the MAX-SUM solver is a very powerful tool for obtaining the MAP estimate of a Markov random field (MRF). We build the MRF on superpixels to speed up the segmentation while preserving color and texture. We propose new quality functions for setting the MRF, exploiting priors from small representative image seeds, provided either manually or automatically. We show that the proposed automatic segmentation method outperforms previous techniques in terms of the global consistency error evaluated on the Berkeley segmentation database.

We present a method that partitions a single image into two layers, requiring that one layer has similar properties in terms of pixel colour variation to a provided template patch. First, the paper provides a new view on dening a similarity function for a pixel with its small neighbourhood to be part of the texture described by the template patch. This results in better description of pixels near the texture boundary. Second, it is shown how the Maximally Stable Extremal Regions (MSERs), originally designed for wide baseline stereo matching, can be used to locally merge pixels having the same intensity and thus reduce the dimension of the graph representing the image. The MSERs help in texture description and yield signicant reduction of memory and computation time. Finally the graph is fed into the min.cut/max.o w algorithm to cut the graph into two parts. Performance of the method is presented on some images from the Berkeley database. Finally, restrictions in using the method are discussed.

This paper presents a method for fully automatic and robust estimation of two-view geometry, autocalibration, and 3D metric reconstruction from point correspondences in images taken by cameras with wide circular field of view. We focus on cameras which have more than 180 degrees field of view and for which the standard perspective camera model is not sufficient, e.g., the cameras equipped with circular fish-eye lenses Nikon FC-E8 (183 degrees), Sigma 8mm-f4-EX (180 degrees), or with curved conical mirrors. We assume a circular field of view and axially symmetric image projection to autocalibrate the cameras. Many wide field of view cameras can still be modeled by the central projection followed by a nonlinear image mapping. Examples are the above-mentioned fish-eye lenses and properly assembled catadioptric cameras with conical mirrors. We show that epipolar geometry of these cameras can be estimated from a small number of correspondences by solving a polynomial eigenvalue problem. This allows the use of efficient RANSAC robust estimation to find the image projection model, the epipolar geometry, and the selection of true point correspondences from tentative correspondences contaminated by mismatches. Real catadioptric cameras are often slightly noncentral. We show that the proposed autocalibration with approximate central models is usually good enough to get correct point correspondences which can be used with accurate noncentral models in a bundle adjustment to obtain accurate 3D scene reconstruction. Noncentral camera models are dealt with and results are shown for catadioptric cameras with parabolic and spherical mirrors.

We generalize the method of simultaneous linear estimation of multiple view geometry and lens distortion, introduced by Fitzgibbon at CVPR 2001, to an omnidirectional (angle of view larger than 180/spl deg/) camera. The perspective camera is replaced by a linear camera with a spherical retina and a nonlinear mapping of the sphere into the image plane. Unlike the previous distortion-based models, the new camera model is capable to describe a camera with an angle of view larger than 180/spl deg/ at the cost of introducing only one extra parameter. A suitable linearization of the camera model and of the epipolar constraint is developed in order to arrive at a quadratic eigenvalue problem for which efficient algorithms are known. The lens calibration is done from automatically established image correspondences only. Besides rigidity, no assumptions about the scene are made (e.g. presence of a calibration object). We demonstrate the method in experiments with Nikon FC-E8 fish-eye converter for COOLPIX digital camera. In practical situations, the proposed method allows to incorporate the new omnidirectional camera model into RANSAC - a robust estimation technique.