This paper introduces the Meshroom software and its underlying 3D computer vision framework AliceVision. This solution provides a photogrammetry pipeline to reconstruct 3D scenes from a set of unordered images. It also features other pipelines for fusing multi-bracketing low dynamic range images into high dynamic range, stitching multiple images into a panorama and estimating the motion of a moving camera. Meshroom's node-graph architecture allows the user to customize the different pipelines to adjust them to their domain specific needs. The user can interactively add other processing nodes to modify a pipeline, export intermediate data to analyze the result of the algorithms and easily compare the outputs given by different sets of parameters. The software package is released in open source and relies on open file formats. These features enable researchers to conveniently run the pipelines, access and visualize the data at each step, thus promoting the sharing and the reproducibility of the results.

In the context of 2D/3D registration, this paper introduces an approach that allows for matching features detected in two different modalities, photographs, and 3D models, by using a common 2D representation. More precisely, 2D images are matched with a set of depth images representing the 3D model. After introducing the concept of Curvilinear Saliency, which is related to curvature estimation, we propose a new ridge and valley detector for depth images rendered from 3D models. A variant of this detector is adapted to photographs, first by considering multi-scale features and second by integrating the focus curve principle. Finally, a registration algorithm determines the correct view of the 3D model and, thus, the pose of the photograph. This approach relies on the Histogram of Curvilinear Saliency (HCS), an adaptation of the Histogram of Oriented Gradients (HOG) to the proposed features in 2D and 3D. The presented results highlight both the quality of the features detected in terms of repeatability and the interest of the approach for registration and pose estimation.

We propose a line segment and elliptical arc detector that produces a reduced number of false detections on various types of images without any parameter tuning. For a given region of pixels in a grey-scale image, the detector decides whether a line segment or an elliptical arc is present (model validation). If both interpretations are possible for the same region, the detector chooses the one that best explains the data (model selection ). We describe a statistical criterion based on the a contrario theory, which serves for both validation and model selection. The experimental results highlight the performance of the proposed approach compared to state-of-the-art detectors, when applied on synthetic and real images.

Using fiducial markers ensures reliable detection and identification of planar features in images. Fiducials are used in a wide range of applications, especially when a reliable visual reference is needed, e.g., to track the camera in cluttered or textureless environments. A marker designed for such applications must be robust to partial occlusions, varying distances and angles of view, and fast camera motions. In this paper, we present a robust, highly accurate fiducial system, whose markers consist of concentric rings, along with its theoretical foundations. Relying on projective properties, it allows to robustly localize the imaged marker and to accurately detect the position of the image of the (common) circle center. We demonstrate that our system can detect and accurately localize these circular fiducials under very challenging conditions and the experimental results reveal that it outperforms other recent fiducial systems.

Object recognition based on local features computed at multiple locations is robust to occlusions, strong viewpoint changes and object deformations. These features should be repeatable, precise and distinctive. We present an operator for repeatable feature detection on depth images (relative to 3D models) as well as 2D intensity images. The proposed detector is based on estimating the curviness saliency at multiple scales in each kind of image. We also propose quality measures that evaluate the repeatability of the features between depth and intensity images. The experiments show that the proposed detector outperforms both the most powerful, classical point detectors (e.g., SIFT) and edge detection techniques.

We consider an emerging situation where multiple cameras are filming the same event simultaneously from a diverse set of angles. The captured videos provide us with the multiple view geometry and an understanding of the 3D structure of the scene. We further extend this understanding by introducing the concept of 3D interest map in this paper. As most users naturally film what they find interesting from their respective viewpoints, the 3D structure can be annotated with the level of interest, naturally crowdsourced from the users. A 3D interest map can be understood as an extension of saliency maps in the 3D space that captures the semantics of the scene. We evaluate the idea of 3D interest maps on two real datasets, taken from the environment or the cameras that are equipped enough to have an estimation of the poses of cameras and a reasonable synchronization between them. We study two aspects of the 3D interest maps in our evaluation. First, by projecting them into 2D, we compare them to state-of-the-art saliency maps. Second, to demonstrate the usefulness of the 3D interest maps, we apply them to a video mashup system that automatically produces an edited video from one of the datasets.

High-resolution panoramic video with a wide field-of-view is popular in many contexts. However, in many examples, like surveillance and sports, it is often desirable to zoom and pan into the generated video. A challenge in this respect is real-time support, but in this demo, we present an end-to-end real-time panorama system with interactive zoom and panning. Our system installed at Alfheim stadium, a Norwegian premier league soccer team, generates a cylindrical panorama from five 2K cameras live where the perspective is corrected in real-time when presented to the client. This gives a better and more natural zoom compared to existing systems using perspective panoramas and zoom operations using plain crop. Our experimental results indicate that virtual views can be generated far below the frame-rate threshold, i.e., on a GPU, the processing requirement per frame is about 10 milliseconds.

Panorama video is becoming increasingly popular, and we present an end-to-end real-time system to interactively zoom and pan into high-resolution panoramic videos. Compared to existing systems using perspective panoramas with cropping, our approach creates a cylindrical panorama. Here, the perspective is corrected in real-time, and the result is a better and more natural zoom. Our experimental results also indicate that such zoomed virtual views can be generated far below the frame-rate threshold. Taking into account recent trends in device development, our approach should be able to scale to a large number of concurrent users in the near future.

This work aims at introducing a new unified Structure from Motion (SfM) paradigm in which images of circular point-pairs can be combined with images of natural points. An imaged circular point-pair encodes the 2D Euclidean structure of a world plane and can easily be derived from the image of a planar shape, especially those including circles. A classical SfM method generally runs two steps: first a projective factorization of all matched image points (into projective cameras and points) and second a camera self calibration that updates the obtained world from projective to Euclidean. This work shows how to introduce images of circular points in these two SfM steps while its key contribution is to provide the theoretical foundations for combining "classical" linear self-calibration constraints with additional ones derived from such images. We show that the two proposed SfM steps clearly contribute to better results than the classical approach. We validate our contributions on synthetic and real images.

A C2Tag refers to a set of concentric circles of different radii. C2Tags have been recently introduced in computer vision, in particular for camera calibration, as they offer highly interesting photometric and geometric properties, compared to the classical “checkerboard” tags. In this work, we propose the general paradigm of camera tracking based on a planar marker consisting of at least two C2Tags. All the involved steps are described: detection, identification, 2D reconstruction, calibration and 3D reconstruction. Our contribution is to introduce the following missing steps required to deal with long video sequences under time constraints: key-frame selection, bundle adjustment and intermediate pose refinement.

We have at our disposal a large database containing images of various configurations of coplanar circles, randomly laid-out, called “Bubble Tags”. The images are taken from different viewpoints. Given a new image (query image), the goal is to find in the database the image containing the same bubble tag as the query image. We propose representing the images through projective invariant signatures which allow identifying the bubble tag without passing through an Euclidean reconstruction step. This is justified by the size of the database, which imposes the use of queries in 1D/vectorial form, i.e. not in 2D/matrix form. The experiments carried out confirm the efficiency of our approach, in terms of precision and complexity.

This purely theoretical work investigates the problem of artificial singularities in camera self-calibration. Self-calibration allows one to upgrade a projective reconstruction to metric and has a concise and well-understood formulation based on the Dual Absolute Quadric (DAQ), a rank-3 quadric envelope satisfying (nonlinear) ‘spectral constraints’: it must be positive of rank 3. The practical scenario we consider is the one of square pixels, known principal point and varying unknown focal length, for which generic Critical Motion Sequences (CMS) have been thoroughly derived. The standard linear self-calibration algorithm uses the DAQ paradigm but ignores the spectral constraints. It thus has artificial CMSs, which have barely been studied so far. We propose an algebraic model of singularities based on the confocal quadric theory. It allows to easily derive all types of CMSs. We first review the already known generic CMSs, for which any self-calibration algorithm fails. We then describe all CMSs for the standard linear self-calibration algorithm; among those are artificial CMSs caused by the above spectral constraints being neglected. We then show how to detect CMSs. If this is the case it is actually possible to uniquely identify the correct self-calibration solution, based on a notion of signature of quadrics. The main conclusion of this paper is that a posteriori enforcing the spectral constraints in linear self-calibration is discriminant enough to resolve all artificial CMSs.

We present MARCH, a complete solution for enhanced cultural heritage discovery using the mobile phone. Simply point the camera of a mobile device at prehistoric cave engravings. Then MARCH augments the captured images with the expert's drawings, highlighting in real time the animal engravings, which are almost impossible to observe with the naked eye. We have created a mobile augmented reality application which runs at 14 FPS for 320x240 frames on a Nokia N95 smartphone. We describe the optimizations and the requirements needed to obtain these results on mobile devices.

We investigate the problem of finding the metric structure of a general 3D scene viewed by a moving camera with square pixels and constant unknown focal length. While the problem has a concise and well-understood formulation in the stratified framework thanks to the absolute dual quadric, two open issues remain. The first issue concerns the generic Critical Motion Sequences, i.e. camera motions for which self-calibration is ambiguous. Most of the previous work focuses on the varying focal length case. We provide a thorough study of the constant focal length case. The second issue is to solve the nonlinear set of equations in four unknowns arising from the dual quadric formulation. Most of the previous work either does local nonlinear optimization, thereby requiring an initial solution, or linearizes the problem, which introduces artificial degeneracies, most of which likely to arise in practice. We use interval analysis to solve this problem. The resulting algorithm is guaranteed to find the solution and is not subject to artificial degeneracies. Directly using interval analysis usually results in computationally expensive algorithms. We propose a carefully chosen set of inclusion functions, making it possible to find the solution within few seconds. Comparisons of the proposed algorithm with existing ones are reported for simulated and real data.

Plane-based calibration is now a very popular procedure because of its flexibility. One key step consists in detecting a set of coplanar features, from which the Euclidean structure of the corresponding 3D plane has to be computed. We suggest to use confocal conics as calibration targets, as they offer undeniable advantages over other ones (e.g., points or lines) in terms of detection and estimation, especially in the presence of partial occlusion. We introduce important projective and Euclidean properties of the linear family of conics (i.e., the confocal conic range), spanned by two confocal conics. In particular, we rely on the fact that the circular point-envelope - a rank-2 conic that encodes the 2D Euclidean structure - is a degenerate member of any confocal conic range. This allows us to give closed-form solutions in three cases: one conic with known foci, two confocal conics with known product of ratios of semi axes, and two unknown confocal conics. The performances of the proposed algorithms (consisting of a few lines of Matlab-like code) show up high accuracies for both intrinsic and extrinsic camera parameters. In addition to experiments with synthetic data, a video sequence is processed, showing off the interest of using confocal conics as calibration targets, for augmented reality purposes.

We investigate the projective properties of the feature consisting of two concentric circles. We demonstrate there exist geometric and algebraic constraints on its projection. We show how these constraints greatly simplify the recoveries of the affine and Euclidean structures of a 3D plane. As an application, we assess the performances of two camera calibration algorithms.

We consider the problem of camera self-calibration from images of a planar object with unknown Euclidean structure. The general case of possibly varying focal length is addressed. This problem is nonlinear in general. One of our contributions is a nonlinear approach that makes abstraction of the (possibly varying) focal length, resulting in a computationally efficient algorithm. In addition, it does not require a good initial estimate of the focal length, unlike previous approaches. As for the initialization of other parameters, we propose a practical approach that simply requires taking one image in roughly fronto-parallel position. Closed-form solutions for various configurations of unknown intrinsic parameters are provided. Our methods are evaluated and compared to previous approaches using simulated and real images. Besides our practical contributions, we also provide a detailed geometrical interpretation of the principles underlying our approach.

We are concerned with the recovery of the metric structure of the 3D space from the metric structure of a plane, given as the world-plane to image homography. Previous works have pointed out that the metric structure of the 3D space can be defined by a set of three mutually conjugated (vanishing) points with respect to the imaged absolute conic (the vanishing self-polar triangle) and three additional factors, defined up to a scale factor. Given that the homography H only allows the image to inherit two of these points and two of these factors, we show how the (three) unknown quantities can be recovered from H, by investigating some camera constraints which are discussed. We propose two solutions in the cases of (i) of known principal point coordinates (ii) of zero-skew and known pixel aspect ratio. We demonstrate that, under these assumptions, the third vanishing point, associated with the direction orthogonal to the reference plane, lies on a line that we call the central line. Adding an additional constraint, zero-skew for (i) and known camera height for (ii), a direct solution is found. Results on real images have proved to be quite accurate and encouraging for the use of our approach.

The absolute conic is the key for recovering the Euclidean structure in 3D space. The image of the absolute conic can be parametrized by the set of three vanishing points corresponding to orthogonal directions in space with two independent additional factors. Whereas a single plane to image homography only allows the image to inherit two vanishing points and one of these factors, we describe a method which recovers the metric structures of any perpendicular plane to the ground from the ground to image homography matrix. Indeed, under the assumption of a natural camera, we demonstrate that the third vanishing point lies on a line that we call central line. A direct solution is found with an additional constraint relevant to the application we deal with: a "pseudo-parallelism" between one of the camera axis and the ground plane. Direct applications on video MPEG-encoded images are presented in the experiment section showing a very satisfactory accuracy.

This paper shows how to estimate the 3D planar motion of point coordinates from uncalibrated sequence images, irrespective of the camera motion. Assuming that four stationary reference points with known scene coordinates are seen, the reconstruction problem is solved through recursive coordinates estimations in the 3D reference plane while tracking is done in a space of invariant projective parameters instead of affine or Euclidean parameters. The invariant perspective coordinates plane-axis correspond to cross-ratios defining the relative position of a point with respect to four reference points-supplies a matching space invariant for any homography or camera motion. According to the extended Kalman filter formalism, we discuss the discriminatory power of the invariant perspective coordinates and describe geometrical constraints in order to reduce false matches. Experiments are done with a reference rectangle of known Euclidean coordinates. Results show the interest of this method yielding to a high reconstruction accuracy.