This paper presents the first 3D reconstruction system using unsynchronized and helmet-held consumer cameras, without the use of a calibration pattern. Our assumptions are easy to meet in practice: the cameras have the same setting (frequency, image resolution, field-of-view, roughly equiangular). First, the time offsets between cameras are estimated without accurate calibration as input. Second, both inter-camera rotations and intrinsic parameters are refined using structure-from-motion and bundle adjustment. We experiment both synchronization and self-calibration on four GoPro cameras mounted on a helmet, such that the resulting multi-camera is assumed to be central and provides a 360 degree field-of-view in the horizontal plane. A surface is also estimated from a multi-camera video acquired by walking in a city.

In the recent years, a family of 2-manifold surface reconstruction methods from a sparse Structure-from-Motion points cloud based on 3D Delaunay triangulation was developed. This family consists of batch and incremental variations which include a step that remove visual artifacts. Although been necessary in the term of surface quality, this step is slow compared to the other parts of the algorithm and is not well suited to be used in an incremental manner. In this paper, we present two other methods for removing visual artifacts. They are evaluated and compared to the previous one in the incremental context where the need of new methods is the highest. Taken separately, they provide medium results, but used together they are as good as the old method in the terms of surface quality, and at the same time, processing time is almost three times smaller.

International audience

Two problems occur when bundle adjustment (BA) is applied on long image sequences: large calculation time and drift (or error accumulation). In recent work, the calculation time is reduced by local BAs applied in an incremental scheme. The drift may be reduced by fusion of GPS and Structure-from-Motion. An existing fusion method is BA minimizing a weighted sum of image and GPS errors. This paper introduces two constrained BAs for fusion which enforce an upper bound for the reprojection error. These BAs are alternatives to the existing fusion BA which does not guarantee a small reprojection error and requires a weight as input. Then, the three fusion BAs are integrated in an incremental Structure-from-Motion method based on local BA. Last, we will compare the fusion results on long monocular image sequences and low cost GPS.

Two problems occur when bundle adjustment (BA) is applied on long image sequences: the large calculation time and the drift (or error accumulation). In recent work, the calculation time is reduced by local BAs applied in an incremental scheme. The drift may be reduced by fusion of GPS and Structure-from-Motion. An existing fusion method is BA minimizing a weighted sum of image and GPS errors. This paper introduces two constrained BAs for fusion, which enforce an upper bound for the reprojection error. These BAs are alternatives to the existing fusion BA, which does not guarantee a small reprojection error and requires a weight as input. Then the three fusion BAs are integrated in an incremental Structure-from-Motion method based on local BA. Lastly, we will compare the fusion results on a long monocular image sequence and a low cost GPS.

Local Bundle Adjustments were recently introduced for visual SLAM (Simultaneous Localization and Mapping). In Monocular Visual SLAM, the scale factor is not observable and the reconstruction scale drifts as time goes by. On long trajectory, this problem makes absolute localisation not usable. To overcome this major problem, data fusion is a possible solution. In this paper, we describe Weighted Local Bundle Adjustment(WLBA) for monocular visual SLAM purposes. We show that W-LBA used with local covariance gives better results than Local Bundle Adjustment especially on the scale propagation. Moreover W-LBA is well designed for sensor fusion. Since odometer is a common sensor and is reliable to obtain a scale information, we apply W-LBA to fuse visual SLAM with odometry data. The method performance is shown on a large scale sequence.

The Simultaneous Localisation And Mapping (SLAM) for a camera moving in a scene is a long term research problem. Here we improve a recent visual SLAM which applies Local Bundle Adjustments (LBA) on selected key-frames of a video: we show how to correct the scale drift observed in long monocular video sequence using an additional odometry sensor. Our method and results are interesting for several reasons: (1) the pose accuracy is improved on real examples (2) we do not sacrifice the consistency between the reconstructed 3D points and image features to fit odometry data (3) the modification of the original visual SLAM method is not difficult.

Local bundle adjustment (LBA) has recently been introduced to estimate the geometry of image sequences taken by a calibrated camera. Its advantage over standard (global) bundle adjustment is a great reduction of computational complexity, which allows real-time performances with a similar accuracy. However, no confidence measure on the LBA result such as uncertainty or covariance has yet been introduced. This paper introduces statistical models and estimation methods for uncertainty with two desirable properties: (1) uncertainty propagation along the sequence and (2) real-time calculation. We also explain why this problem is more complicated than it may appear at first glance, and we provide results on video sequences.

We propose an automatic approach to generate street-side 3D photo-realistic models from images captured along the streets at ground level. We first develop a multi-view semantic segmentation method that recognizes and segments each image at pixel level into semantically meaningful areas, each labeled with a specific object class, such as building, sky, ground, vegetation and car. A partition scheme is then introduced to separate buildings into independent blocks using the major line structures of the scene. Finally, for each block, we propose an inverse patch-based orthographic composition and structure analysis method for façade modeling that efficiently regularizes the noisy and missing reconstructed 3D data. Our system has the distinct advantage of producing visually compelling results by imposing strong priors of building regularity. We demonstrate the fully automatic system on a typical city example to validate our methodology.

The automatic reconstruction of 3D models from image sequences is still a very active field of research. All existing methods are designed for a given camera model, and a new (and ambitious) challenge is 3D modeling with a method which is exploitable for any kind of camera. A similar approach was recently suggested for structure-from-motion thanks to the use of generic camera models. In this paper, we first introduce geometric tools designed for 3D scene modeling with a generic camera model. Then, these tools are used to solve many issues: matching errors, wide range of point depths, depth discontinuities, and view-point selection for reconstruction. Experiments are provided for perspective and catadioptric cameras.

International audience

Fully automatic 3D modeling from a catadioptric image sequence has rarely been addressed until now, although this is a long-standing problem for perspective images. All previous catadioptric approaches have been limited to dense reconstruction for a few view points, and the majority of them require calibration of the camera. This paper presents a method which deals with hundreds of images, and does not require precise calibration knowledge. In this context, the same 3D point of the scene may be visible and reconstructed in a large number of images at very different accuracies. So the main part of this paper concerns the selection of reconstructed points, a problem largely ignored in previous works. Summaries of the structure from motion and dense stereo steps are also given. Experiments include the 3D model reconstruction of indoor and outdoor scenes, and a walkthrough in a city.

In this paper we describe a method that estimates the motion of a calibrated camera (settled on an experimental vehicle) and the tridimensional geometry of the environment. The only data used is a video input. In fact, interest points are tracked and matched between frames at video rate. Robust estimates of the camera motion are computed in real-time, key-frames are selected and permit the features 3D reconstruction. The algorithm is particularly appropriate to the reconstruction of long images sequences thanks to the introduction of a fast and local bundle adjustment method that ensures both good accuracy and consistency of the estimated camera poses along the sequence. It also largely reduces computational complexity compared to a global bundle adjustment. Experiments on real data were carried out to evaluate speed and robustness of the method for a sequence of about one kilometer long. Results are also compared to the ground truth measured with a differential GPS.

Many methods exist for the automatic and optimal 3D reconstruction of camera motion and scene structure from image sequence ('structure from motion' or SfM). The solution to this problem is not unique: we obtain an other solution by changing the coordinate system where points and cameras are defined. On the other hand, existing methods provide some measure of confidence or uncertainty for the estimation when the ground truth is not available, using a gauge constraint which settle the reconstruction coordinate system. Here we justify and describe a method which estimate the uncertainty ellipsoids, when previous methods are not straightforward to use due to a huge number of parameters to fit. Many examples are given and discussed for big reconstructions

This paper introduces an incremental method for "structure from motion" of complex scenes from a video sequence. More precisely, we estimate the 3D positions of the viewed points in images and the camera positions and orientations through the sequence. The method can be seen as a fast but accurate alternative to classical reconstruction methods that use bundle adjustment, and that can become slow and computation time expensive for very long scenes. Our results are compared to the reconstruction obtained by the classical hierarchical bundle adjustment method. They have also been successfully used as a reference sequence for the vision based localization of an autonomous mobile robot

In this paper we present a method for computing the localization of a mobile robot with reference to a learning video sequence. The robot is first guided on a path by a human, while the camera records a monocular learning sequence. Then a 3D reconstruction of the path and the environment is computed off line from the learning sequence. The 3D reconstruction is then used for computing the pose of the robot in real time (30 Hz) in autonomous navigation. Results from our localization method are compared to the ground truth measured with a differential GPS.

In this paper, a complete system for outdoor robot navigation is presented. It uses only monocular vision. The robot is first guided on a path by a human. During this learning step, the robot records a video sequence. From this sequence, a three dimensional map of the trajectory and the environment is built. When this map has been computed, the robot is able to follow the same trajectory by itself. Experimental results carried out with an urban electric vehicle are shown and compared to the ground truth.

This paper proposes a quasi-dense approach to 3D surface model acquisition from uncalibrated images. First, correspondence information and geometry are computed based on new quasi-dense point features that are resampled subpixel points from a disparity map. The quasi-dense approach gives more robust and accurate geometry estimations than the standard sparse approach. The robustness is measured as the success rate of full automatic geometry estimation with all involved parameters fixed. The accuracy is measured by a fast gauge-free uncertainty estimation algorithm. The quasi-dense approach also works for more largely separated images than the sparse approach, therefore, it requires fewer images for modeling. More importantly, the quasidense approach delivers a high density of reconstructed 3D points on which a surface representation can be reconstructed. This fills the gap of insufficiency of the sparse approach for surface reconstruction, essential for modeling and visualization applications. Second, surface reconstruction methods from the given quasi-dense geometry are also developed. The algorithm optimizes new unified functionals integrating both 3D quasi-dense points and 2D image information, including silhouettes. Combining both 3D data and 2D images is more robust than the existing methods using only 2D information or only 3D data. An efficient bounded regularization method is proposed to implement the surface evolution by level-set methods. Its properties are discussed and proven for some cases. As a whole, a complete automatic and practical system of 3D modeling from raw images captured by hand-held cameras to surface representation is proposed. Extensive experiments demonstrate the superior performance of the quasi-dense approach with respect to the standard sparse approach in robustness, accuracy, and applicability.

In this paper we present a method for computing the localization of a mobile robot with reference to a learning video sequence. The robot is first guided on a path by a human, while the camera records a monocular learning sequence. Then the computer builds a map of the environment. This is done by first extracting key frames from the learning sequence. Then the epipolar geometry and camera motion are computed between key frames. Additionally a hierarchical bundle adjustment is used to refine the reconstruction. The map stored for the localization include the position of the camera associated with each key frame as well as a set of interest points detected in the images and reconstructed in 3D. Using this map it is possible to compute the localization of the robot in real time during the automatic driving phase. 1

The general approach for object modeling is to construct the surface from the high-quality range points obtained from laser scanners. In this paper, we face the noise point cloud obtained from image sequences by a common camera and develop a novel algorithm of adaptive multi-resolution fitting (AMRF) for object modeling. This algorithm combines the adaptive subdivision scheme with multi-resolution fitting so that the control model is subdivided locally and adaptively according to the local complexity of the point cloud and approximates the 3D data level by level. The proposed method can conquer the holes and outliers efficiently and create full compatibility between the complexity of the mesh model and the representation of the local details. We apply the proposed method to the complete head modeling with the real data, and the results seem very promising.

Surface representation is needed for almost all modeling and visualization applications, but unfortunately, 3D data from a passive vision system are often insufficient for a traditional surface reconstruction technique that is designed for densely scanned 3D point data. In this paper, we develop a new method for surface reconstruction by combining both 3D data and 2D image information. The silhouette information extracted from 2D images can also be integrated as an option if it is available. The new method is a variational approach with a new functional integrating 3D stereo data with 2D image information. This gives a more robust approach than existing methods using only pure 2D information or 3D stereo data. We also propose a bounded regularization method to implement efficiently the surface evolution by level-set methods. The properties of the algorithms are discussed, proved for some cases, and empirically demonstrated through intensive experiments on real sequences.

The creation of novel views using prestored images or image-based rendering has many potential applications, such as visual simulation, virtual reality, and telepresence, for which traditional computer graphics based on geometric modeling would be unsatisfactory particularly with very complex three-dimensional scenes. This paper presents a new image-based rendering system that tackles the two most difficult problems of image-based modeling: pixel matching and visibility handling. We first introduce the joint view triangulation (JVT), a novel representation for pairs of images that handles the visibility and occlusion problems created by the parallaxes between the images. The JVT is built from matched planar patches regularized by local smooth constraints encoded by plane homographies. Then, we introduce an incremental edge-constrained construction algorithm. Finally, we present a pseudo-painter's rendering algorithm for the JVT and demonstrate the performance of these methods experimentally.

This paper presents a quasi-dense matching algorithm between images based on the match propagation principle. The algorithm starts from a set of sparse seed matches, then propagates to the neighboring pixels by the best-first strategy, and produces a quasi-dense disparity map. The quasi-dense matching aims at broad modeling and visualization applications which rely heavily on matching information. Our algorithm is robust to initial sparse match outliers due to the best-first strategy. It is efficient in time and space as it is only output sensitive. It handles half-occluded areas because of the simultaneous enforcement of newly introduced discrete 2D gradient disparity limit and the uniqueness constraint. The properties of the algorithm are discussed and empirically demonstrated. The quality of quasi-dense matching are validated through intensive real examples.

Relief mosaics are collections of registered images that extend traditional mosaics by supporting motion parallax. A simple parallax interpolation algorithm based on computed correspondence information allows high quality blur-free and ghost-free mosaics to be created using images from moving hand-held cameras that would not be suitable for traditional mosaicing. The renderer can also display local parallax changes, giving a local but visually convincing illusion of depth. Moreover, relief mosaics can be used for approximate plenoptic modeling from hand-held cameras at lower spatial sampling rates than existing light-field methods. We present a fully automatic correspondence based construction system for relief mosaics, and show how they can be used in applications.

General SFM methods give poor results for images captured by constrained motions such as planar motion of concentric mosaics (CM). In this paper we propose new SFM algorithms for both images captured by CM and composite mosaic images from CM. We first introduce ID affine camera model for completing 1D camera models. Then we show that a 2D image captured by CM can be decoupled into two 1D images: one 1D projective and one ID affine; a composite mosaic image can by rebinned into a calibrated ID panorama projective camera. Finally we describe subspace reconstruction methods and demonstrate both in theory and experiments the advantage of the decomposition method over the general SFM methods by incorporating the constrained motion into the earliest stage of motion analysis.

Image-based-interpolation creates smooth and photo-realistic views between two view points. The concept of joint view triangulation (JVT) has been proven to be an efficient multi-view representation to handle visibility issue. However, the existing JVT built only on a regular sampling grid, often produces undesirable artifacts for artificial objects. To tackle these problems, a new edge-constrained joint view triangulation is developed in this paper to integrate contour points and artificial rectilinear objects as triangulation constraints. Also a super-sampling technique is introduced to refine visible boundaries. The new algorithm is successfully demonstrated on many real image pairs.

A new robust dense matching algorithm is introduced. The algorithm starts from matching the most textured points, then a match propagation algorithm is developed with the best first strategy to dense matching. Next, the matching map is regularised by using the local geometric constraints encoded by planar affine applications and by using the global geometric constraint encoded by the fundamental matrix. Two most distinctive features are a match propagation strategy developed by analogy to region growing and a successive regularisation by local and global geometric constraints. The algorithm is efficient, robust and can cope with wide disparity. The algorithm is demonstrated on many real image pairs, and applications on image interpolation and a creation of novel views are also presented.

Creating novel views by interpolating prestored images or view morphing has many applications in visual simulation. We present in this paper a new method of automatically interpolating two images which tackles two most difficult problems of morphing due to the lack of depth informational pixel matching and visibility handling. We first describe a quasi-dense matching algorithm based on region growing with the best first strategy for match propagation. Then, we describe a robust construction of matched planar patches using local geometric constraints encoded by a homography. After that we introduce a novel representation, joint view triangulation, for visible and half-occluded patches in two images to handle their visibility during the creation of new view. Finally we demonstrate these techniques on real image pairs.

: We describe a simple and efficient dense matching method based on region growing techniques, which can be applied to a wide range of globally textured images. Our method can deal with non-rigid scenes and large camera motions. First a few highly distinctive features like points or areas are extracted and matched. These initial matches are then used in a correlation-based region growing step which propagates the matches in textured and more ambiguous regions of the images. The implementation of the algorithm is also given and is demonstrated on both synthetic and real image pairs. Key-words: Dense Matching, Region Growing, Correlation (R'esum'e : tsvp) Unite de recherche INRIA Rhone-Alpes 655, avenue de l'Europe, 38330 MONTBONNOT ST MARTIN (France) Telephone : 04 76 61 52 00 - International: +33 4 76 61 52 00 Telecopie : 04 76 61 52 52 - International: +33 4 76 61 52 Appariement dense d'images textur'ees par croissance de r'egions R'esum'e : Un algorithme simple et rapide d'appari...