We introduce a method that simultaneously learns to explore new large environments and to reconstruct them in 3D from color images only. This is closely related to the Next Best View problem (NBV), where one has to identify where to move the camera next to improve the coverage of an unknown scene. However, most of the current NBV methods rely on depth sensors, need 3D supervision and/or do not scale to large scenes. Our method requires only a color camera and no 3D supervision. It simultaneously learns in a self-supervised fashion to predict a “volume occupancy field” from color images and, from this field, to predict the NBV. Thanks to this approach, our method performs well on new scenes as it is not biased towards any training 3D data. We demonstrate this on a recent dataset made of various 3D scenes and show it performs even better than recent methods requiring a depth sensor, which is not a realistic assumption for outdoor scenes captured with a flying drone.

Neural implicit surfaces have become an important technique for multi-view 3D reconstruction but their accuracy remains limited. In this paper, we argue that this comes from the difficulty to learn and render high frequency textures with neural networks. We thus propose to add to the standard neural rendering optimization a direct photo-consistency term across the different views. Intuitively, we optimize the implicit geometry so that it warps views on each other in a consistent way. We demonstrate that two elements are key to the success of such an approach: (i) warping entire patches, using the predicted occupancy and normals of the 3D points along each ray, and measuring their similarity with a robust structural similarity (SSIM); (ii) handling visibility and occlusion in such a way that incorrect warps are not given too much importance while encouraging a reconstruction as complete as possible. We evaluate our approach, dubbed NeuralWarp, on the standard DTU and EPFL benchmarks and show it outperforms state of the art unsupervised implicit surfaces reconstructions by over 20% on both datasets. Our code is available at https://github.com/fdarmon/NeuralWarp

Next Best View computation (NBV) is a long-standing problem in robotics, and consists in identifying the next most informative sensor position(s) for reconstructing a 3D object or scene efficiently and accurately. Like most current methods, we consider NBV prediction from a depth sensor like Lidar systems. Learning-based methods relying on a volumetric representation of the scene are suitable for path planning, but have lower accuracy than methods using a surface-based representation. However, the latter do not scale well with the size of the scene and constrain the camera to a small number of poses. To obtain the advantages of both representations, we show that we can maximize surface metrics by Monte Carlo integration over a volumetric representation. In particular, we propose an approach, SCONE, that relies on two neural modules: The first module predicts occupancy probability in the entire volume of the scene. Given any new camera pose, the second module samples points in the scene based on their occupancy probability and leverages a self-attention mechanism to predict the visibility of the samples. Finally, we integrate the visibility to evaluate the gain in surface coverage for the new camera pose. NBV is selected as the pose that maximizes the gain in total surface coverage. Our method scales to large scenes and handles free camera motion: It takes as input an arbitrarily large point cloud gathered by a depth sensor as well as camera poses to predict NBV. We demonstrate our approach on a novel dataset made of large and complex 3D scenes.

Deep multi-view stereo (MVS) methods have been developed and extensively compared on simple datasets, where they now outperform classical approaches. In this paper, we ask whether the conclusions reached in controlled scenarios are still valid when working with Internet photo collections. We propose a methodology for evaluation and explore the influence of three aspects of deep MVS methods: network architecture, training data, and supervision. We make several key observations, which we extensively validate quantitatively and qualitatively, both for depth prediction and complete 3D reconstructions. First, complex unsupervised approaches cannot train on data in the wild. Our new approach makes it possible with three key elements: upsampling the output, softmin based aggregation and a single reconstruction loss. Second, supervised deep depthmap-based MVS methods are state-of-the art for reconstruction of few internet images. Finally, our evaluation provides very different results than usual ones. This shows that evaluation in uncontrolled scenarios is important for new architectures.

We tackle the problem of finding accurate and robust keypoint correspondences between images. We propose a learning-based approach to guide local feature matches via a learned approximate image matching. Our approach can boost the results of SIFT to a level similar to state-of-the-art deep descriptors, such as Superpoint, ContextDesc, or D2-Net and can improve performance for these descriptors. We introduce and study different levels of supervision to learn coarse correspondences. In particular, we show that weak supervision from epipolar geometry leads to performances higher than the stronger but more biased point level supervision and is a clear improvement over weak image level supervision. We demonstrate the benefits of our approach in a variety of conditions by evaluating our guided keypoint correspondences for localization of internet images on the YFCC100M dataset and indoor images on the SUN3D dataset, for robust localization on the Aachen day-night benchmark and for 3D reconstruction in challenging conditions using the LTLL historical image data.

Usual Structure-from-Motion (SfM) techniques require at least trifocal overlaps to calibrate cameras and reconstruct a scene. We consider here scenarios of reduced image sets with little overlap, possibly as low as two images at most seeing the same part of the scene. We propose a new method, based on line coplanarity hypotheses, for estimating the relative scale of two independent bifocal calibrations sharing a camera, without the need of any trifocal information or Manhattan-world assumption. We use it to compute SfM in a chain of up-to-scale relative motions. For accuracy, we however also make use of trifocal information for line and/or point features, when present, relaxing usual trifocal constraints. For robustness to wrong assumptions and mismatches, we embed all constraints in a parameterless RANSAC-like approach. Experiments show that we can calibrate datasets that previously could not, and that this wider applicability does not come at the cost of inaccuracy.

Stereoscopic reconstruction is important to automatic vision systems. As an intermediate step, estimating this reconstruction is not enough for good performance of the whole system, and its uncertainty must be characterized. Several methods propose uncertainty indexes based on specific data features, thus incomplete, while others are based on learning. We propose a simple index, named ambiguity index, taking into account both data and regularization, and derived directly from the optimization process. Exploiting properties of dynamic programming, this index is related to the posterior variance of the solution when the Semi-Global Matching (SGM) algorithm is used for stereo reconstruction. To illustrate its interest, improvements in refining stereo reconstruction are shown on the KITTI datasets when the index is used.

In this paper, we propose a dense two-frame stereo algorithm which handles occlusion in a variational framework. Our method is based on a new regularization model which includes both a constraint on the occlusion width and a visibility constraint in nonoccluded areas. The minimization of the resulting energy functional is done by convex relaxation. A post-processing then detects and fills the occluded regions. We also propose a novel dissimilarity measure that combines color and gradient comparison with a variable respective weight, to benefit from the robustness of the comparison based on local variations while avoiding the fattening effect it may generate.

This paper addresses the question of identifying the right camera direct or inverse distortion model, permitting a high subpixel precision to fit to real camera distortion. Five classic camera distortion models are reviewed and their precision is compared for direct or inverse distortion. By definition, the three radially symmetric models can only model a distortion radially symmetric around some distortion center. They can be extended to deal with non-radially symmetric distortions by adding tangential distortion components, but still may be too simple for very accurate modeling of real cameras. The polynomial and the rational models instead miss a physical or optical interpretation, but can cope equally with radially and non-radially symmetric distortions. Indeed, they do not require the evaluation of a distortion center. When requiring high precisions, we found that the distortion modeling must also be evaluated primarily as a numerical problem. Indeed, all models except the polynomial involve a non-linear minimization, which increases the numerical risk. The estimation of a polynomial distortion model leads instead to a linear problem, which is secure and much faster. We concluded by extensive numerical experiments that, although high degree polynomials were required to reach a high precision of 1/100 pixels, such polynomials were easily estimated and produced a precise distortion modeling without overfitting. Our conclusion is validated by three independent experimental setups: the models were compared first on the lens distortion database of the Lensfun library by their distortion simulation and inversion power; second by fitting real camera distortions estimated by a non parametric algorithm; and finally by the absolute correction measurement provided by the photographs of tightly stretched strings, warranting a high straightness.

We propose a multiscale extension of a well-known line segment detector, LSD. We show that its multiscale nature makes it much less prone to over-segmentation, more robust to low contrast and less sensitive to noise, while keeping the parameterless advantage of LSD and still being fast. Moreover, we show that in scenes with little or no feature points, but where it is however possible to perform structure from motion from matched line segments, the accuracy is significantly improved. This provides an objective and automatic quantitative assessment of our detector that goes much beyond the usual qualitative visual inspection found in the literature.

We evaluate and improve the matching precision of the SIFT method [1], defined as the root mean square error (RMSE) under a ground truth geometric transform. We first argue that the matching precision reflects to some extent the average relative localization precision between two images. For scale invariant feature detectors like SIFT, we show that the matching precision decreases with the scale of the keypoints, and that this is caused by the scale space sub-sampling in SIFT. We verify that canceling this sub-sampling therefore improves drastically the matching precision. Yet, in case of scale change, this improvement is marginal due to the coarse scale quantization in the scale space. A more sophisticated method is therefore also proposed to improve the matching precision even in case of scale change. This incremented precision is a key ingredient in many important image processing tasks requiring the best precision, such as registration, stitching, and camera calibration.

This paper introduces a topological approach to local invariant feature detection motivated by Morse theory. We use the critical points of the graph of the intensity image, revealing directly the topology information as initial interest points. Critical points are selected from what we call a tree-based shape-space. In particular, they are selected from both the connected components of the upper level sets of the image (the Max-tree) and those of the lower level sets (the Min-tree). They correspond to specific nodes on those two trees: 1) to the leaves (extrema) and 2) to the nodes having bifurcation (saddle points). We then associate to each critical point the largest region that contains it and is topologically equivalent in its tree. We call such largest regions the tree-based Morse regions (TBMRs). The TBMR can be seen as a variant of maximally stable extremal region (MSER), which are contrasted regions. Contrarily to MSER, TBMR relies only on topological information and thus fully inherit the invariance properties of the space of shapes (e.g., invariance to affine contrast changes and covariance to continuous transformations). In particular, TBMR extracts the regions independently of the contrast, which makes it truly contrast invariant. Furthermore, it is quasi-parameter free. TBMR extraction is fast, having the same complexity as MSER. Experimentally, TBMR achieves a repeatability on par with state-of-the-art methods, but obtains a significantly higher number of features. Both the accuracy and robustness of TBMR are demonstrated by applications to image registration and 3D reconstruction.

Multi-view structure from motion (SfM) estimates the position and orientation of pictures in a common 3D coordinate frame. When views are treated incrementally, this external calibration can be subject to drift, contrary to global methods that distribute residual errors evenly. We propose a new global calibration approach based on the fusion of relative motions between image pairs. We improve an existing method for robustly computing global rotations. We present an efficient a contrario trifocal tensor estimation method, from which stable and precise translation directions can be extracted. We also define an efficient translation registration method that recovers accurate camera positions. These components are combined into an original SfM pipeline. Our experiments show that, on most datasets, it outperforms in accuracy other existing incremental and global pipelines. It also achieves strikingly good running times: it is about 20 times faster than the other global method we could compare to, and as fast as the best incremental method. More importantly, it features better scalability properties.

Plumb line lens distortion correction methods permit to avoid numerical compensation between the camera internal and external parameters in global calibration method. Once the distortion has been corrected by a plumb line method, the cam era is ensured to transform, up to the distortion precision, 3D straight lines into 2D straight lines, and therefore becomes a pinhole camera. This paper introduces a plumb line method for correcting and evaluating camera lens distortion with high precision. The evaluation criterion is defined as the average standard deviation from straightness of a set of approximately equally spaced straight strings photographed uniformly in all directions by the camera, so that their image crosses the whole camera field. The method uses an easily built "calibration harp," namely a frame on which good quality strings have been tightly stretched to ensure a very high physical straight ness. Real experiments confirm that our method produces high precision corrections (less than 0.05 pixel), approximating the distortion with a large number of degrees of freedom given by a polynomial model of order eleven.

International audience

This paper presents an image processing algorithm simulating a sub-pixel evolution of an image by mean curvature motion or by affine curvature motion. The sub-pixel algorithm computes the image curvature directly on the smoothed level lines, and yields a microscopic visualization of the curvature map revealing many image details, and getting rid of aliasing effects. This “curvature microscope” showing curvatures in false colors runs on line on any image proposed by users at http://www.ipol.im/pub/algo/cmmm_image_curvature_microscope/.

This paper points out and attempts to remedy a serious discrepancy in results obtained by global calibration methods: The re-projection error can be rendered very small by these methods, but we show that the optical distortion correction is far less accurate. This discrepancy can only be explained by internal error compensations in the global methods that leave undetected the inadequacy of the distortion model. This fact led us to design a model-free distortion correction method where the distortion can be any image domain diffeomorphism. The obtained precision compares favorably to the distortion given by state of the art global calibration and reaches a RMSE of 0.08 pixels. Nonetheless, we also show that this accuracy can still be improved.

This paper presents a software-based parameter-free method for the demultiplexing of a video stream (L. Rudin et al., 2004) that is missing camera labeling information. The method is based on the observation that frames coming from the same input camera share some common characteristic features. These features are extracted from the input frames and grouped together according to statistical criteria. As a result of this grouping the number of different input sources in the video stream is inferred and it is possible to ascertain the source for each frame.

We present a new method of photogrammetry, requiring no direct measurements in the observed scene, no availability of the original camera taking the snapshot and no knowledge of its position. The practical setup involves minimal human intervention, with only the datum of another snapshot (which may not be simultaneous to the first one, i.e. can be temporary anterior or posterior) of the environment from a calibrated camera. This largely automatic configuration permits to eliminate usual points measurements data acquisition step (e.g. in forensic photogrammetry), and to get accurate lengths, angles, and areas measurements in the original scene. The crucial observation is that the two snapshots, even if not simultaneous and from different cameras, provide a stereo system. Therefore the correspondence of 8 points in the views gives the epipolar geometry of the stereo setup, and as one of the camera is calibrated, the calibration can be "transferred" to the other camera through the epipolar matrix. This transfer yields a calibration of the original camera (internal parameters and position in the scene) even if it is not available any more, its settings have changed, its orientation is different or it was moved. A related to this "automatic" photogrammetry application is ability to compare relative size of objects captured by different non-co-located cameras at different times, but at the same general scene location.

This paper sets out a new representation of an image which is contrast independent. The image is decomposed into a tree of "shapes" based on connected components of level sets, which provides a full and nonredundant representation of the image. A fast algorithm to compute the tree, the fast level lines transform (FLLT), is explained in detail. Some simple and direct applications of this representation are shown.

We propose a method for image registration which seems to be useful under the three following conditions. First, both images are globally and roughly the result of a translation and rotation. Second, some occlusions due to moving objects occur from image 1 to image 2. Third, because of changes of illumination, contrast may have changed globally and even locally. Under such unfavorable conditions, correlation-based global registration may become inaccurate, because of the global compromise it yields between several displacements. Our method avoids these difficulties by defining a set of local contrast invariant features in order to achieve contrast invariant matching. A voting procedure allows one to eliminate "wrong" matching features due to the displacement of small objects and yields sub-pixel accuracy. This method was tested successfully for registration of watches with moving hands and for road control applications.