We present a comprehensive survey and benchmark of both traditional and learning-based methods for surface reconstruction from point clouds. This task is particularly challenging for real-world acquisitions due to factors such as noise, outliers, non-uniform sampling, and missing data. Traditional approaches often simplify the problem by imposing handcrafted priors on either the input point clouds or the resulting surface, a process that can require tedious hyperparameter tuning. In contrast, deep learning models have the capability to directly learn the properties of input point clouds and desired surfaces from data. We study the influence of handcrafted and learned priors on the precision and robustness of surface reconstruction techniques. We evaluate various time-tested and contemporary methods in a standardized manner. When both trained and evaluated on point clouds with identical characteristics, the learning-based models consistently produce higher-quality surfaces compared to their traditional counterparts-even in scenarios involving novel shape categories. However, traditional methods demonstrate greater resilience to the diverse anomalies commonly found in real-world 3D acquisitions. For the benefit of the research community, we make our code and datasets available, inviting further enhancements to learning-based surface reconstruction. This can be accessed at https://github.com/raphaelsulzer/dsr-benchmark.

Current stereo-vision pipelines produce high accuracy 3D reconstruction when using multiple pairs or triplets of satellite images. However, these pipelines are sensitive to the changes between images that can occur as a result of multidate acquisitions. Such variations are mainly due to variable shadows, reflexions and transient objects (cars, vegetation). To take such changes into account, Neural Radiance Fields (NeRF) have recently been applied to multi-date satellite imagery. However, Neural methods are very compute-intensive, taking dozens of hours to learn, compared with minutes for standard stereo-vision pipelines. Following the ideas of Instant Neural Graphics Primitives we propose to use an efficient sampling strategy and multi-resolution hash encoding to accelerate the learning. Our model, Satellite Neural Graphics Primitives (SAT-NGP) decreases the learning time to 15 minutes while maintaining the quality of the 3D reconstruction.

Recent works on 3D semantic segmentation propose to exploit the synergy between images and point clouds by processing each modality with a dedicated network and projecting learned 2D features onto 3D points. Merging large-scale point clouds and images raises several challenges, such as constructing a mapping between points and pixels, and aggregating features between multiple views. Current methods require mesh reconstruction or specialized sensors to recover occlusions, and use heuristics to select and aggregate available images. In contrast, we propose an end-to-end trainable multi-view aggregation model leveraging the viewing conditions of 3D points to merge features from images taken at arbitrary positions. Our method can combine standard 2D and 3D networks and outperforms both 3D models operating on colorized point clouds and hybrid 2D/3D networks without requiring colorization, meshing, or true depth maps. We set a new state-of-the-art for large-scale indoor/outdoor semantic segmentation on S3DIS (74.7 mIoU 6-Fold) and on KITTI-360 (58.3 mIoU). Our full pipeline is accessible at https://github.com/drprojects/DeepViewAgg, and only requires raw 3D scans and a set of images and poses.

Most current neural networks for reconstructing surfaces from point clouds ignore sensor poses and only operate on point locations. Sensor visibility, however, holds meaningful information regarding space occupancy and surface orientation. In this paper, we present two simple ways to augment point clouds with visibility information, so it can directly be leveraged by surface reconstruction networks with minimal adaptation. Our proposed modifications consistently improve the accuracy of generated surfaces as well as the generalization capability of the networks to unseen domains. Our code, data and pretrained models can be found online: https://github.com/raphaelsulzer/dsrv-data.

3D Geospatial information plays a key role in many soaring sectors such as sustainable and smart cities, climate monitoring, ecological mobility, and economic intelligence. The availability of huge volumes of satellite, airborne and insitu data now makes this production feasible at large scale. It needs nonetheless a certain level of manual intervention to secure the level of quality, which prevents mass production. This paper presents the AI4GEO program that aims at developing an end to end solution to produce automatically qualified 3D Digital model at scale together with multiple layers of information.

We present an out-of-core and distributed surface reconstruction algorithm which scales efficiently on arbitrarily large point clouds (with optical centres) and produces a 3D watertight triangle mesh representing the surface of the underlying scene. Surface reconstruction from a point cloud is a difficult problem and existing state of the art approaches are usually based on complex pipelines making use of global algorithms (i.e. Delaunay triangulation, graph-cut optimisation). For one of these approaches, we investigate the distribution of all the steps (in particular Delaunay triangulation and graph-cut optimisation) in order to propose a fully scalable method. We show that the problem can be tiled and distributed across a cloud or a cluster of PCs by paying a careful attention to the interactions between tiles and using Spark computing framework. We confirm the efficiency of this approach with an in-depth quantitative evaluation and the successful reconstruction of a surface from a very large data set which combines more than 350 million aerial and terrestrial LiDAR points.

Abstract We introduce a novel learning‐based, visibility‐aware, surface reconstruction method for large‐scale, defect‐laden point clouds. Our approach can cope with the scale and variety of point cloud defects encountered in real‐life Multi‐View Stereo (MVS) acquisitions. Our method relies on a 3D Delaunay tetrahedralization whose cells are classified as inside or outside the surface by a graph neural network and an energy model solvable with a graph cut. Our model, making use of both local geometric attributes and line‐of‐sight visibility information, is able to learn a visibility model from a small amount of synthetic training data and generalizes to real‐life acquisitions. Combining the efficiency of deep learning methods and the scalability of energy‐based models, our approach outperforms both learning and non learning‐based reconstruction algorithms on two publicly available reconstruction benchmarks.

3D urban models provide convincing analytic tools for decision making, city planning, and smart city services. However, developing a fully automated method that can produce 3D building models of high quality, fidelity and accuracy is still a challenging task. Currently, most of the proposed approaches handle polyhedral roofs (consisting of planar polygons) because they assume that all roofs in a single area follow this prior. However, the reconstruction method could have its prior adapted to the roof type. In this paper, we are dealing with a specific roof case which is piecewise horizontal roofs which are very frequent in most countries of North Africa and in particular in Tunisia. Our building reconstruction method follows four main steps: building LiDAR points extraction, piecewise horizontal roof clustering, boundary creation and 3D geometric modeling. In order to prove the suitability and the effectiveness of the introduced method, experiments are conducted with real LiDAR data and aerial RGB image.

A key problem for autonomous car navigation is the understanding, at an object level, of the current driving situation. Addressing this issue requires the extraction of meaningful information from on-board stereo imagery by classifying the fundamental elements of urban scenes into semantic categories that can more easily be interpreted and be reflected upon (streets, buildings, pedestrians, vehicles, signs, etc.). A probabilistic method is proposed to fuse a coarse prior 3D map data with stereo imagery classification. A novel fusion architecture based on the Stixel framework is presented for combining semantic pixel-wise segmentation from a convolutional neural network (CNN) with depth information obtained from stereo imagery while integrating coarse prior depth and label information. The proposed approach was tested on a manually labeled data set in urban environments. The results show that the classification accuracy of the fundamental elements composing the urban scene was significantly enhanced by this method compared to what is obtained from the semantic pixel-wise segmentation of a CNN alone.

In the context of urban autonomous navigation systems for going from point A to point B, a practicable trajectory which takes into account drivable areas and permanent obstacles should be designed first. A robot should then follow this trajectory while avoiding not only dynamic obstacles such as cars and pedestrians but also permanent obstacles such as road sides and central islands. To this end, a robot must be aware of its exact position and must be informed of what its immediate environment is at all times. In dense urban areas, GNSS systems generally suffer from lack of precision due to masks and multipaths. Localization systems have to model these phenomena and even merge with vision based methods in order to obtain the high accuracies required in the process. In this paper we propose an integrated geographic database enabling, on the one hand, the GNSS and vision based localization methods to obtain the required accuracy, and on the other, to provide the robots with information about its surroundings such as drivable surfaces and permanent obstacles. The database is comprised of 3D buildings, 3D roads and a set of 3D visual landmarks. Our system provides most of the information required for autonomous navigation in dense urban areas and has successfully been embedded in real experiments, thanks to a real-time querying system.

In this paper we present relevant results of a work in progress1that deals with the texturing of 3D urban facade models by fusing terrestrial multi-source data acquired by a Mobile Mapping System (MMS). Some of current 3D urban facade models often are textured by using images that contain parts of urban objects that belong to the street. These urban objects represent in this case occlusions since they are located between the acquisition system and the facades. We show the potential use of georeferenced images and 3D point cloud that are acquired at street level by the MMS in generating occlusion-free facade textures. We describe a methodology for reconstructing texture parts of facades that are highly occluded by wide frontal objects.

Abstract Many objects have patterns that vary in appearance at different surface locations. We say that these are differences in materials, and we present a material‐space approach for interactively designing such textures. At the heart of our approach is a new method to pre‐calculate and use a 3D texture tile that is periodic in the spatial dimensions (s, t) and that also has a material axis along which the materials change smoothly. Given two textures and their feature masks, our algorithm produces such a tile in two steps. The first step resolves the features morphing by a level set advection approach, improved to ensure convergence. The second step performs the texture synthesis at each slice in material‐space, constrained by the morphed feature masks. With such tiles, our system lets a user interactively place and edit textures on a surface, and in particular, allows the user to specify which material appears at given positions on the object. Additional operations include changing the scale and orientation of the texture. We support these operations by using a global surface parameterization that is closely related to quad re‐meshing. Re‐parameterization is performed on‐the‐fly whenever the user's constraints are modified.

Many algorithms in texture synthesis, nonphotorealistic rendering (hatching), or remeshing require to define the orientation of some features (texture, hatches, or edges) at each point of a surface. In early works, tangent vector (or tensor) fields were used to define the orientation of these features. Extrapolating and smoothing such fields is usually performed by minimizing an energy composed of a smoothness term and of a data fitting term. More recently, dedicated structures ( N -RoSy and N -symmetry direction fields ) were introduced in order to unify the manipulation of these fields, and provide control over the field's topology (singularities). On the one hand, controlling the topology makes it possible to have few singularities, even in the presence of high frequencies (fine details) in the surface geometry. On the other hand, the user has to explicitly specify all singularities, which can be a tedious task. It would be better to let them emerge naturally from the direction extrapolation and smoothing. This article introduces an intermediate representation that still allows the intuitive design operations such as smoothing and directional constraints, but restates the objective function in a way that avoids the singularities yielded by smaller geometric details. The resulting design tool is intuitive, simple, and allows to create fields with simple topology, even in the presence of high geometric frequencies. The generated field can be used to steer global parameterization methods (e.g., QuadCover).

Abstract We present an explicit method to compute a generalization of the Fourier Transform on a mesh. It is well known that the eigenfunctions of the Laplace Beltrami operator (Manifold Harmonics) define a function basis allowing for such a transform. However, computing even just a few eigenvectors is out of reach for meshes with more than a few thousand vertices, and storing these eigenvectors is prohibitive for large meshes. To overcome these limitations, we propose a band‐by‐band spectrum computation algorithm and an out‐of‐core implementation that can compute thousands of eigenvectors for meshes with up to a million vertices. We also propose a limited‐memory filtering algorithm, that does not need to store the eigenvectors. Using this latter algorithm, specific frequency bands can be filtered, without needing to compute the entire spectrum. Finally, we demonstrate some applications of our method to interactive convolution geometry filtering. These technical achievements are supported by a solid yet simple theoretic framework based on Discrete Exterior Calculus (DEC). In particular, the issues of symmetry and discretization of the operator are considered with great care.

Many algorithms in computer graphics and geometry processing use two orthogonal smooth direction fields (unit tangent vector fields) defined over a surface. For instance, these direction fields are used in texture synthesis, in geometry processing or in nonphotorealistic rendering to distribute and orient elements on the surface. Such direction fields can be designed in fundamentally different ways, according to the symmetry requested: inverting a direction or swapping two directions might be allowed or not. Despite the advances realized in the last few years in the domain of geometry processing, a unified formalism is still lacking for the mathematical object that characterizes these generalized direction fields. As a consequence, existing direction field design algorithms are limited to using nonoptimum local relaxation procedures. In this article, we formalize N -symmetry direction fields, a generalization of classical direction fields. We give a new definition of their singularities to explain how they relate to the topology of the surface. Specifically, we provide an accessible demonstration of the Poincaré-Hopf theorem in the case of N -symmetry direction fields on 2-manifolds. Based on this theorem, we explain how to control the topology of N -symmetry direction fields on meshes. We demonstrate the validity and robustness of this formalism by deriving a highly efficient algorithm to design a smooth field interpolating user-defined singularities and directions.

Accurately representing higher-order singularities of vector fields defined on piecewise linear surfaces is a non-trivial problem. In this work, we introduce a concise yet complete interpolation scheme of vector fields on arbitrary triangulated surfaces. The scheme enables arbitrary singularities to be represented at vertices. The representation can be considered as a facet-based "encoding" of vector fields on piecewise linear surfaces. The vector field is described in polar coordinates over each facet, with a facet edge being chosen as the reference to define the angle. An integer called the period jump is associated to each edge of the triangulation to remove the ambiguity when interpolating the direction of the vector field between two facets that share an edge. To interpolate the vector field, we first linearly interpolate the angle of rotation of the vectors along the edges of the facet graph. Then. we use a variant of Nielson's side-vertex scheme to interpolate the vector field over the entire surface. With our representation, we remove the bound imposed on the complexity of singularities that a vertex can represent by its connectivity. This bound is a limitation generally exists in vertex-based linear schemes. Furthermore, using our data structure, the index of a vertex of a vector field can be combinatorily determined. We show the simplicity of the interpolation scheme with a GPU-accelerated algorithm for a LIC-based visualization of the so-defined vector fields, operating in image space. We demonstrate the algorithm applied to various vector fields on curved surfaces.

International audience