This study addresses the challenge of generating online 3D Gaussian Splatting (3DGS) models from RGB-only frames. Previous studies have employed dense SLAM techniques to estimate 3D scenes from keyframes for 3DGS model construction. However, these methods are limited by their reliance solely on keyframes, which are insufficient to capture an entire scene, resulting in incomplete reconstructions. Moreover, building a generalizable model requires incorporating frames from diverse viewpoints to achieve broader scene coverage. However, online processing restricts the use of many frames or extensive training iterations. Therefore, we propose a novel method for high-quality 3DGS modeling that improves model completeness through adaptive view selection. By analyzing reconstruction quality online, our approach selects optimal non-keyframes for additional training. By integrating both keyframes and selected non-keyframes, the method refines incomplete regions from diverse viewpoints, significantly enhancing completeness. We also present a framework that incorporates an online multi-view stereo approach, ensuring consistency in 3D information throughout the 3DGS modeling process. Experimental results demonstrate that our method outperforms state-of-the-art methods, delivering exceptional performance in complex outdoor scenes.

This study addresses the challenge of performing visual localization in demanding conditions such as night-time scenarios, adverse weather, and seasonal changes. While many prior studies have focused on improving image matching performance to facilitate reliable dense keypoint matching between images, existing methods often heavily rely on predefined feature points on a reconstructed 3D model. Consequently, they tend to overlook unobserved keypoints during the matching process. Therefore, dense keypoint matches are not fully exploited, leading to a no-table reduction in accuracy, particularly in noisy scenes. To tackle this issue, we propose a novel localization method that extracts reliable semi-dense 2D-3D matching points based on dense keypoint matches. This approach involves regressing semi-dense 2D keypoints into 3D scene coordinates using a point inference network. The network utilizes both geometric and visual cues to effectively in-fer 3D coordinates for unobserved keypoints from the observed ones. The abundance of matching information significantly enhances the accuracy of camera pose estimation, even in scenarios involving noisy or sparse 3D models. Comprehensive evaluations demonstrate that the proposed method outperforms other methods in challenging scenes and achieves competitive results in large-scale visual localization benchmarks. The code will be available at https://github.com/TruongKhang/DeViLoc.

This study tackles image matching in difficult scenarios, such as scenes with significant variations or limited texture, with a strong emphasis on computational efficiency. Previous studies have attempted to address this challenge by encoding global scene contexts using Transformers. However, these approaches have high computational costs and may not capture sufficient high-level contextual information, such as spatial structures or semantic shapes. To overcome these limitations, we propose a novel image-matching method that leverages a topic-modeling strategy to capture high-level contexts in images. Our method represents each image as a multinomial distribution over topics, where each topic represents semantic structures. By incorporating these topics, we can effectively capture comprehensive context information and obtain discriminative and high-quality features. Notably, our coarse-level matching network enhances efficiency by employing attention layers only to fixed-sized topics and small-sized features. Finally, we design a dynamic feature refinement network for precise results at a finer matching stage. Through extensive experiments, we have demonstrated the superiority of our method in challenging scenarios. Specifically, our method ranks in the top 9% in the Image Matching Challenge 2023 without using ensemble techniques. Additionally, we achieve an approximately 50% reduction in computational costs compared to other Transformer-based methods. Code is available at https://github.com/TruongKhang/TopicFM.

This study addresses an image-matching problem in challenging cases, such as large scene variations or textureless scenes. To gain robustness to such situations, most previous studies have attempted to encode the global contexts of a scene via graph neural networks or transformers. However, these contexts do not explicitly represent high-level contextual information, such as structural shapes or semantic instances; therefore, the encoded features are still not sufficiently discriminative in challenging scenes. We propose a novel image-matching method that applies a topic-modeling strategy to encode high-level contexts in images. The proposed method trains latent semantic instances called topics. It explicitly models an image as a multinomial distribution of topics, and then performs probabilistic feature matching. This approach improves the robustness of matching by focusing on the same semantic areas between the images. In addition, the inferred topics provide interpretability for matching the results, making our method explainable. Extensive experiments on outdoor and indoor datasets show that our method outperforms other state-of-the-art methods, particularly in challenging cases.

This study addresses a view-path-planning problem during 3-D scanning of a large-scale structure based on multiview stereo (MVS) for unmanned aerial platforms. Recently, most studies have adopted an explore-then-exploit strategy for 3-D scanning. The strategy first generates a coarse model from a simple overhead scanning and then plans an inspection path to cover the entire surface of the coarse model. However, even though the inspection path may be optimal, it is difficult to guarantee a complete and accurate reconstruction result due to defective factors of MVS such as occlusions, textureless surfaces, and insufficient parallaxes. Furthermore, the entire procedure of this strategy is inefficiently slow because of path redundancies and long MVS processing time. Therefore, we propose a novel view-path-planning method for 3-D scanning based on an online MVS reconstruction algorithm. The suggested method incrementally reconstructs the target model online and iteratively plans view paths by analyzing the current partial reconstructions. The method continuously analyzes the quality of the model and detects inaccurately reconstructed surfaces. It then plans an inspection path that provides a complete coverage of the detected surfaces while maximizing the performance of MVS. This method can construct a complete 3-D model in a single scanning trial without the need for rescanning. Extensive experiments show that our method outperforms the other state-of-the-art methods, especially in terms of the model completeness of complex structures.

Multi-view stereo (MVS) algorithms have been commonly used to model large-scale structures. When processing MVS, image acquisition is an important issue because its reconstruction quality depends heavily on the acquired images. Recently, an explore-then-exploit strategy has been used to acquire images for MVS. This method first constructs a coarse model by exploring an entire scene using a pre-allocated camera trajectory. Then, it rescans the unreconstructed regions from the coarse model. However, this strategy is inefficient because of the frequent overlap of the initial and rescanning trajectories. Furthermore, given the complete coverage of images, MVS algorithms do not guarantee an accurate reconstruction result.In this study, we propose a novel view path-planning method based on an online MVS system. This method aims to incrementally construct the target three-dimensional (3D) model in real time. View paths are continually planned based on online feedbacks from the partially constructed model. The obtained paths fully cover low-quality surfaces while maximizing the reconstruction performance of MVS. Experimental results demonstrate that the proposed method can construct high quality 3D models with one exploration trial, without any rescanning trial as in the explore-then-exploit method.

In this study, we addressed a path planning problem of a mobile robot to construct highly accurate 3D models of an unknown environment. Most studies have focused on exploration approaches, which find the most informative viewpoint or trajectories by analyzing a volumetric map. However, the completion of a volumetric map does not necessarily describe the completion of a 3D model. A highly complicated structure sometimes cannot be represented as a volumetric model. We propose a novel exploration algorithm that considers not only a volumetric map but also reconstructed surfaces. Unlike previous approaches, we evaluate the model completeness according to the quality of the reconstructed surfaces and extract low-confidence surfaces. The surface information is used to guide the computation of the exploration path. Experimental results showed that the proposed algorithm performed better than other state-of-the-art exploration methods and especially improved the completeness and confidence of the 3D models.

In this paper, we propose a novel algorithm for planning exploration paths to generate 3D models of unknown environments by using a micro-aerial vehicle (MAV). Our algorithm initially determines a next-best-view (NBV) that maximizes information gain and plans a collision-free path to reach the NBV. Along the path, the MAV explores the greatest unknown area although it sometimes misses minor unreconstructed region, such as a hole or a sparse surface. To cover such a region, we propose an online inspection algorithm that consistently provides an optimal coverage path toward the NBV in real time. The algorithm iteratively refines an inspection path according to the acquired information until the modeling of a specific local area is complete. We evaluated the proposed algorithm by comparing it with other state-of-the-art approaches through simulated experiments. The results show that our algorithm outperforms the other approaches in both exploration and 3D modeling scenarios.