In recent years, vision-language models (VLMs) have advanced open-vocabulary mapping, enabling mobile robots to simultaneously achieve environmental reconstruction and high-level semantic understanding. While integrated object cognition helps mitigate semantic ambiguity in point-wise feature maps, efficiently obtaining rich semantic understanding and robust incremental reconstruction at the instance-level remains challenging. To address these challenges, we introduce OpenVox, a real-time incremental open-vocabulary probabilistic instance voxel representation. In the front-end, we design an efficient instance segmentation and comprehension pipeline that enhances language reasoning through encoding captions. In the back-end, we implement probabilistic instance voxels and formulate the cross-frame incremental fusion process into two subtasks: instance association and live map evolution, ensuring robustness to sensor and segmentation noise. Extensive evaluations across multiple datasets demonstrate that OpenVox achieves state-of-the-art performance in zero-shot instance segmentation, semantic segmentation, and open-vocabulary retrieval. The project page of OpenVox is available at https://open-vox.github.io/.

In everyday life, frequently used objects like cups often have unfixed positions and multiple instances within the same category, and their carriers frequently change as well. As a result, it becomes challenging for a robot to efficiently navigate to a specific instance. To tackle this challenge, the robot must capture and update scene changes and plans continuously. However, current object navigation approaches primarily focus on the semantic level and lack the ability to dynamically update scene representation. To address these limitations, this paper captures the relationships between frequently used objects and their static carriers. Specifically, it constructs an open-vocabulary Carrier-Relationship Scene Graph (CRSG) and updates the carrying status during robot navigation to reflect the dynamic changes of the scene. Based on the CRSG, we further propose an instance navigation strategy that models the navigation process as a Markov Decision Process. At each step, decisions are informed by Large Language Model’s commonsense knowledge and visual-language feature similarity. We designed a series of long-sequence navigation tasks for frequently used everyday items in the Habitat simulator. The results demonstrate that by updating the CRSG, the robot can efficiently navigate to moved targets. Additionally, we deployed our algorithm on a real robot and validated its practical effectiveness. The project page can be found here: https://OpenObject-Nav.github.io.

Loop closure detection and pose estimation play a significant role in correcting odometry trajectories and generating globally consistent point cloud maps. Geometric feature descriptor methods neglect object-level spatial topology features, resulting in inadequate performance in loop closure detection. Semantic graph-based loop closing methods improve upon this, however, they still follow the paradigm of "first generating descriptors, then comparing similarity, and finally achieving alignment (6D pose)". Specifically, they compare two semantic graphs that are not spatially aligned, which makes direct node correspondences impossible and necessitates extensive descriptor extraction and comparison. This decouples similarity comparison from 6D pose estimation, resulting in a cumbersome process that limits practicality and scalability. This paper proposes SLOOP, a novel descriptor-free semantic graph matching method that "aligns two graphs first, followed by efficient similarity comparison". Specifically, we first design a dedicated neighborhood semantic feature module to extract high-quality matched node pairs. Next, we seek the aligned coordinate systems for candidate loops based on the robust ground normal vectors and two suitable node pairs examined by the two-stage global geometric consistency metrics. Finally, the aligned coordinate systems enable efficient extraction and comparison of node spatial distributions. We conducted extensive outdoor loop detection experiments and compared with various loop closure detection approaches, demonstrating the improved performance of SLOOP in loop closure detection and its practicality. The code and related materials are available at https://github.com/bit-tyj/sloop_c.

Open-vocabulary scene understanding is critical for robotics, yet existing 3D Gaussian Splatting (3DGS) methods rely on compressed feature embeddings, compromising semantic fidelity and fine-grained interpretation. Although utilizing uncompressed high-dimensional features offers a potential solution, their direct integration imposes prohibitive memory and computational costs. To address this challenge, we propose OpenMIGS, a novel 3DGS-based framework for multi-granularity, information-preserving open-vocabulary understanding across both object and part levels. Specifically, OpenMIGS first constructs object-level Gaussian fields as structured carriers where a two-stage clustering strategy ensures global consistency in object labeling, and a code-book subsequently associates these object label with their uncompressed high-dimensional features. Building on this, a lightweight implicit field processes the geometric coordinates of object Gaussians to regress part-level high-dimensional features, enabling multi-granularity understanding. Experimental results on multiple datasets show that OpenMIGS outperforms existing methods in open-vocabulary understanding and retrieval tasks. It also supports multi-granularity scene editing for flexible semantic manipulation. The code is available at https://github.com/jingyuzhao1010/OpenMIGS.

In robotic flexible ureteroscopy, great care should be taken to prevent excessive collision force at the ureteroscope tip to avoid urinary tissue damage. However, the passive flexure of a flexible ureteroscope from reaction forces with the surrounding tissue introduces great uncertainties in force control. To address this, a probability-based force control method is proposed in this paper. A morphological wavelet-based statistic test is proposed to detect collision by identifying the change point of the probability distribution of force signal, which is measured by a fiber optical sensor at the ureteroscope tip. The force signal and its change points are applied as an admittance model inputs to generate position command. It is augmented to the robot controller for minimizing the collision force. Meanwhile, a probabilistic model approximates the real interactive system, and combines Bayesian inference for safely online learning the admittance parameters. Experimental results on a flexible ureteroscopy robot show that collision can be detected instantly, and the force is reduced remarkably during the advancement of the robotic ureteroscope. Note to Practitioners—This paper is motivated by the rapidly increasing applications for robotic flexible ureteroscopy. As ureteroscope enters into urethra, urinary bladder, ureter, and reaches renal pelvis, it sometimes penetrates soft tissue, especially in the case of no ureteral access sheath placement. It is important to control the ureteroscope tip force for surgery safety. Therefore, a force control method is proposed in this paper. Experiments conducted on a flexible ureteroscopy robot demonstrate that the proposed method is a protective measure to mitigate the risk of inner surface perforation.

Capturing object semantics and their spatial relationships is crucial to estimating scene similarity for loop closure detection. Existing semantic loop closure detection methods generally treat semantics as landmarks or extract the object topology to compare the similarity of frames. However, they often neglect the absolute spatial distribution of objects, which is essential to capture distinctive features of the scene. A fundamental requirement is to register the spatial coordinates of both frames in a unified reference frame. To address this, we construct aligned coordinate systems between two frames and extract absolute spatial distribution features of objects for loop closure detection. Building on this, we introduce SeGraM, a unified semantic graph matching approach applicable to both indoor and outdoor environments. Specifically, for each pair of semantic graphs, we first establish correspondences between nodes, referred to as node pairs. We then evaluate the geometric and semantic consistency of these pairs, along with the local graph features in the surrounding. To facilitate meaningful comparisons, two node pairs are carefully selected to establish aligned spherical coordinate systems, with ground normals to define the Z axes outdoors. SeGraM is validated in both indoor and outdoor scenarios and is compared with multiple algorithms, demonstrating improvements in loop closure detection accuracy. The code is accessible at https://github.com/BIT-TYJ/SeGraM.

Robotic systems demand accurate and comprehensive 3D environment perception, requiring simultaneous capture of photo-realistic appearance (optical), precise layout shape (geometric), and open-vocabulary scene understanding (semantic). Existing methods typically achieve only partial fulfillment of these requirements while exhibiting optical blurring, geometric irregularities, and semantic ambiguities. To address these challenges, we propose OmniMap. Overall, OmniMap represents the first online mapping framework that simultaneously captures optical, geometric, and semantic scene attributes while maintaining real-time performance and model compactness. At the architectural level, OmniMap employs a tightly coupled 3DGS-Voxel hybrid representation that combines fine-grained modeling with structural stability. At the implementation level, OmniMap identifies key challenges across different modalities and introduces several innovations: adaptive camera modeling for motion blur and exposure compensation, hybrid incremental representation with normal constraints, and probabilistic fusion for robust instance-level understanding. Extensive experiments show OmniMap's superior performance in rendering fidelity, geometric accuracy, and zero-shot semantic segmentation compared to state-of-the-art methods across diverse scenes. The framework's versatility is further evidenced through a variety of downstream applications, including multi-domain scene Q&A, interactive editing, perception-guided manipulation, and map-assisted navigation.

Recently the dense Simultaneous Localization and Mapping (SLAM) based on neural implicit representation has shown impressive progress in hole filling and high-fidelity mapping. Nevertheless, existing methods either heavily rely on known scene bounds or suffer inconsistent reconstruction due to drift in potential loop-closure regions, or both, which can be attributed to the inflexible representation and lack of local constraints. In this paper, we present LCP-Fusion, a neural implicit SLAM system with enhanced local constraints and computable prior, which takes the sparse voxel octree structure containing feature grids and SDF priors as hybrid scene representation, enabling the scalability and robustness during mapping and tracking. To enhance the local constraints, we propose a novel sliding window selection strategy based on visual overlap to address the loop-closure, and a practical warping loss to constrain relative poses. Moreover, we estimate SDF priors as coarse initialization for implicit features, which brings additional explicit constraints and robustness, especially when a light but efficient adaptive early ending is adopted. Experiments demonstrate that our method achieve better localization accuracy and reconstruction consistency than existing RGB-D implicit SLAM, especially in challenging real scenes (ScanNet) as well as self-captured scenes with unknown scene bounds. The code is available at https://github.com/laliwang/LCP-Fusion.

Filtering out outlier data associations between local maps can improve the robustness and accuracy of multi-robot localization. When the overlap is low and the field of view difference is large, it is likely to produce outlier data associations between local maps, which will reduce the matching accuracy and even lead to the failure of collaborative localization. To solve this problem, this paper proposes a novel outdoor robust collaborative localization algorithm (HORCL) capable for high outlier ratio scenes. The Mixture Probability Model (MPM) and the Hierarchical EM (Expectation Maximization) algorithm in HORCL are applied to screen two levels of outliers (loop closure constraints and point pairs) and improve localization performance. Specifically, the inlier probabilities of data associations are calculated in MPM to identify outliers by considering geometric distances, semantic consistency, and spatial consistency. Then, outlier loop closures and outlier point pairs in inlier constraints are filtered by applying the Hierarchical EM algorithm, thereby relieving the adverse effect of outliers on localization accuracy. The proposed algorithm is validated on public datasets and compared with the latest methods, demonstrating the improvement in localization accuracy and robustness. The code is available at https://github.com/BIT-TYJ/HORCL.

Capturing the semantics of objects and the topological relationship allows the robot to describe the scene more intelligently like a human and measure the similarity between scenes (loop closure detection) more accurately. However, many current semantic graph matching methods are based on walk descriptors, which only extract adjacency relations between objects. In such way, the comprehensive information in the semantic graph is not fully exploited, which may lead to false closed-loop detection. This paper proposes a novel spatial semantic graph matching method (SSGM) in indoor environments, which considers multifaceted information of the semantic graphs. Firstly, two semantic graphs are aligned in the same coordinate space contributed by the second-order spatial compatibility metric between objects and local graph features of objects in semantic graphs. Secondly, the similarity of the spatial distribution of overall semantic graphs is further evaluated. The proposed algorithm is validated on public datasets and compared with the latest semantic graph matching methods, demonstrating improved accuracy and efficiency in loop closure detection. The code is available at https://github.com/BIT-TYJ/SSGM.

Autonomous robots are often required to acquire high-level prior knowledge by continuously reconstructing the semantics and geometry of the surrounding scene, which is the basis of exploration and planning. Most existing continuous semantic mapping algorithms cannot distinguish potential differences in voxels, resulting in an over-inflated map. Furthermore, fixed-size query ranges introduce high computational complexity. Based on the limitation of over-inflation and inefficiency, this paper proposes a novel incremental continuous semantic occupancy mapping algorithm (S-MKI). The key innovation of this work comes from the two models in the preprocessing stage. On the one hand, Redundant Voxel Filter Model utilizes context entropy to filter out redundant voxels to improve the confidence of the final map, where objects have accurate boundaries with sharp edges. On the other hand, Adaptive Kernel Length Model adaptively adjusts the kernel length with class entropy, which reduces the inherent amount of training data. The final multientropy kernel inference function is formulated to integrate these two models to infer sparse noisy sensor data into dense accurate 3D maps. Experimental results conducted in both indoors and outdoors datasets validate that S-MKI outperforms existing methods.

The collaborative operation of multiple robots can make up for the shortcomings of a single robot, such as limited field of perception or sensor failure. multirobots collaborative semantic mapping can enhance their comprehensive contextual understanding of the environment. However, existing multirobots collaborative semantic mapping algorithms mainly apply discrete occupancy map inference, and do not compensate for inconsistent labels of local maps caused by differences in robot perspectives, which leads to greatly reduced availability and accuracy of the final global map. To address the challenges of discontinuous maps and inconsistent semantic labels, this paper proposes a novel hierarchical and dense collaborative continuous semantic occupancy mapping algorithm (HD-CCSOM). This work decomposes and formulates robot collaborative continuous semantic occupancy mapping problem at two levels. At the single robot level, the multi-entropy kernel inference method smoothly processes the registered semantic point cloud and infers a local continuous semantic occupancy map for each robot. At the collaborative robots level, the local maps are fused into a global enhanced and consistent semantic map via the label diffusion method based on a graph model. The proposed algorithm has been validated on public datasets and in simulated and real scenes, demonstrating significant improvements in mapping accuracy and efficiency.