This paper presents FLAF, a focal line and feature-constrained active view planning method for autonomous orientation adjustment of a rotatable active camera during mobile robot navigation. FLAF is built on a visual teach-and-repeat (VT&R) system, which enables robots to cruise various paths that fulfill many daily autonomous navigation requirements. The VT&R system integrates Visual Simultaneous Localization and Mapping (VSLAM) with trajectory following. However, tracking failures in feature-based VSLAM, particularly in textureless regions common in human-made environments, poses a significant challenge to real-world VT&R deployment. To address this, the proposed view planner is integrated into a feature-based VSLAM system, creating an active camerabased VSLAM (AC-SLAM) solution that mitigates tracking failures. Our system features a Pan-Tilt Unit (PTU)-based active camera mounted on a mobile robot. FLAF actively directs the camera toward more map points during path learning and toward more feature-identifiable map points while following the learned trajectory. Using FLAF, the AC-SLAM system constructs a complete path map during teaching and maintains stable localization during repeating. Experimental results in real scenarios show that FLAF significantly outperforms existing methods by accounting for feature identifiability, particularly the view angle of the features. During effectively dealing with low-texture regions in active view planning, considering feature identifiability enables our active VT&R system to perform well in challenging environments.

Robotic manipulation in dynamic environments often requires seamless transitions between different grasp types to maintain stability and efficiency. However, achieving smooth and adaptive grasp transitions remains a challenge, particularly when dealing with external forces and complex motion constraints. Existing grasp transition strategies often fail to account for varying external forces and do not optimize motion performance effectively. In this work, we propose an Imitation-Guided Bimanual Planning Framework that integrates efficient grasp transition strategies and motion performance optimization to enhance stability and dexterity in robotic manipulation. Our approach introduces Strategies for Sampling Stable Intersections in Grasp Manifolds for seamless transitions between uni-manual and bi-manual grasps, reducing computational costs and regrasping inefficiencies. Additionally, a Hierarchical Dual-Stage Motion Architecture combines an Imitation Learning-based Global Path Generator with a Quadratic Programming-driven Local Planner to ensure real-time motion feasibility, obstacle avoidance, and superior manipulability. The proposed method is evaluated through a series of force-intensive tasks, demonstrating significant improvements in grasp transition efficiency and motion performance. A video demonstrating our simulation results can be viewed at https://youtu.be/3DhbUsv4eDo.

Keyframe selection plays a crucial role in balancing computational efficiency and localization accuracy in Visual Simultaneous Localization and Mapping (VSLAM) systems. Existing keyframe selection methods often struggle to capture high-level semantic information in environments where multiple semantic dimensions interact. In this paper, we propose the Multi-dimensional Semantic Analysis (MSA) module based on Visual-Language Models (VLMs). By leveraging the capability of VLMs to extract rich semantic features, we compute the similarity between each image frame and a set of textual descriptions, generating a scene descriptor that quantifies the semantic distance between frames across multiple dimensions (e.g., object count, texture, and lighting). We then introduce the Scene Change Assessment (SCA) module based on Bayesian On-line Changepoint Detection (BOCD), which identifies keyframes with significant semantic information gain, thereby reducing the total number of keyframes. Extensive experiments on an open dataset demonstrate that our method not only significantly reduces the number of keyframes but also maintains high localization accuracy. Furthermore, the inference speed of the MSA module satisfies the real-time requirements of VSLAM. These results underscore the potential of our approach to enhance the efficiency of keyframe selection.

Accurate indoor location-based services are important for mobile robots, especially in complex indoor environments. In this paper, we propose a heterogeneous graph network-based ultra-wide band (UWB) localization method to provide accurate and robust localization results for mobile robots in complex indoor scenarios. The core of our approach lies in constructing the anchors, ranging measurements and tags into a heterogeneous graph structure according to the topological structure of the UWB localization system, and then design a spatial-temporal heterogeneous graph attention neural network to extract high-level features and estimate the tag locations from the graph. Therefore, the geometric relationships contained in the UWB localization system are comprehensively established, while the spatial and temporal information contained in the ranging measurements can also be extracted. We validate the proposed method through real-world experiments. The results demonstrate that, compared to existing deep learning-based methods, the constructed heterogeneous graph better represents the geometric structure of the UWB localization system, and the designed heterogeneous graph neural network effectively extracts the spatial-temporal and geometric features. Consequently, the accuracy and robustness of UWB localization are significantly improved.

Visual Simultaneous Localization and Mapping (VSLAM) is a critical technology for autonomous driving and mobile robotics. Traditional VSLAM methods based on discrete representations, such as point clouds, offer high computational efficiency and excellent localization accuracy, but they exhibit limited robustness. In contrast, methods employing field representations, like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3D GS), provide greater robustness at the expense of increased computational demands and reduced localization accuracy. Hybrid VSLAM approaches that attempt to combine these representations typically rely on serial, synchronous cascades, which compromise robustness, computational efficiency, and GPU memory usage. This paper introduces a novel adaptive cross-representation VSLAM framework that applies different representation modeling techniques to distinct regions of an image sequence and adopts asynchronous parallel modeling in overlapping regions. A Vision Language Model (VLM) is used to analyze the image sequence, enabling the detection of representation modeling regions and adaptive switching between representations. Cross-representation data association is performed through a coarse-to-fine feature selection process, resulting in a globally consistent map. The proposed method is evaluated on both public and custom-collected datasets, where experimental results show that it surpasses state-of-the-art methods in terms of robustness, computational efficiency, localization accuracy, and GPU memory usage.

Limited by the working principles, LiDAR-SLAM systems suffer from the degeneration phenomenon in environments such as long corridors and tunnels, due to the lack of sufficient geometric features for frame-to-frame matching. The accuracy and sensitivity of existing degeneracy detection methods need to be further improved. In this paper, we propose a novel method for degeneracy detection using local geometric models based on point-to-distribution matching. To obtain an accurate description of local geometric models, an adaptive adjustment of voxel segmentation according to the point cloud distribution and density is designed. The codes of the proposed method is open-source and available at https://github.com/jisehua/Degenerate-Detection.git. Experiments with public datasets and self-build robots were conducted to evaluate the methods. The results exhibit that our proposed method achieves higher accuracy than the other existing approaches. Applying our proposed method is beneficial for improving the robustness of the LiDAR-SLAM systems.

Large-scale point cloud consists of a multitude of individual objects, thereby encompassing rich structural and underlying semantic contextual information, resulting in a challenging problem in efficiently segmenting a point cloud. Most existing researches mainly focus on capturing intricate local features without giving due consideration to global ones, thus failing to leverage semantic context. In this paper, we propose a Similarity-Weighted Convolution and local-global Fusion Network, named SWCF-Net, which takes into account both local and global features. We propose a Similarity-Weighted Convolution (SWConv) to effectively extract local features, where similarity weights are incorporated into the convolution operation to enhance the generalization capabilities. Then, we employ a downsampling operation on the K and V channels within the attention module, thereby reducing the quadratic complexity to linear, enabling Transformer to deal with large-scale point cloud. At last, orthogonal components are extracted in the global features and then aggregated with local features, thereby eliminating redundant information between local and global features and consequently promoting efficiency. We evaluate SWCF-Net on large-scale outdoor datasets SemanticKITTI and Toronto3D. Our experimental results demonstrate the effectiveness of the proposed network. Our method achieves a competitive result with less computational cost, and is able to handle large-scale point clouds efficiently. The code is available at https://github.com/Sylva-Lin/SWCF-Net.

The utilization of cloud infrastructure and its extensive range of Internet-accessible resources holds significant potential for advancing intelligent transportation and robotics. Over the past two decades, interest in cloud-edge collaborative simultaneous localization and mapping (SLAM) has grown markedly. Consequently, a comprehensive review of current trends in this field is crucial for both novice and experienced researchers. This paper examines robots and automation systems that rely on network-based data or code, particularly in the context of SLAM development. Applying SLAM to mobile robots with limited computing power is essential for achieving autonomous navigation, and cloud-edge collaborative SLAM has emerged as an efficient solution. The review is structured around four key benefits of cloud-edge collaborative SLAM: Assisted Cloud Computing, which provides access to cloud computation and reduces the burden on edge devices; Total Cloud Computing, where the majority of computation is offloaded to the cloud, while edge devices primarily handle sensing and low-cost pre-processing; Data Storage, enabling access to large datasets, such as high-resolution environment maps and extensive training datasets, enhancing overall performance; and Data Transmission, involving cloud-edge communication for efficient data transfer and data association. Additionally, we address the challenges in existing work and the development of asynchronous collaboration and implicit representation transmission, which could mitigate transmission latency in communication-constrained environments. We believe that this review will bridge the gap between SLAM systems and deployed robotic systems, promoting the advancement of cloud-edge collaborative SLAM.

Visual simultaneous localization and mapping (VSLAM) is a crucial technology in intelligent vehicles that relies on either explicit or implicit representations. Explicit methods are prevalent in real-time systems, offer precise geometric control, and are easy to visualize. However, they struggle with complex, dynamic environments and require high storage capacity. On the other hand, implicit techniques excel in handling intricate, changing shapes due to their compact representation and inference ability while requiring more complex display and rendering processes. A combination of both types of representations could significantly enhance the performance of VSLAM, but the cross-data association method for standalone explicit and implicit representations is still lacking. To this end, this paper proposes a data association scheme that bridges the gap between explicit and implicit representations by individually modeling the uncertainties in each representation. Our approach features a multi-level feature selection process tailored for data association. It initially extracts coarse-level features during explicit representation generation based on Bayesian estimation and refines them using the implicit representation based on ray sampling, which enhances robustness while reducing rendering costs. We rigorously evaluated our proposed methodology against current state-of-the-art approaches using public datasets and real robot scenes. The results show that our coarse-to-fine feature selection method outperforms existing techniques both quantitatively and qualitatively, suggesting its potential to significantly boost the contemporary VSLAM system performance.

Robot person following (RPF) is a capability that supports many useful human-robot-interaction (HRI) applications. However, existing solutions to person following often as-sume full observation of the tracked person. As a consequence, they cannot track the person reliably under partial occlusion where the assumption of full observation is not satisfied. In this paper, we focus on the problem of robot person following under partial occlusion caused by a limited field of view of a monocular camera. Based on the key insight that it is possible to locate the target person when one or more of hislher joints are visible, we propose a method in which each visible joint contributes a location estimate of the followed person. Experiments on a public person-following dataset show that, even under partial occlusion, the proposed method can still locate the person more reliably than the existing SOTA methods. As well, the application of our method is demonstrated in real experiments on a mobile robot.

Human-aware path planner is essential for achieving harmonious coexistence between humans and robots in highly dynamic environments. In this paper, we propose an integrated framework to find the optimal path in the complex environment with considering collision risk, social norms, and crowded areas. In the proposed framework, a general dynamic group model (g-space) based on the Gaussian Mixed Model (GMM) is proposed as the social norms of dynamic groups, which not only considers the factors of humans (e.g., pose, quantity, distribution, psychology) but also establishes the proximity and human interacting constraints of dynamic groups. An integrated Collision Risk and Human Space (CR&HS) model is applied to achieve human-acceptable behaviors, in which both collision avoidance, human comfort, and interference-free constraints have been involved. Moreover, an Improved Virtual Doppler Method (IVDM) has been used to realize safety navigation to avoid the robot falling into the crowded area. Finally, the proposed framework has been utilized with the sampling-based rapidly-exploring random tree. Experimental results demonstrate that the proposed method can generate the optimal human-aware collision-free path in complex environments. Note to Practitioners—This paper aims to plan an optimal trajectory for the robot in highly dynamic environments. In this field, it is still a challenging task to plan a trajectory with collision-free, human-aware, and crowd-aware. To do that, we present an integrated framework to generate the optimal trajectory by involving the collision risk, social norms, and human density. First, the g-space model is adopted as interference-free constraints of dynamic groups. The integrated knowledge fusion model (CR&HS) then penalizes the manners which have higher collision risk and adverse effects on human interaction or human comfortable. Besides, human motion and density are provided to a robot by IVDM. The proposed framework is utilized in the sampling-based rapidly-exploring random tree as the evaluation module. Finally, the feasibility and reliability of the proposed method have been verified by experiments in different simulated environments. The proposed framework can be applied in most mobile service robots to achieve human-friendly manners.

Autonomous pedestrian-aware navigation in shared human-robot environments is a challenging problem. Here we consider a common situation in which a large crowd of pedestrians moves together in a limited space. Traditional planners struggle to find collision-free paths in such situations since the free space is limited and always changing. To solve this problem, we proposed a flow map-based RRT* method (FM-RRT*) containing a velocity layer and a minimally-intrusive layer. The proposed method models the velocity of the pedestrian flow and the area where the robot is less invasive to pedestrians. Furthermore, we propose an adaptive bias sampling, which drives the robot considering relative velocity, or minimal intrusion, according to the pedestrian flow. The evaluation is conducted in the Crowdbot Challenge simulator. The results show that our method can find a feasible path considering collision risk while simultaneously avoiding intrusive human movement.

Autonomous mobile manipulation robots that can collect trolleys are widely used to liberate human resources and fight epidemics. Most prior robotic trolley collection solutions only detect trolleys with 2D poses or are merely based on spe-cific marks and lack the formal design of planning algorithms. In this paper, we present a novel mobile manipulation system with applications in luggage trolley collection. The proposed system integrates a compact hardware design and a progressive perception and planning framework, enabling the system to efficiently and robustly collect trolleys in dynamic and complex environments. For perception, we first develop a 3D trolley detection method that combines object detection and keypoint estimation. Then, a docking process in a short distance is achieved with an accurate point cloud plane detection method and a novel manipulator design. On the planning side, we formulate the robot's motion planning under a nonlinear model predictive control framework with control barrier functions to improve obstacle avoidance capabilities while maintaining the target in the sensors' field of view at close distances. We demonstrate our design and framework by deploying the system on actual trolley collection tasks, and their effectiveness and robustness are experimentally validated. (Video11Video demonstration: https://youtu.be/6SwjgGvRtno.)

As the basics of Visual Simultaneous Localization And Mapping (VSLAM), keyframes play an essential role. In previous works, keyframes are selected according to a series of view change-based strategies for short-term data association (STDA). However, the texture enrichment of frames is always ignored, resulting in the failure of long-term data association (LTDA). In this paper, we propose an information enrichment selection strategy with an information occupancy grid model and a deep descriptor. Frame is expressed by a deep global descriptor for a statistical explainable abstraction, in which the texture enrichment is indicated. Based on the abstraction, an information occupancy grid model is established to measure the information enrichment and the potential LTDA ability. Evaluations on variant datasets are conducted, showing the advantage of our proposed method in terms of keyframe selection and tracking precision. Also, the statistical explainability of the deep descriptor is provided. The proposed keyframe selection strategy can improve LTDA and tracking precision, especially in situations with repeated observations and loop-closures.

Assistive robot systems have been developed to help people accomplish daily manipulation tasks especially for those with disabilities, where scene understanding plays a crucial role in enabling robots to interpret the surroundings and behave accordingly. Most of the current systems approach scene understanding without considering the functional dependencies between objects. However, it is only valuable to interact with some objects when their function-relevant counterparts are considered. In this paper, we augment an assistive robotic arm system with an end-to-end semantic relationship reasoning model. It incorporates functional relationships between pairs of objects for semantic scene understanding. To ensure good generalization to unseen objects and relationships, the model works in a category-agnostic manner. We evaluate our design and three baseline methods on a self-collected benchmark with two levels of difficulty. To further demonstrate the effectiveness, the model is integrated with a symbolic planner for goal-oriented, multi-step manipulation task on a real-world assistive robotic arm platform.

Point cloud is a discrete and unordered expression of 3D data. A lot of methods have been proposed to solve the problem in 3D object classification and scene recognition. To handle the huge amount of unordered point cloud, down-sampling before processing is needed. The shortage of existing sampling methods is the lack of geometry information consideration, which is essential for point cloud classification and segmentation tasks. Our method is mainly motivated by the observation that points with a high curvature variation can depict the outlines of objects. Thus, we propose a curvature variation based sampling method for point cloud classification and segmentation tasks. We aim to sample points with high curvature variations, which are considered to be more suitable for classification and segmentation tasks than the traditional sampling method. We combine the proposed sampling algorithm with the existing sampling method for multiple information fusion, and a higher accuracy and mean IoU can be achieved. The experimental results verify the advantage of considering curvature variation in classification and segmentation tasks.

Recent works on semantic Simultaneous Localization and Mapping (SLAM) utilizing object landmarks have shown superiority in terms of robustness and accuracy in tracking and localization. 3D object landmarks represented by a cubic or quadric surface are inferred from 2D object bounding boxes which are typically captured from multiple views by an object detector. Nevertheless, bounding box noises and small camera baseline may lead to an inaccurate 3D object landmark inference. Inspired by the dual quadric enveloping property, in this work, we introduce the horizontal support assumption to constrain rotation w.r.t. roll and pitch for a quadric representation. As the result, we reduce the number of quadric parameters and narrow down the solution space, and ultimately produce a relatively accurate inference. Extensive experimental evaluations under both simulated and real scenarios are conducted in this paper. Quantitative results demonstrate that our approach outperforms the state-of-the-art.

Keypoint matching is an important operation in computer vision and its applications such as visual simultaneous localization and mapping (SLAM) in robotics. This matching operation heavily depends on the descriptors of the keypoints, and it must be performed reliably when images undergo conditional changes such as those in illumination and viewpoint. In this paper, a descriptor fusion model (DFM) is proposed to create a robust keypoint descriptor by fusing CNN-based descriptors using autoencoders. Our DFM architecture can be adapted to either trained or pre-trained CNN models. Based on the performance of existing CNN descriptors, we choose HardNet and DenseNet169 as representatives of trained and pre-trained descriptors. Our proposed DFM is evaluated on the latest benchmark datasets in computer vision with challenging conditional changes. The experimental results show that DFM is able to achieve state-of-the-art performance, with the mean mAP that is 6.45% and 6.53% higher than HardNet and DenseNet169, respectively.

Keypoint matching is an important operation in computer vision and its applications such as visual simultaneous localization and mapping (SLAM) in robotics. This matching operation heavily depends on the descriptors of the keypoints, and it must be performed reliably when images undergo condition changes such as those in illumination and viewpoint. Previous research in keypoint description has pursued three classes of descriptors: hand-crafted, those from trained convolutional neural networks (CNN), and those from pre-trained CNNs. This paper provides a comparative study of the three classes of keypoint descriptors, in terms of their ability to handle conditional changes. The study is conducted on the latest benchmark datasets in computer vision with challenging conditional changes. Our study finds that (a) in general CNN-based descriptors outperform hand-crafted descriptors, (b) the trained CNN descriptors perform better than pre-trained CNN descriptors with respect to viewpoint changes, and (c) pre-trained CNN descriptors perform better than trained CNN descriptors with respect to illumination changes. These findings can serve as a basis for selecting appropriate keypoint descriptors for various applications.

Motivated by the need to improve the performance of visual loop closure verification via multi-view geometry (MVG) under significant illumination and viewpoint changes, we propose a keypoint matching method that uses landmarks as an intermediate image representation in order to leverage the power of deep learning. In environments with various changes, the traditional verification method via MVG may encounter difficulty because of their inability to generate a sufficient number of correctly matched keypoints. Our method exploits the excellent invariance properties of convolutional neural network (ConvNet) features, which have shown outstanding performance for matching landmarks between images. By generating and matching landmarks first in the images and then matching the keypoints within the matched landmark pairs, we can significantly improve the quality of matched keypoints in terms of precision and recall measures. The proposed method is validated on challenging datasets that involve significant illumination and viewpoint changes, to establish its superior performance to the standard keypoint matching method.

For VSLAM (Visual Simultaneous Localization and Mapping), localization is a challenging task, especially for some challenging situations: textureless frames, motion blur, etc. To build a robust exploration and localization system in a given space, a submap-based VSLAM system is proposed in this paper. Our system uses a submap back-end and a visual front-end. The main advantage of our system is its robustness with respect to tracking failure, a common problem in current VSLAM algorithms. The robustness of our system is compared with the state-of-the-art in terms of average tracking percentage. The precision of our system is also evaluated in terms of ATE (absolute trajectory error) RMSE (root mean square error) comparing the state-of-the-art. The ability of our system in solving the “kidnapped” problem is demonstrated. Our system can improve the robustness of visual localization in challenging situations.

Block adjustment can improve the accuracy of airborne InSAR data which covers large areas under the conditions of sparse control points, and it can improve the planar and vertical accuracy using a few of gound control points (GCPs). In this paper, we research the orthogonal decomposition of look vector model, three dimensional reconstruction model for airborne InSAR and the sensitivity equations at first. Secondly, we analyze the error sources and adjustment parameters for the Chinese airborne InSAR system of Chinese Academy of Surveying and Mapping SAR (CASMSAR). Finally, InSAR block adjustment method has been proposed for the airborne CASMSAR system, and the DEM is directly extracted. Experiments are carried out with X-band interferometric data acquired by CASMSAR system, and the effectiveness and practicability of the process method are verified. It can be seen from this article the results met the requirements of topographic mapping for the scale of 1∶10000 in China.

The problem of corner detection on planar curves is examined based on human perception of local graphic features. First, a set of fuzzy patterns of contour points are established. Then, corner detection is characterized as a fuzzy classification problem that contains three stages: evaluation, classification, and location. Compared with existing methods, the proposed approach is superior in that it explains the curve, instead of simple labeling, and it performs based on human perception. Experimental results on shapes of various complexities are presented. The performance with respect to noise is also addressed.