Accurate visualization of subtle vascular dynamics is a knowledge-intensive challenge in minimally invasive surgery. Conventional imaging systems struggle to reveal these imperceptible motions amidst the dynamic complexity of surgical scenes, limiting decision-making reliability. We introduce EndoControlMag , a Lagrangian framework that employs mask-conditioned magnification to selectively enhance vascular motion while preserving the structural integrity of surrounding tissues in endoscopic videos. Our approach integrates two key designs: Periodic Reference Resetting (PRR) , which divides videos into short overlapping clips with dynamically updated reference frames to alleviate error accumulation while maintaining temporal coherence, and Hierarchical Tissue-aware Magnification (HTM) , which combines pretrained visual tracking for accurate vessel localization with dual-mode adaptive softening strategies. HTM employs either motion-based softening that modulates magnification strength proportional to observed tissue displacement, or distance-based exponential decay that simulates biomechanical force attenuation. This strategy enables robust performance across diverse surgical scenarios where motion-based softening excels with complex tissue deformations and distance-based softening provides stability under unreliable optical flow conditions. To validate generality and scalability, we construct EndoVMM24, a benchmark dataset spanning four surgical specialties and diverse intraoperative scenarios. Extensive quantitative metrics, qualitative assessments, and expert surgeon evaluations demonstrate that EndoControlMag significantly outperforms existing methods in magnification accuracy, image quality, and robustness. This work advances engineering informatics for surgical vision by providing a reproducible, context-aware framework that supports reliable decision-making in minimally invasive procedures. The code, dataset, and video results are available at https://cho-haz.github.io/EndoControlMag/ .

Robotic systems operating in dynamic and uncertain environments require motion planners that can rapidly adapt to environmental changes while maintaining safety and efficiency. Frequent replanning is inevitable in such scenarios as obstacle configurations evolve and previously feasible trajectories become invalid. However, real-time replanning remains challenging for many applications due to the high computational cost of collision checking and graph maintenance, especially when edge evaluations are expensive or when the environment changes continuously. The paper introduces an asymptotically optimal lifelong sampling-based path planning algorithm that combines the merits of lifelong planning algorithms and lazy search algorithms for rapid replanning in dynamic environments where edge evaluation is expensive. The algorithm maintains an incremental search graph which is reused throughout the entire navigation process. By evaluating only sub-path candidates for the optimal solution, the algorithm saves considerable evaluation time and reduces the overall planning cost. It employs a novel informed rewiring cascade to efficiently repair the search tree when the underlying search graph changes. Theoretical analysis indicates that the proposed algorithm converges to the optimal solution as long as sufficient planning time is given. Planning results on robotic systems with SE(3) and R7state spaces in challenging environments highlight the superior performance of the proposed algorithm over various state-of-the-art sampling-based planners in both static and dynamic motion planning tasks. The experiment of planning for a Turtlebot 4 operating in a dynamic environment with several moving pedestrians further verifies the feasibility and advantages of the proposed algorithm.

Loop closure detection is important for simultaneous localization and mapping (SLAM), which associates current observations with historical keyframes, achieving drift correction and global relocalization. However, a falsely detected loop can be fatal, and this is especially difficult in repetitive environments where appearance-based features fail due to the high similarity. Therefore, verifying a loop closure is a critical step to avoid false-positive detections. Existing works in loop closure verification predominantly focus on learning invariant appearance features, neglecting the prior knowledge of the robot’s spatial-temporal motion cue, i.e., trajectory. In this article, we propose ROVER, a loop closure verification method that leverages the historical trajectory as a prior constraint to reject false loops in challenging repetitive environments. For each loop candidate, it is first used to estimate the robot trajectory with pose-graph optimization. This trajectory is then submitted to a scoring scheme that assesses its compliance with the trajectory without the loop, which we refer to as the trajectory prior constraint (TPC), to determine if the loop candidate should be accepted. Benchmark comparisons and real-world experiments demonstrate the effectiveness of the proposed method. Furthermore, we integrate ROVER into state-of-the-art SLAM systems to verify its robustness and efficiency.

Human-robot interaction (HRI) encompasses a wide range of collaborative tasks, with handover being one of the most fundamental. As robots become more integrated into human environments, the potential for service robots to assist in handing objects to humans is increasingly promising. In robot-to-human (R2H) handover, selecting the optimal grasp is crucial for success, as it requires avoiding interference with the human's preferred grasp region and minimizing intrusion into their workspace. Existing methods either inadequately consider geometric information or rely on data-driven approaches, which often struggle to generalize across diverse objects. To address these limitations, we propose a novel zero-shot system that combines semantic and geometric information to generate optimal handover grasps. Our method first identifies grasp regions using semantic knowledge from vision-language models (VLMs) and, by incorporating customized visual prompts, achieves finer granularity in region grounding. A grasp is then selected based on grasp distance and approach angle to maximize human ease and avoid interference. We validate our approach through ablation studies and real-world comparison experiments. Results demonstrate that our system improves handover success rates and provides a more user-preferred interaction experience. Videos, appendixes and more are available at https://sites.google.com/view/vlm-handover.

Sampling-based algorithms are widely used for motion planning in high-dimensional configuration spaces. However, due to low sampling efficiency, their performance often diminishes in complex configuration spaces with narrow corridors. Existing approaches address this issue using handcrafted or learned heuristics to guide sampling toward useful regions. Unfortunately, these strategies often lack generalizability to various problems or require extensive prior training. In this paper, we propose a simple yet efficient sampling-based planning framework along with its bidirectional version that overcomes these issues by integrating different levels of planning granularity. Our approach probes configuration spaces with uniform random samples at varying resolutions and explores these multi-resolution samples online with a bias towards sparse samples when traveling large free configuration spaces. By seamlessly transitioning between sparse and dense samples, our approach can navigate complex configuration spaces while maintaining planning speed and completeness. The simulation results demonstrate that our approach outperforms several state-of-the-art sampling-based planners in SE(2), SE(3), and R14with challenging terrains. Furthermore, experiments conducted with the Franka Emika Panda robot operating in a constrained workspace provide additional evidence of the superiority of the proposed method.

The bundle adjustment (BA) algorithm is a widely used nonlinear optimization technique in simultaneous localization and mapping (SLAM) systems. By leveraging the co-view relationships of landmarks from multiple perspectives, the BA method constructs a joint estimation model for both poses and landmarks, enabling the system to generate refined maps and reduce front-end localization errors. However, exploring a robust LiDAR BA estimator and achieving accurate solutions is a challenge. In this work, firstly we propose a novel mean square group metric (MSGM) to build the optimization objective of the LiDAR BA algorithm. This metric applies a mean square transformation to uniformly process the measurements of plane landmarks during one sampling period. The transformed metric ensures scale interpretability and does not require a time-consuming point-by-point calculation. Secondly, by integrating a robust kernel function, the metrics involved in the BA algorithm are reweighted, thus enhancing the robustness of the solution process. Thirdly, based on the proposed robust LiDAR BA model, we derived an explicit second-order estimator (RSO-BA). This estimator employs analytical formulas for Hessian and gradient calculations, ensuring the precision of the BA solution. Finally, we verify the merits of the proposed RSO-BA estimator against existing implicit second-order and explicit approximate second-order estimators using publicly available datasets and physical experiments. The experimental results demonstrate that the RSO-BA estimator outperforms its counterparts in terms of registration accuracy and robustness, particularly in dynamic or complex unstructured environments. Note to Practitioners—The motivation of this paper is to develop a novel LiDAR bundle adjustment (BA) algorithm that ensures accurate and consistent 3D scene modeling. Currently, most LiDAR BA algorithms use “group” processing to construct cost metrics, aiming to reduce the computational complexity of point-by-point operations. However, the cost metrics of these approaches lack scale interpretability, which makes it difficult to incorporate robust kernel functions into the model design, ultimately weakening the system’s robustness and reducing estimation accuracy. To address these issues, we propose a mean square group metric (MSGM) that considers the number of measurement points, to construct the optimization objective for the LiDAR BA (RSO-BA) problem. In each optimization iteration, a robust kernel function reweights each metric to ensure robustness in the solution. Additionally, we derive the analytical Hessian matrix and gradient vector required for solving the RSO-BA, with the inclusion of second-order terms enhancing estimation accuracy. The proposed method can be directly applied to the mobile robot system for automatic driving, home service and unmanned security applications. Furthermore, the proposed algorithm can be extended to sensors such as depth cameras, which provide direct depth information, for broader practical applications.

Robots are increasingly deployed in dynamic and crowded environments, such as urban areas and shopping malls, where efficient and robust navigation is crucial. Traditional risk-based motion planning algorithms face challenges in such scenarios due to the lack of a well-defined search region, leading to inefficient exploration in irrelevant areas. While bi-directional and multi-directional search strategies can improve efficiency, they still result in significant unnecessary exploration. This article introduces the Neural Adaptive Multi-directional Risk-based Rapidly-exploring Random Tree (NAMR-RRT) to address these limitations. NAMR-RRT integrates neural network-generated heuristic regions to dynamically guide the exploration process, continuously refining the heuristic region and sampling rates during the planning process. This adaptive feature significantly enhances performance compared to neural-based methods with fixed heuristic regions and sampling rates. NAMR-RRT improves planning efficiency, reduces trajectory length, and ensures higher success by focusing the search on promising areas and continuously adjusting to environments. The experiment results from both simulations and real-world applications demonstrate the robustness and effectiveness of our proposed method in navigating dynamic environments. A website about this work is available athttps://sites.google.com/view/namr-rrt. Note to Practitioners— The growing demand for autonomous robots to navigate efficiently and robustly in dynamic, crowded environments like public areas has motivated this work. Traditional risk-based motion planning algorithms often suffer from unfocused search processes, leading to inefficient exploration and performance bottlenecks. This article introduces the NAMR-RRT algorithm to address these issues by integrating neural network-generated heuristic regions to guide the search process. NAMR-RRT adaptively updates both the heuristic region and sampling rate during the planning process, allowing it to focus on more promising areas and dynamically adjust to environmental changes. Unlike conventional methods relying on random exploration, NAMR-RRT improves efficiency by focusing searches in regions more likely to lead to the feasible path, thereby reducing trajectory length and enhancing overall performance. This approach is valuable for mobile robots operating in human-robot coexisting environments, where dynamic adaptability and efficient navigation are critical. The experiment results demonstrate that NAMR-RRT provides a reliable and efficient solution for motion planning in such complex scenarios.

Advancements in intelligent vehicle technologies and autonomous driving are now used for tasks like collecting luggage trolleys at airports. The robotic autonomous luggage trolley collection system employs robots to gather and transport scattered luggage trolleys. However, existing methods for detecting and locating luggage trolleys often fail when they are not fully visible. To address this, we introduce the hierarchical progressive perception system, which enhances the detection and localization of luggage trolleys under partial occlusion. The proposed system integrates the hierarchical process structure and progressive perception strategy. This innovative structure processes the luggage trolley’s position and orientation separately. It can accurately determine the luggage trolley’s position with just one well-detected keypoint and estimate its orientation when partially occluded. Once the luggage trolley’s initial pose is detected, the progressive perception strategy continuously refines this information until the robot begins grasping. The proposed system only needs RGB images for labeling and training, eliminating the need for complex data collection and annotation. The experiments on detection and localization demonstrate that the proposed system is more reliable under partial occlusion compared to existing methods. Its effectiveness and robustness have also been confirmed through practical tests in actual luggage trolley collection tasks. A website about this work is available at https://sites.google.com/view/robot-perception.

Using underwater robots instead of humans for the inspection of coastal piers can enhance efficiency while reducing risks. A key challenge in performing these tasks lies in achieving efficient and rapid path planning within complex environments. Sampling-based path planning methods, such as Rapidly-exploring Random Tree* (RRT*), have demonstrated notable performance in high-dimensional spaces. In recent years, researchers have begun designing various geometry-inspired heuristics and neural network-driven heuristics to further enhance the effectiveness of RRT*. However, the performance of these general path planning methods still requires improvement when applied to highly cluttered underwater environments. In this paper, we propose PierGuard, which combines the strengths of bidirectional search and neural network-driven heuristic regions. We design a specialized neural network to generate high-quality heuristic regions in cluttered maps, thereby improving the performance of the path planning. Through extensive simulation and real-world ocean field experiments, we demonstrate the effectiveness and efficiency of our proposed method compared with previous research. Our method achieves approximately 2.6 times the performance of the state-of-the-art geometric-based sampling method and nearly 4.9 times that of the state-of-the-art learning-based sampling method. Our results provide valuable insights for the automation of pier inspection and the enhancement of maritime safety. (Video1).

Trajectory planning for quadrotors in cluttered environments has been challenging in recent years. While many trajectory planning frameworks have been successful, there still exists potential for improvements, particularly in enhancing the speed of generating efficient trajectories. In this paper, we present a novel hierarchical trajectory planning framework to reduce computational time and memory usage called MINER-RRT*, which consists of two main components. First, we propose a sampling-based path planning method boosted by neural networks, where the predicted heuristic region accelerates the convergence of rapidly-exploring random trees. Second, we utilize the optimal conditions derived from the quadrotor’s differential flatness properties to construct polynomial trajectories that minimize control effort in multiple stages. Extensive simulation and real-world experimental results demonstrate that, compared to several state-of-the-art (SOTA) approaches, our method can generate high-quality trajectories with better performance in 3D cluttered environments (https://youtu.be/fXuuMRX19q0). Note to Practitioners—The motivation is the problem of planning trajectories for quadrotor autonomous flight in 3D cluttered and complex scenarios such as wild forest exploration and subterranean environment search-and-rescue. Sampling-based path planning methods are suitable for dealing with the complexity of the physical environment but are not convenient for computing dynamics and their differentials. Optimization-based trajectory generation methods are appropriate for handling various high-order constraints but rely on high-quality initial path solutions. Therefore, this paper combines the advantages of the two methods to propose a novel trajectory planning framework that can generate high-quality trajectories for quadrotors faster than many previous algorithms. We conduct numerous simulations and real-world experiments to verify that our method can be effectively deployed in real scenarios and empower quadrotors for complex autonomous tasks in the future.

In this paper, we present a simultaneous exploration and object search framework for the application of autonomous trolley collection. For environment representation, a task-oriented environment partitioning algorithm is presented to extract diverse information for each sub-task. First, LiDAR data is classified as potential objects, walls, and obstacles after outlier removal. Segmented point clouds are then transformed into a hybrid map with the following functional components: object proposals to avoid missing trolleys during exploration; room layouts for semantic space segmentation; and polygonal obstacles containing geometry information for efficient motion planning. For exploration and simultaneous trolley collection, we propose an efficient exploration-based object search method. First, a traveling salesman problem with precedence constraints (TSP-PC) is formulated by grouping frontiers and object proposals. The next target is selected by prioritizing object search while avoiding excessive robot backtracking. Then, feasible trajectories with adequate obstacle clearance are generated by topological graph search. We validate the proposed framework through simulations and demonstrate the system with real-world autonomous trolley collection tasks.

Continuum robots, characterized by their high flexibility and infinite degrees of freedom (DoFs), have gained prominence in applications such as minimally invasive surgery and hazardous environment exploration. However, the intrinsic complexity of continuum robots requires a significant amount of time for their motion planning, posing a hurdle to their practical implementation. To tackle these challenges, efficient motion planning methods such as Rapidly Exploring Random Trees (RRT) and its variant, RRT*, have been employed. This paper introduces a unique RRT*-based motion control method tailored for continuum robots. Our approach embeds safety constraints derived from the robots’ posture states, facilitating autonomous navigation and obstacle avoidance in rapidly changing environments. Simulation results show efficient trajectory planning amidst multiple dynamic obstacles and provide a robust performance evaluation based on the generated postures. Finally, preliminary tests were conducted on a two-segment cable-driven continuum robot prototype, confirming the effectiveness of the proposed planning approach. This method is versatile and can be adapted and deployed for various types of continuum robots through parameter adjustments.

Scoliosis diagnosis and assessment depend largely on the measurement of the Cobb angle in spine X-ray images. With the emergence of deep learning techniques that employ landmark detection, tilt prediction, and spine segmentation, automated Cobb angle measurement has become increasingly popular. However, these methods encounter difficulties such as high noise sensitivity, intricate computational procedures, and exclusive reliance on a single type of morphological information. In this paper, we introduce the Multiple Morphology-Aware Network (MMA-Net), a novel framework that improves Cobb angle measurement accuracy by integrating multiple spine morphology as attention information. In the MMA-Net, we first feed spine X-ray images into the segmentation network to produce multiple morphological information (spine region, centerline, and boundary) and then concatenate the original X-ray image with the resulting segmentation maps as input for the regression module to perform precise Cobb angle measurement. Furthermore, we devise joint loss functions for our segmentation and regression network training, respectively. We evaluate our method on the AASCE challenge dataset and achieve superior performance with the SMAPE of 7.28% and the MAE of 3.18°, indicating a strong competitiveness compared to other outstanding methods. Consequently, we can offer clinicians automated, efficient, and reliable Cobb angle measurement.

Adaptively Informed Trees (AIT*) is an algorithm that uses the problem-specific heuristic to avoid unnecessary searches, which significantly improves its performance, especially when collision checking is expensive. However, the heuristic estimation in AIT* consumes lots of computational resources, and its asymmetric bidirectional searching strategy cannot fully exploit the potential of the bidirectional method. In this article, we propose an extension of AIT* called BiAIT*. Unlike AIT*, BiAIT* uses symmetrical bidirectional search for both the heuristic and space searching. The proposed method allows BiAIT* to find the initial solution faster than AIT*, and update the heuristic with less computation when a collision occurs. We evaluated the performance of BiAIT* through simulations and experiments, and the results show that BiAIT* can find the solution faster than state-of-the-art methods. We also analyze the reasons for the different performances between BiAIT* and AIT*. Furthermore, we discuss two simple but effective modifications to fully exploit the potential of the adaptively heuristic method.Note to Practitioners—This work is inspired by the adaptively heuristic method and the symmetrical bidirectional searching method. The article introduces a novel algorithm that uses the symmetrical bidirectional method to calculate the adaptive heuristic and efficiently search the state space. The problem-specific heuristic in BiAIT* is derived from a lazy-forward tree and a lazy-reverse tree, which are constructed without collision checking. The lazy-forward and lazy-reverse trees are enabled to meet in the middle, thus generating the effective and accurate heuristic. In BiAIT*, the lazy-forward and lazy-reverse trees share heuristic information and jointly guide the growth of the forward and reverse trees, which conduct collision checking and guarantee the feasibility of their edges. Compared with state-of-the-art methods, BiAIT* finds the initial heuristic and updates the heuristic more quickly. The proposed algorithm can be applied to industrial robots, medical robots, or service robots to achieve efficient path planning. The implementation of BiAIT* is available at https://github.com/Licmjy-CU/BiAITstar.

Balancing the trade-off between safety and efficiency is of significant importance for path planning under uncertainty. Many developed risk-aware path planners explicitly limit the probability of collision to an acceptable bound in uncertain environments. However, convex obstacles or Gaussian uncertainties are usually assumed to make the problem tractable in the existing method. These assumptions limit the generalization and application of path planners in real-world implementations. In this article, we propose to apply deep learning methods to the sampling-based planner, developing a novel risk bounded near-optimal path planning algorithm named neural risk-aware RRT (NR-RRT). Specifically, a deterministic risk contours map is maintained by perceiving the probabilistic nonconvex obstacles, and a neural network sampler is proposed to predict the next most-promising safe state. Furthermore, the recursive divide-and-conquer planning and bidirectional search strategies are used to accelerate the convergence to a near-optimal solution with guaranteed bounded risk. Worst-case theoretical guarantees can also be proven owing to a standby safety guaranteed planner utilizing a uniform sampling distribution. Simulation experiments demonstrate that the proposed algorithm outperforms the state-of-the-art for finding risk bounded low-cost paths in uncertain nonconvex environments with seen and unseen scene layouts. Note to Practitioners—This article is motivated by developing an efficient risk-aware path planner that can quickly find risk bounded solutions in uncertain nonconvex environments for practical applications, such as autonomous vehicles and search-and-rescue robots. Sampling-based planning approaches such as rapidly-exploring random tree (RRT) and its variants are popular for their good performance in exploring the state space. However, it is quite time-consuming to look for risk bounded paths in uncertain environments, especially under nonconvex and non-Gaussian constraints. The initial paths are often of poor quality. Therefore, we propose the NR-RRT algorithm to rapidly find near-optimal solutions with guaranteed bounded risk. It utilizes an informed bidirectional search strategy after having past experiences in the challenging environments. It can be applied in not only seen uncertain scenarios but also those have unseen scene layouts different from the training scenarios. However, the algorithm cannot handle the problem in environments that contain entirely unseen obstacles. In future research, we will address the problem of planning under robot model uncertainty.

Building a general and efficient path planning framework in uncertain nonconvex environments is challenging due to the safety constraints and complex configuration. Traditional avenues usually involve convexifying obstacles and presume Gaussian distribution, which are not universal. Meanwhile, the fast convergence of high-quality solutions is not guaranteed. Therefore, we develop a novel neural risk-bounded path planner to quickly find near-optimal solutions that have an acceptable collision probability in the complex environments. Firstly, we retrieve the nonconvex obstacles with arbitrary probabilistic uncertainties in the form of a deterministic point cloud map. A neural network sampler encodes it into a latent embedding and is trained with sufficient expert demonstrations, predicting states in the potential subspace. We construct a neural cost estimator to select the best informed state from those samples. Then, we recursively use the simple yet effective neural networks to march toward the start and goal bidirectionally. The collision risk of the intermediate connections is verified based on sum-of-squares optimization. Simulation results show that our approach significantly saves time and resources in finding comparable solutions over the state-of-the-art methods in the seen and unseen challenging environments.Note to Practitioners—More and more robots are deployed in unstructured environments, such as forests and subterranean caves. However, uncertainty in the environment situational awareness usually causes accidents. To quickly generate safe paths without over-conservation in uncertain complex environments, we propose a neural risk-bounded sampling-based path planner. Conventional methods consume lots of computation time and resources to generate satisfactory results. Our learning-based risk-bounded path planning framework can efficiently find paths with a guaranteed risk tolerance avoiding uncertain nonconvex static obstacles. It imitates the expert to generate informed states in a subspace that potentially contains the optimal solution. In practice, we need to formulate the observed uncertain obstacle at a grid map into the polynomial containing random variables and determine their probability distributions.

Reorienting objects by using supports is a practical yet challenging manipulation task. Owing to the intricate geometry of objects and the constrained feasible motions of the robot, multiple manipulation steps are required for object reorientation. In this work, we propose a pipeline for predicting various object placements from point clouds. This pipeline comprises three stages: a pose generation stage, followed by a pose refinement stage, and culminating in a placement classification stage. We also propose an algorithm to construct manipulation graphs based on point clouds. Feasible manipulation sequences are determined for the robot to transfer object placements. Both simulated and real-world experiments demonstrate that our approach is effective. The simulation results underscore our pipeline’s capacity to generalize to novel objects in random start poses. Our predicted placements exhibit a 20% enhancement in accuracy compared to the state-of-the-art baseline. Furthermore, the robot finds feasible sequential steps in the manipulation graphs constructed by our algorithm to accomplish object reorientation manipulation.Note to Practitioners—Object reorientation is a prevalent manipulation task in both domestic and industrial manufacturing scenarios. Extrinsic supporting items are often used to provide diverse object placements that allow for feasible grasp configurations for robotic manipulation. In previous methods, utilizing mesh models of objects was necessary to ascertain stable placements and construct manipulation graphs. In this work, we propose a data-driven approach to predict various object placements conditioned on point clouds. Moreover, we use predicted point cloud placements to construct manipulation graphs, which facilitate collision-free pick-and-place steps to reorient objects. Our approach demonstrates the capacity to generalize to novel objects. In future work, we will enhance the performance of our pipeline by optimizing the distance metric used for measuring pose discrepancies and improving the classifier model.

The utilization of cloud infrastructure and its extensive range of Internet-accessible resources holds significant potential for advancing intelligent transportation and robotics. Over the past two decades, interest in cloud-edge collaborative simultaneous localization and mapping (SLAM) has grown markedly. Consequently, a comprehensive review of current trends in this field is crucial for both novice and experienced researchers. This paper examines robots and automation systems that rely on network-based data or code, particularly in the context of SLAM development. Applying SLAM to mobile robots with limited computing power is essential for achieving autonomous navigation, and cloud-edge collaborative SLAM has emerged as an efficient solution. The review is structured around four key benefits of cloud-edge collaborative SLAM: Assisted Cloud Computing, which provides access to cloud computation and reduces the burden on edge devices; Total Cloud Computing, where the majority of computation is offloaded to the cloud, while edge devices primarily handle sensing and low-cost pre-processing; Data Storage, enabling access to large datasets, such as high-resolution environment maps and extensive training datasets, enhancing overall performance; and Data Transmission, involving cloud-edge communication for efficient data transfer and data association. Additionally, we address the challenges in existing work and the development of asynchronous collaboration and implicit representation transmission, which could mitigate transmission latency in communication-constrained environments. We believe that this review will bridge the gap between SLAM systems and deployed robotic systems, promoting the advancement of cloud-edge collaborative SLAM.

Planning a collision-free path efficiently among obstacles is crucial in robotics. Conventional one-shot unidirectional path planning algorithms work well in the static environment, but cannot respond to the environment changes timely in the dynamic environment. To tackle this issue and improve the search efficiency, we propose a bidirectional incremental search method, Bidirectional Lifelong Planning A* (BLPA*), which searches in the forward and backward directions and performs incremental search bidirectionally when the environment changes. Furthermore, inspired by the robot perception range limitation and BLPA*, we propose the fractional bidirectional D* Lite (fBD* Lite(dp)), which constraints the forward search to the robot perception range and uses the backward search to expand the rest area. Our simulation results demonstrate BLPA* and mD* Lite(dp) can achieve superior performance in the dynamic environment. It reveals that the bidirectional incremental search strategy can be a general and efficient technique for graph-search-based robot path planning methods.

Cooperative object transportation using multiple robots has been intensively studied in the control and robotics literature, but most approaches are either only applicable to omnidirectional robots or lack a complete navigation and decision-making framework that operates in real time. This paper presents an autonomous nonholonomic multi-robot system and an end-to-end hierarchical autonomy framework for collaborative luggage trolley transportation. This framework finds kinematic-feasible paths, computes online motion plans, and provides feedback that enables the multi-robot system to handle long lines of luggage trolleys and navigate obstacles and pedestrians while dealing with multiple inherently complex and coupled constraints. We demonstrate the designed collaborative trolley transportation system through practical transportation tasks, and the experiment results reveal their effectiveness and reliability in complex and dynamic environments. (Video11Video demonstration: https://youtu.be/efnPERm0Rco.)

Sampling-based path planning algorithms usually implement uniform sampling methods to search the state space. However, uniform sampling may lead to unnecessary exploration in many scenarios, such as the environment with a few dead ends. Our previous work proposes to use the promising region to guide the sampling process to address the issue. However, the predicted promising regions are often disconnected, which means they cannot connect the start and goal states, resulting in a lack of probabilistic completeness. This work focuses on enhancing the connectivity of predicted promising regions. Our proposed method regresses the connectivity probability of the edges in the x and y directions. In addition, it calculates the weight of the promising edges in loss to guide the neural network to pay more attention to the connectivity of the promising regions. We conduct a series of simulation experiments, and the results show that the connectivity of promising regions improves significantly. Furthermore, we analyze the effect of connectivity on sampling-based path planning algorithms and conclude that connectivity plays an essential role in maintaining algorithm performance.Note to Practitioners—This work is derived from the promising region prediction for sampling-based path planning. The sampling-based path planning methods have been widely used in robotics due to their efficiency. To further improve the efficiency of these algorithms, sampling in the promising region predicted by a neural network is introduced into the sampling procedure. However, the connectivity of the promising region has yet to be considered, and it will affect the performance of the algorithms in several aspects. To demonstrate this problem, we compare the performance of the neural heuristic algorithms under different connectivity statuses in this paper. Furthermore, to enhance the connectivity of the predicted promising region, the novel prediction output and loss function are proposed. The simulation results show improvements in the algorithms after utilizing our method.

Autonomous mobile manipulation robots that can collect trolleys are widely used to liberate human resources and fight epidemics. Most prior robotic trolley collection solutions only detect trolleys with 2D poses or are merely based on spe-cific marks and lack the formal design of planning algorithms. In this paper, we present a novel mobile manipulation system with applications in luggage trolley collection. The proposed system integrates a compact hardware design and a progressive perception and planning framework, enabling the system to efficiently and robustly collect trolleys in dynamic and complex environments. For perception, we first develop a 3D trolley detection method that combines object detection and keypoint estimation. Then, a docking process in a short distance is achieved with an accurate point cloud plane detection method and a novel manipulator design. On the planning side, we formulate the robot's motion planning under a nonlinear model predictive control framework with control barrier functions to improve obstacle avoidance capabilities while maintaining the target in the sensors' field of view at close distances. We demonstrate our design and framework by deploying the system on actual trolley collection tasks, and their effectiveness and robustness are experimentally validated. (Video11Video demonstration: https://youtu.be/6SwjgGvRtno.)

In this article, based on the rapidly-exploring random tree (RRT), we propose a novel and efficient motion planning algorithm using bidirectional RRT search. First, a RRT extend function is used to organize the sampled states under kinodynamic constraints. Meanwhile, the bidirectional search strategy is implemented to grow a forward tree and backward tree simultaneously in the tree extension process. When these two trees meet each other, the backward tree will act as a heuristic to guide the forward tree to continuously grow toward the goal state, where the algorithm switches to unidirectional search mode. Therefore, the two-point boundary value problem (BVP) in the connection process is avoided, and the extension process gets much accelerated. We also prove that probabilistic completeness is guaranteed. Numerical simulations are conducted to demonstrate that the proposed algorithm performs much better than the state-of-the-art algorithms in different environments.Note to Practitioners—The motivation of this work is to develop an efficient sampling-based motion planning algorithm for mobile robots. Conventional sampling-based algorithms are time-consuming to find a feasible solution under differential constraints. When applying bidirectional search strategy to improve them, the complex 2-point BVP is required to solve. In this article, the backward free is regarded as a heuristic to guide the tree growth. On the one hand, the advantage of bidirectional search is retained. On the other hand, the 2-point BVP is avoided. Therefore, the bidirectional-unidirectional technique can achieve efficient robot motion planning. The proposed algorithm can be extended to other specified sampling-based algorithms to further improve their performance. Besides, it can be also applied to autonomous driving, service robot and medical robots to achieve efficient motion planning.

Robot path planning in 3D space is a challenging problem for its complex configuration. Sampling-based algorithms have gained great success in solving path planning problems in 3D space, but the quality of the initial path is not guaranteed and the convergence to the optimal solution is slow. To address these problems, in this article, we present a novel sampling-based path planning framework enhanced by the deep neural network (DNN) with applications to 3D space. In the proposed framework, we first train the DNN with a number of successful path planning cases in 3D space. Then the DNN is utilized to predict the promising region where the feasible path probably exists for a given path planning problem. This predicted promising region serves as a nonuniform sampling heuristic to bias the sampling process of the path planner. In this way, the path planner can focus on the promising region in the exploration and exploitation process so that the path planning speed gets accelerated. We conduct numerical simulations to evaluate the performance of the proposed algorithm and the results show that it can perform much better than conventional path planning algorithms. Furthermore, we also investigate the performance of different DNN architectures for path planning in 3D space. Note to Practitioners—In this work, we aim to provide an efficient learning-based method to accelerate the robot path planning process in 3D space. Conventional path planning algorithms need to perceive the environment first, and then implement a series of calculations such as collision checking and data storing to generate a feasible path. When facing complex and high-dimensional environments, they do not perform well. But the proposed neural network method in this article can predict the promising region where the feasible path exists for any given environment. This prediction result is used to guide the path planning process so that the algorithm performance can get significantly improved. Apart from sampling-based algorithms, the proposed neural network model can also be extended to other types of path planning algorithms.

In this article, a two-level planner is proposed to provide a solution to the autonomous luggage trolley collection problem at the airport. In the higher level planner, a decision-making problem is tackled where a sequence of luggage trolleys is determined with which the robot can collect them one by one. Based on the traditional traveling salesman problem (TSP), this decision-making problem is formulated as an open dynamic traveling salesman problem with fixed start (ODTSP-FS). Incorporating the modified transition rule, elitist global update rule, and additional local update rule, an efficient algorithm is proposed to handle this decision-making problem. The experimental results demonstrate that the proposed algorithm achieves fast convergence and smaller cost compared with the state-of-the-art algorithms. In the lower level planner, based on the pipeline of rapid-exploring random tree (RRT) scheme, a novel real-time path planning algorithm is introduced, which can adjust itself to moving obstacles and moving targets by retaining the whole tree and using two rewiring strategies. Finally, the proposed two-level planner is evaluated in a simulation environment similar to the airport to validate the effectiveness and efficiency of the proposed algorithm.

As a common cause of anemia and gastrointestinal bleeding, angiodysplasia (AD) diagnosis in wireless capsule endoscopy (WCE) images is important in clinical. Current manual review requires undivided concentration of the gastroenterologists, which is laborious and time-consuming. The development of computational methods that can assist automated diagnosis of angiodysplasia is highly desirable. In this paper, we present a new approach, ADNet, for angiodysplasia segmentation using convolutional neural networks (CNNs). Compared with previous learning strategies, ADNet gains accuracy from attentionguided and domain-adversarial training via a multibranch CNN architecture. Specifically, the core branch is constructed for AD segmentation in a fully convolutional manner. Then we propose an attention module embedded in the attention branch to enhance network feature learning, which allows ADNet to focus on the most informative and AD relevant regions while processing. Furthermore, an adaptation branch is built to learn domain-invariant features by adversarial training, aiming to improve the performance when datasets are expanded while preventing the degradation induced by the variations in WCE image acquisition. ADNet is evaluated using two WCE datasets with angiodysplasia and the results show the accuracy gains we obtain, where the state-of-the-art segmentation performance on the public dataset of GIANA’17 is achieved.

Robot path planning is difficult to solve due to the contradiction between the optimality of results and the complexity of algorithms, even in 2D environments. To find an optimal path, the algorithm needs to search all the state space, which costs many computation resources. To address this issue, we present a novel recurrent generative model (RGM), which generates efficient heuristic to reduce the search efforts of path planning algorithms. This RGM model adopts the framework of general generative adversarial networks (GAN), which consists of a novel generator that can generate heuristic by refining the outputs recurrently and two discriminators that check the connectivity and safety properties of heuristic. We test the proposed RGM module in various 2D environments to demonstrate its effectiveness and efficiency. The results show that, compared with a model without recurrence, the RGM successfully generates appropriate heuristic in both seen and new unseen maps with higher accuracy, demonstrating the good generalization ability of the RGM model. We also compare the rapidly-exploring random tree star (RRT*) with generated heuristic and the conventional RRT* in four different maps, showing that the generated heuristic can guide the algorithm to efficiently find both initial and optimal solutions in a faster and more efficient way.

Interactions with either environments or expert policies during training are needed for most of the current imitation learning (IL) algorithms. For IL problems with no interactions, a typical approach is Behavior Cloning (BC). However, BC-like methods tend to be affected by distribution shift. To mitigate this problem, we come up with a Robust Model-Based Imitation Learning (RMBIL) framework that casts imitation learning as an end-to-end differentiable nonlinear closed-loop tracking problem. RMBIL applies Neural ODE to learn a precise multi-step dynamics and a robust tracking controller via Nonlinear Dynamics Inversion (NDI) algorithm. Then, the learned NDI controller will be combined with a trajectory generator, a conditional VAE, to imitate an expert’s behavior. Theoretical derivation shows that the controller network can approximate an NDI when minimizing the training loss of Neural ODE. Experiments on Mujoco tasks also demonstrate that RMBIL is competitive to the state-of-the-art generative adversarial method (GAIL) and achieves at least 30% performance gain over BC in uneven surfaces.

The sampling-based partial motion planning algorithm has come into widespread application in dynamic mobile robot navigation due to its low calculation costs and excellent performance in avoiding obstacles. However, when confronted with complicated scenarios, the motion planning algorithms are easily caught in traps. In order to solve this problem, this paper proposes a knowledge-based fast motion planning algorithm based on Risk-RRT, which guides motion planning by constructing a topological feature tree and generating a heuristic path from the tree. Firstly, an online topological feature learning method is proposed to simultaneously extract the features during the motion of the robot by means of the dual-channel scale filter and the secondary distance fusion. The learning process is completed until the feature points can represent arbitrary obstacle-free grid points of the whole map. Secondly, the topological feature tree is constructed with environmental feature points and the heuristic motion planning can be carried out on the feature tree. For one map, once the construction of the feature tree finishes, it can be reused as a prior knowledge in the following heuristic motion planning process, which will further improve the efficiency of searching feasible paths. The experimental results demonstrate that our proposed method can remarkably reduce the time taken to find a heuristic path and enhance the success rate of navigation in trapped environments.

In this paper, we propose a novel path planning algorithm for the nonholonomic multiple mobile robot system with applications to a robotic autonomous luggage trolley collection system at airports. We consider this path planning algorithm as a Multiple Traveling Salesman Problem (MTSP). Our path planning algorithm consists of three parts. First, we use the Minimum Spanning Tree (MSP) algorithm to divide the MTSP into a number of independent TSPs, which achieves the task assignment for each mobile robot. Secondly, we implement a closed-loop forward control policy based on the kinematic model of the mobile robot to get a feasible and smooth path. The control cost of the path is used as the new metric in solving the TSPs. Finally, in order to adapt to our case, we modify the TSP as an Open Dynamic Traveling Salesman Problem with Fixed Start (ODTSP-FS) and implement an ant colony algorithm to achieve the path planning for each mobile robot. We evaluate our algorithm with simulation experiments and the experimental results demonstrate that our algorithm can quickly generate feasible and smooth paths for each robot while satisfying the nonholonomic constraints.

Rapidly random-exploring tree (RRT) and its variants are very popular due to their ability to quickly and efficiently explore the state space. However, they suffer sensitivity to the initial solution and slow convergence to the optimal solution, which means that they consume a lot of memory and time to find the optimal path. It is critical to quickly find a short path in many applications such as the autonomous vehicle with limited power/fuel. To overcome these limitations, we propose a novel optimal path planning algorithm based on the convolutional neural network (CNN), namely the neural RRT* (NRRT*). The NRRT* utilizes a nonuniform sampling distribution generated from a CNN model. The model is trained using quantities of successful path planning cases. In this article, we use the A* algorithm to generate the training data set consisting of the map information and the optimal path. For a given task, the proposed CNN model can predict the probability distribution of the optimal path on the map, which is used to guide the sampling process. The time cost and memory usage of the planned path are selected as the metric to demonstrate the effectiveness and efficiency of the NRRT*. The simulation results reveal that the NRRT* can achieve convincing performance compared with the state-of-the-art path planning algorithms. Note to Practitioners-The motivation of this article stems from the need to develop a fast and efficient path planning algorithm for practical applications such as autonomous driving, warehouse robot, and countless others. Sampling-based algorithms are widely used in these areas due to their good scalability and high efficiency. However, the quality of the initial path is not guaranteed and it takes much time to converge to the optimal path. To quickly obtain a high-quality initial path and accelerate the convergence speed, we propose the NRRT*. It utilizes a nonuniform sampling distribution and achieves better performance. The NRRT* can be also applied to other sampling-based algorithms for improved results in different applications.

In a human-robot coexisting environment, it is pivotal for a mobile service robot to arrive at the goal position safely and efficiently. In this article, an elastic band-based rapidly exploring random tree (EB-RRT) algorithm is proposed to achieve real-time optimal motion planning for the mobile robot in the dynamic environment, which can maintain a homotopy optimal trajectory based on current heuristic trajectory. Inspired by the EB method, we propose a hierarchical framework consisting of two planners. In the global planner, a time-based RRT algorithm is used to generate a feasible heuristic trajectory for a specific task in the dynamic environment. However, this heuristic trajectory is nonoptimal. In the dynamic replanner, the time-based nodes on the heuristic trajectory are updated due to the internal contraction force and the repulsive force from the obstacles. In this way, the heuristic trajectory is optimized continuously, and the final trajectory can be proved to be optimal in the homotopy class of the heuristic trajectory. Simulation experiments reveal that compared with two stateof-the-art algorithms, our proposed method can achieve better performance in dynamic environments.

In a human-robot coexisting environment, reaching the target place efficiently and safely is pivotal for a mobile service robot. In this paper, a Risk-based Dual-Tree Rapidly exploring Random Tree (Risk-DTRRT) algorithm is proposed for the robot motion planning in a dynamic environment, which provides a homotopy optimal trajectory on the basis of a heuristic trajectory. A dual-tree framework consisting of an RRT tree and a rewired tree is proposed for the trajectory searching. The RRT tree is a time-based tree, considering the future trajectory predictions of the pedestrians, and this tree is utilized to generate a heuristic trajectory. However, the heuristic trajectory is usually nonoptimal. Then, a line-of-sight (LoS) control checking algorithm is proposed to detect whether two time-based nodes can be rewired with the least cost. On the basis of the LoS control checking algorithm, a tree rewiring algorithm is proposed to optimize the heuristic trajectory. The tree generated in the tree rewiring process is called the rewired tree. The trajectory generated by the Risk-DTRRT algorithm proves to be optimal in the homotopy class of the heuristic trajectory. The navigation run time and the lengths of the planned trajectories are selected to demonstrate the effectiveness of the proposed algorithm. The experimental results in both simulation studies and real-world implementations reveal that our proposed method achieves convincing performance in both static and dynamic environments. Note to Practitioners-This paper is motivated by planning optimized trajectories for the mobile service robots in dynamic environments with pedestrians. In this area, the sampling-based motion planning algorithms have been widely used for their high efficiency and robustness. However, the real-time optimality of the motion planning cannot be guaranteed due to the challenges caused by the moving pedestrians. In this paper, we propose a dual-tree framework to solve this problem. First, a classic Rapidly exploring Random Tree (RRT) is constructed to generate a heuristic trajectory. Then, instead of reconnecting the nodes on the heuristic trajectory directly, a rewired tree is built to optimize the heuristic trajectory. This proposed dual-tree framework can fully exploit the information of the RRT tree and ensure the completeness of the motion planning. The proposed motion planning algorithm also considers the constraints of the nonholonomic mobile robots, and it can be applied in most mobile service robots to improve their motion planning quality.