Service robots are increasingly deployed in diverse and dynamic environments, where both physical layouts and social contexts change over time and across locations. In these unstructured settings, conventional navigation systems that rely on fixed parameters often fail to generalize across scenarios, resulting in degraded performance and reduced social acceptance. Although recent approaches have leveraged reinforcement learning to enhance traditional planners, these methods often fail in real-world deployments due to poor generalization and limited simulation diversity, which hampers effective sim-to-real transfer. To tackle these issues, we present LE-Nav, an interpretable and scene-aware navigation framework that leverages multi-modal large language model reasoning and conditional variational autoencoders to adaptively tune planner hyperparameters. To achieve zero-shot scene understanding, we utilize one-shot exemplars and chain-of-thought prompting strategies. Additionally, a conditional variational autoencoder captures the mapping between natural language instructions and navigation hyperparameters, enabling expert-level tuning. Experiments show that LE-Nav can generate hyperparameters achieving human-level tuning across diverse planners and scenarios. Real-world navigation trials and a user study on a smart wheelchair platform demonstrate that it outperforms state-of-the-art methods on quantitative metrics such as success rate, efficiency, safety, and comfort, while receiving higher subjective scores for perceived safety and social acceptance. Code is available at https://github.com/Cavendish518/LE-Nav.

Neural implicit scene representations have recently shown promising results in dense visual SLAM. However, existing implicit SLAM algorithms are constrained to single-agent scenarios, and fall difficulty in large indoor scenes and long sequences. Existing multi-agent SLAM frameworks cannot meet the constraints of communication bandwidth. To this end, we propose the first distributed multi-agent collaborative SLAM framework with distributed mapping and camera tracking, joint scene representation, intra-to-inter loop closure, and multi-submap fusion. Specifically, our proposed distributed neural mapping and tracking framework only needs peer-to-peer communication, which can greatly improve multi-agent cooperation and communication efficiency. A novel intra-to-inter loop closure method is designed to achieve local (single-agent) and global (multi-agent) consistency. Furthermore, to the best of our knowledge, there is no real-world dataset for NeRF-based/GS-based SLAM that provides both continuous-time trajectories groundtruth and high-accuracy 3D meshes groundtruth. To this end, we propose the first real-world indoor neural slam (INS) dataset covering both single-agent and multi-agent scenarios, ranging from small room to large-scale scenes, with high-accuracy ground truth for both 3D mesh and continuous-time camera trajectory. This dataset can advance the development of the community. Experiments on various datasets demonstrate the superiority of the proposed method in both mapping, tracking, and communication. The dataset and code will be open-source on https://github.com/dtc111111/MNESLAM.

4D imaging radar is a type of low-cost millimeter-wave radar(costing merely 10-20% of lidar systems) capable of providing range, azimuth, elevation, and Doppler velocity information. Accurate extrinsic calibration between millimeter-wave radar and camera systems is critical for robust multimodal perception in robotics, yet remains challenging due to inherent sensor noise characteristics and complex error propagation. This paper presents a systematic calibration framework to address critical challenges through a spatial 3d uncertainty-aware PnP algorithm (3DUPnP) that explicitly models spherical coordinate noise propagation in radar measurements, then compensating for non-zero error expectations during coordinate transformations. Finally, experimental validation demonstrates significant performance improvements over state-of-the-art CPnP baseline, including improved consistency in simulations and enhanced precision in physical experiments. This study provides a robust calibration solution for robotic systems equipped with millimeter-wave radar and cameras, tailored specifically for autonomous driving and robotic perception applications.

Recent research has begun exploring novel view synthesis (NVS) for LiDAR point clouds, aiming to generate realistic LiDAR scans from unseen viewpoints. However, most existing approaches do not reconstruct semantic labels, which are crucial for many downstream applications such as autonomous driving and robotic perception. Unlike images, which benefit from powerful segmentation models, LiDAR point clouds lack such large-scale pre-trained models, making semantic annotation time-consuming and labor-intensive. To address this challenge, we propose SN-LiDAR, a method that jointly performs accurate semantic segmentation, high-quality geometric reconstruction, and realistic LiDAR synthesis. Specifically, we employ a coarse-to-fine planar-grid feature representation to extract global features from multi-frame point clouds and leverage a CNN-based encoder to extract local semantic features from the current frame point cloud. Extensive experiments on SemanticKITTI and KITTI-360 demonstrate the superiority of SN-LiDAR in both semantic and geometric reconstruction, effectively handling dynamic objects and large-scale scenes. Codes will be available on https://github.com/dtc111111/SN-Lidar.

Recent work has shown that 3D Gaussian-based SLAM enables high-quality reconstruction, accurate pose estimation, and real-time rendering of scenes. However, these approaches are built on a tremendous number of redundant 3D Gaussian ellipsoids, leading to high memory and storage costs and slow training speed. To address this limitation, we propose a compact 3D Gaussian Splatting SLAM system that reduces the number and the parameter size of Gaussian ellipsoids. A sliding window-based masking strategy is first proposed to reduce the redundant ellipsoids. Then, a novel geometry codebook-based quantization method is proposed to further compress 3D Gaussian geometric attributes. Robust and accurate pose estimation is achieved by a local-to-global bundle adjustment method with reprojection loss. Extensive experiments demonstrate that our method achieves faster training, rendering speed, and low memory usage while maintaining the state-of-the-art (SOTA) quality of the scene representation.

Deformation of flexible robots can be practically assessed using extension/compression, shear, curvature, and torsion. Sensing based on one or more of the above characteristics enables closed-loop control for delicate tasks that require precision and dexterity. Due to the increasing popularity of flexible robotics in recent years, significant research effort has been directed to this burgeoning field. Although numerous studies have addressed soft sensing technologies, their successful integration into flexible robotic systems remains limited. This article provides a comprehensive review of sensing methods, from multidimensional deformation to the underlying principles of deriving hard-to-measure deformation from surrogate parameters. It focuses on sensing modalities such as strain measurement via piezoelectric, capacitive, resistive, and optical techniques. The applications of deformation sensing in industrial and service robotics are described. Future challenges and potential research issues including resolution, conformability, multifunctionality, crosstalk, and miniaturization are discussed. The need for a synergistic approach across disciplines is highlighted, emphasizing the integration of new materials, microstructures, advanced manufacturing technologies, and state-of-the-art signal processing techniques.

Dense scene reconstruction for photo-realistic view synthesis has various applications, such as VR/AR, and robotics navigation. Existing dense reconstruction methods are primarily designed for small room scenarios, but in practice, the scenes encountered by robots are typically large-scale environments. Most existing methods have difficulties in large-scale scenes due to three core challenges:(a) inaccurate depth input. Depth information is crucial for both scene geometry reconstruction and pose estimation. Accurate depth input is impossible to get in real-world large-scale scenes.(b) inaccurate pose estimation. Existing methods are not robust enough with the growth of cumulative errors in large scenes and long sequences.(c) insufficient scene representation capability. A single global radiance field lacks the capacity to scale effectively to large-scale scenes. To this end, we propose an incremental joint learning framework, which can achieve accurate depth, pose estimation, and large-scale dense scene reconstruction. For depth estimation, a vision transformer-based network is adopted as the backbone to enhance performance in scale information estimation. For pose estimation, a feature-metric bundle adjustment (FBA) method is designed for accurate and robust camera tracking in large-scale scenes and eliminates pose drift. In terms of implicit scene representation, we propose an incremental scene representation method to construct the entire large-scale scene as multiple local radiance fields to enhance the scalability of 3D scene representation. In local radiance fields, we propose a tri-plane based scene representation method to further improve the accuracy and efficiency of scene reconstruction. We conduct extensive experiments on various datasets, including our own collected data, to demonstrate the effectiveness and accuracy of our method in depth estimation, pose estimation, and large-scale scene reconstruction. The code has been open-sourced on https://github.com/dtc111111/incre-dpsr.

In recent years, there have been significant advancements in 3D reconstruction and dense RGB-D SLAM systems. One notable development is the application of Neural Radiance Fields (NeRF) in these systems, which utilizes implicit neural representation to encode 3D scenes. However, the depth images obtained from consumer-grade RGB-D sensors are often sparse and noisy, which poses significant challenges for 3D reconstruction and affects the accuracy of the representation of the scene geometry. Furthermore, existing methods select random pixels for camera tracking, leading to inaccurate localization in real-world indoor environments. To this end, we present NeSLAM, an advanced framework that achieves accurate and dense depth estimation, robust camera tracking, and realistic synthesis of novel views. First, a depth completion and denoising network is designed to provide dense geometry prior and guide the neural implicit representation optimization. Second, we propose a NeRF-based self-supervised feature tracking algorithm for robust real-time tracking. Experiments on various indoor datasets demonstrate the effectiveness and accuracy of the system in reconstruction, tracking quality, and novel view synthesis. Note to Practitioners—Traditional SLAM methods usually use the sparse point cloud to represent the scene, resulting in poor scene representation capability. Our method proposes a neural implicit representation method with depth completion and denoising network and feature tracking method, achieves accurate scene reconstruction and accurate pose estimation in various indoor scenes. The depth completion and denoising network provide accurate depth information associated with depth uncertainty, which is used to improve the geometry consistency. The NeRF-based self-supervised feature tracking method improve the accuracy and robustness for camera tracking. The experimental results demonstrate the accuracy and effectiveness of this method in different scenes.

In this work, we propose an optimization-based trajectory planner for tractor-trailer vehicles on curvy roads. The lack of analytical expression for the trailer’s errors to the center line pose a great challenge to the trajectory planning for tractor-trailer vehicles. To address this issue, we first use geometric representations to characterize the lateral and orientation errors in Cartesian frame, where the errors would serve as the components of the cost function and the road edge constraints within our optimization process. Next, we generate a coarse trajectory to warm-start the subsequent optimization problems. On the other hand, to achieve a good approximation of the continuous-time kinematics, optimization-based methods usually discretize the kinematics with a large sampling number. This leads to an increase in the number of the variables and constraints, thus making the optimization problem difficult to solve. To address this issue, we design a Progressively Increasing Sampling Number Optimization (PISNO) framework. More specifically, we first find a nearly feasible trajectory with a small sampling number to warm-start the optimization process. Then, the sampling number is progressively increased, and the corresponding intermediate Optimal Control Problem (OCP) is solved in each iteration. Next, we further resample the obtained solution into a finer sampling period, and then use it to warm-start the intermediate OCP in next iteration. This process is repeated until reaching a threshold sampling number. Simulation and experiment results show the proposed method exhibits a good performance and less computational consumption over the benchmarks.

Neural implicit scene representations have recently shown encouraging results in dense visual SLAM. However, existing methods produce low-quality scene reconstruction and low-accuracy localization performance when scaling up to large indoor scenes and long sequences. These limitations are mainly due to their single, global radiance field with finite capacity, which does not adapt to large scenarios. Their end-to-end pose networks are also not robust enough with the growth of cumulative errors in large scenes. To this end, we introduce PLGSLAM, a neural visual SLAM system capable of high-fidelity surface reconstruction and robust camera tracking in real-time. To handle large-scale indoor scenes, PLGSLAM proposes a progressive scene representation method which dynamically allocates new local scene representation trained with frames within a local sliding window. This allows us to scale up to larger indoor scenes and improves robustness (even under pose drifts). In local scene representation, PLGSLAM utilizes tri-planes for local high-frequency features with multilayer perceptron (MLP) networks for the low-frequency feature, achieving smoothness and scene completion in unobserved areas. Moreover, we propose local-to-global bundle adjustment method with a global keyframe database to address the increased pose drifts on long sequences. Experimental results demonstrate that PLGSLAM achieves state-of-the-art scene reconstruction results and tracking performance across various datasets and scenarios (both in small and large-scale indoor environments).

Cable-driven continuum robots are widely used for endoluminal intervention because of their dexterity and shape conforming steerability. However, body contact between the continuum robot and its surrounding anatomy is unavoidable, which imposes a potential safety risk, including vessel wall damage or even perforation. This paper presents an approach for body contact estimation of continuum robots with tension-profile sensing of actuation fibers. First, tension-sensing optical fibers with multiple inscribed fiber Bragg grating (FBG) sensors are used for both actuation and in-situ sensing of the continuum robot. Second, a beam theory-based mechanical model considering segmental differences, multiple fiber interactions and external force interactions is established, followed by robust estimation of contact positions and forces. Finally, detailed simulations are conducted to validate the accuracy and effectiveness of the proposed method. Experiments on a notched continuum robot are carried out, and the results show that the proposed approach can effectively recover in-situ segmental actuation forces without the need of explicit modeling of the friction between the fibers and guiding channels. The method enables the estimation of the number of contact points, as well as contact positions and contact forces along the body of the continuum robot.

Tip force sensing for cable-driven continuum robots are vital to provide the force information for safe and reliable human-robot interaction. However, traditional triaxial force sensors usually have a complicated structure occupying its inner lumen, without enough space for additional instrumental tools. To solve this, this paper proposes a fixed and sliding fiber Bragg grating (FBG) sensors-based triaxial force sensing method for cable-driven continuum robots. The fixed FBG sensors are attached to the circumferential surface of continuum robot at the tip and base, and the sliding optical fibers with FBG sensors are located in the actuation channels as the sensing integrated pulling cables. This configuration guarantees a compact structure and large inner lumen. Two five-degreed-of-freedom (5-DOF) electromagnetic (EM) and a 6-DOF EM sensors are assembled to the tip and the base of the robot respectively, which can obtain the pose of the tip with respect to the base. The tip force in three directions can be decoupled using the information of the Bragg wavelength changes and EM sensors. Results show that the mean errors of force sensing along x-direction, y-direction, and z-direction are 4.1%, 4.7%, and 9.8%, respectively. The proposed sensing method does not rely on the elasticity of continuum robot, enabling its wide applicability for other cable-driven pseudo-continuum robots.

Long-term scene changes pose challenges to localization systems using a pre-built map. This paper presents a LiDAR-based system that provides robust localization against those challenges. Our method starts with activation of a mapping process temporarily when global matching towards the pre-built map is unreliable. The temporary map will be merged onto the pre-built map for later localization sessions once reliable matching is obtained again. We further integrate a LiDAR inertial odometry (LIO) to provide motion-compensated LiDAR scans and a reliable pose initial estimate for the global matching module. To generate a smooth real-time trajectory for navigation purposes, we fuse poses from odometry and global matching by solving a pose graph optimization problem. We evaluate our localization system with extensive experiments on the NCLT dataset including a variety of changing indoor and outdoor environments, and the results demonstrate a robust and accurate long-term localization performance. The implementations are open sourced on GitHub11https://github.com/HaisenbergPeng/ROLL.

We consider the uncalibrated vision-based control problem of robotic manipulators in this work. Though lots of approaches have been proposed to solve this problem, they usually require calibration (offline or online) of the camera parameters in the implementation, and the control performance may be largely affected by parameter estimation errors. In this work, we present new fully uncalibrated visual servoing approaches for position control of the 2DOFs planar manipulator with a fixed camera. In the proposed approaches, no camera calibration is required, and numerical optimization algorithms or adaptive laws for parameter estimation are not needed. One benefit of such features is that exponential convergence of the image position errors can be ensured regardless of the camera parameter uncertainties. Generally, existing uncalibrated approaches only can guarantee asymptotical convergence of the position errors. Moreover, different from most existing approaches which assume that the robot motion plane and the image plane are parallel, one of the proposed approaches allows the camera to be installed at a general pose. This also simplifies the controller implementation and improves the system design flexibility. Finally, simulation and experimental results are provided to illustrate the effectiveness of the presented fully uncalibrated visual servoing approaches.

This study proposes a hybrid vision/force control scheme for interaction with the inner surface of the bottle-like object. Based on the geometry of the object, a new generalized constraint called the bottleneck (BN) constraint is proposed, which ensures the tool passes through a fixed 3-D region and avoid collisions with the boundary of the region. To realize the hybrid vision/force control under the BN constraint, a novel dynamic controller is designed inspired by the hierarchical operational space, which can complete the different tasks defined in the decoupled subspace. To enhance the robustness of the algorithm, we develop a data-driven method and an adaptive method to estimate the Jacobian matrix online in force space and image space, respectively. The asymptotic stability of the closed-loop system is rigorously proved by the Lyapunov theory. Experiments are conducted to validate the performance of the proposed method.

Soft robots have been applied widely to various constrained scenarios due to the advantages over traditional rigid manipulators such as softness, deformability and adaptability to constrained surroundings. To make full use of this merit, this paper proposes a method that integrates collision detection, localization and force estimation for a cable-driven soft manipulator without any prior geometrical knowledge of its surroundings. First of all, a collision detection algorithm is presented based upon Cosserat-rod statics by a threshold method through using the cable tension and the shape information, which are obtained by the load cells and the Vicon system, respectively. Secondly, a collision localization and force estimation method is proposed through optimizing the discrepancy between the actual and the theoretical shapes. Finally, experiments are carried out to validate these algorithms. The experimental results demonstrate that the site, the magnitude as well as the direction can be estimated.

Timely fault detection and identification (FDI) of soft manipulators are critical in the design of surgical systems to improve reliability. However, due to the intrinsic compliance of soft manipulators, their end effectors vibrate during the dynamic control process, which introduces noise into the measured signals and makes FDI of soft manipulators challenging. This paper proposes a novel method to accomplish these tasks based on Long Short Term Memory (LSTM) recurrent neural network. Based on LSTM network, a new Attention-based Noise Compensation (ANC) module is proposed to enable the network to filter the noise merged with signals input in a self-supervision manner. Moreover, weighted cross entropy loss is introduced to balance the normal and faulty samples in the training set. Of the 9930 samples presented to the model, 9489 are correctly diagnosed in less than 1.0 second, which implies that the method can learn the spatial and temporal dependence of the signals and distinguish the healthy modes from the faulty ones. Finally, we compare the ANC-based method with the vanilla LSTM method and the state-of-art Bruin et al. method. From the comparison, we conclude that the ANC-based method proposed in this paper not only shortens the time cost of the FDI process but also suppresses the sensitivity of diagnosis results to noise. Source code, pre-trained models and dataset are available on https://github.com/IRMVLab/ANC-LSTM-fault-detection.

This paper puts forward a novel sensor-less fault detection method with only task errors feedback and applies it to visual servoing tasks of soft robot manipulators. The method is developed by introducing a suitably designed endogenous accessory signal (EAS). On the one hand, EAS transforms the change of jacobian matrix led by faults into the change of task errors, which enables the fault to be directly measured and detected; on the other hand, EAS adjusts the state trajectories according to the distance between states and their boundaries, so that state saturation is avoided. To enhance the robustness of the method, we introduce an artificial potential field that keeps the states from the undesired hyperplanes that lead to the loss of effectiveness of the method. Due to the uncalibrated feature point, its coordinates used in control laws and artificial potential filed are unknown. An adaptive algorithm is developed to guarantee the stability of the system and the convergence of the image errors. Experiments are conducted to validate the performance of the proposed method in both healthy and faulty systems.

This article proposes a novel image-based visual servoing for a flexible aerial refueling boom with an eye-in-hand camera. The dynamic model of the flexible refueling boom is established by absolute nodal coordinate formulation (ANCF), after which the dynamic model is decomposed into a slow subsystem and a fast subsystem based on the singular perturbation approach. With respect to slow subsystem, the image feedback is used to control the flexible refueling boom so that the projection of the point marker on the back of the receiver converges to the desired position. With respect to fast subsystem, linear quadratic regulator (LQR) is applied to stabilize the vibration of the flexible refueling boom. The asymptotic convergence of the image error to zero is verified based on the Lyapunov theory. Simulation is used to demonstrate the effectiveness of the proposed method.

Accurate, robust and real-time localization under constrained-resources is a critical problem to be solved. In this paper, we present a new sparse pose-graph visual-inertial SLAM (SPVIS). Unlike the existing methods that are costly to deal with a large number of redundant features and 3D map points, which are inefficient for improving positioning accuracy, we focus on the concise visual cues for high-precision pose estimating. We propose a novel hierarchical quadtree based optical flow tracking algorithm, it achieves high accuracy and robustness within very few concise features, which is only about one fifth features of the state-of-the-art visual-inertial SLAM algorithms. Benefiting from the efficient optical flow tracking, our sparse pose-graph optimization time cost achieves bounded complexity. By selecting and optimizing the informative features in sliding window and local VIO, the computational complexity is bounded, it achieves low time cost in long-term operation. We compare with the state-of-the-art VIO/VI-SLAM systems on the challenging public datasets by the embedded platform without GPUs, the results effectively verify that the proposed method has better real-time performance and localization accuracy.

To make the controller implementation easier and to enhance the system robustness and control performance in the presence of the camera parameter uncertainties, it is very desired to develop vision-based control approaches without any offline or online camera calibration. In this article, we propose a new calibration-free image-based trajectory tracking control scheme for nonholonomic mobile robots with a truly uncalibrated fixed camera. By developing a novel camera-parameter-independent kinematic model, both offline and online camera calibration can be avoided in the proposed scheme, and any knowledge of the camera is not needed in the controller design. The proposed trajectory tracking control scheme can guarantee exponential convergence of the image position and velocity tracking errors. To illustrate the performance of the proposed scheme, experimental results are provided in this article. Note to Practitioners-This article was motivated by the vision-based motion control problem of mobile robots in uncalibrated environments. Existing vision-based motion control approaches for nonholonomic mobile robots generally depend on offline precise/coarse or online numerical/adaptive calibration of the camera intrinsic and extrinsic parameters and require precise or coarse knowledge of the camera in their implementation. This article presents a novel calibration-free image-based trajectory tracking control scheme, which can be implemented easily in real environments without any offline or online calibration of the camera parameters and can be used to efficiently control the motion of nonholonomic mobile robots with an arbitrarily placed and truly unknown overhead camera. Experimental results show that the proposed scheme can achieve satisfactory trajectory tracking control performance despite the lack of any knowledge about the camera intrinsic and extrinsic parameters and the presence of unknown camera lens distortions, and hence, can provide a simple but efficient solution to the vision-based motion control problem of nonholonomic mobile robots.

Although many vision-based control methods have been proposed for nonholonomic mobile robots, in their implementation, it is usually necessary to calibrate the camera intrinsic and/or extrinsic parameters using offline/online parameter estimation algorithms or online adaptation laws. To avoid the tediousness of camera calibration and to make the system performance highly robust to camera parameter uncertainties, in this article, we propose novel image-based pose stabilization control approaches for nonholonomic mobile robots with a truly uncalibrated overhead fixed camera. In the proposed approaches, only image position information of three feature points from an overhead camera is used for controller design, while information from other sensors (such as wheel encoders) is not required. Furthermore, either offline or online camera calibration is not necessary, and no knowledge about the camera intrinsic and extrinsic parameters is needed, which also can greatly simplify the controller implementation. Simulation and experimental results are given to demonstrate the feasibility and effectiveness of the proposed purely image-based pose stabilization approaches.

In this paper, we consider the self-repairing problem from the viewpoint of robotics and our objective is not only to restore the logical network topology but also to maintain the motion synchronization of the physical mobile robot formation. A gradient-based self-repairing algorithm which only relies on the local interactions among coupling robots is presented. More specifically, aiming to optimize the repair path in a distributed manner, a gradient generation and diffusion mechanism is presented first, which can generate a stable gradient distribution in the robot formation. Then, based on the recursive self-repairing technique and the proposed gradient distribution, several self-repairing rules as well as the corresponding individual control method are presented to solve the self-repairing problem. The improvement of the proposed algorithm on the motion synchronism of the robot formation and the optimality of the selected repair path are proved by theoretical analyses. Finally, the effectiveness and the practical applicability of the proposed algorithm are validated by simulations and real experiments.

A new unsupervised learning method of depth and ego-motion using multiple masks from monocular video is proposed in this paper. The depth estimation network and the ego-motion estimation network are trained according to the constraints of depth and ego-motion without truth values. The main contribution of our method is to carefully consider the occlusion of the pixels generated when the adjacent frames are projected to each other, and the blank problem generated in the projection target imaging plane. Two fine masks are designed to solve most of the image pixel mismatch caused by the movement of the camera. In addition, some relatively rare circumstances are considered, and repeated masking is proposed. To some extent, the method is to use a geometric relationship to filter the mismatched pixels for training, making unsupervised learning more efficient and accurate. The experiments on KITTI dataset show our method achieves good performance in terms of depth and ego-motion. The generalization capability of our method is demonstrated by training on the low-quality uncalibrated bike video dataset and evaluating on KITTI dataset, and the results are still good.

Visual localization is a crucial problem in mobile robotics and autonomous driving. One solution is to retrieve images with known pose from a database for the localization of query images. However, in environments with drastically varying conditions (e.g. illumination changes, seasons, occlusion, dynamic objects), retrieval-based localization is severely hampered and becomes a challenging problem. In this paper, a novel domain-invariant feature learning method (DIFL) is proposed based on ComboGAN, a multi-domain image translation network architecture. By introducing a feature consistency loss (FCL) between the encoded features of the original image and translated image in another domain, we are able to train the encoders to generate domain-invariant features in a self-supervised manner. To retrieve a target image from the database, the query image is first encoded using the encoder belonging to the query domain to obtain a domain-invariant feature vector. We then preform retrieval by selecting the database image with the most similar domain-invariant feature vector. We validate the proposed approach on the CMU-Seasons dataset, where we outperform state-of-the-art learning-based descriptors in retrieval-based localization for high and medium precision scenarios.

In this paper, we propose a novel pose optimizer which can be inserted into either supervised or unsupervised end-to-end visual odometry for the purpose of local pose optimization. The pose optimizer is an analogue of the pose graph optimization used in traditional VSLAM algorithms. Local pose optimization is performed by an attention-based neural network which iteratively refines the predicted pose estimates of an image snippet. Instead of complicated graph convolutional network, the attention mechanism based on geometric consistency of trajectory constraint is utilized because pose features whose spatial distribution is not important can be flattened to vectors and then processed. The pose optimizer is aimed at improving pose estimation accuracy by redistributing errors of pose estimates. Quantitative and qualitative evaluation of the proposed approach on the KITTI Odometry dataset [1] is presented to demonstrate its effectiveness in improving pose estimation accuracy and minimizing pose drift.

With the advance in the field of mobile robots, autonomous robots are required for long-term deployment in dynamic and complex environments. However, the performance of Visual Inertial SLAM systems in long-term operation is not satisfactory, and most long-term SLAM systems assumes periodic changes in the environment. This paper presents a novel solution for long-term monocular VI SLAM system in dynamic environment based on autoregression(AR) modeling and map prediction. Map points are first classified into static and semi-static map points according to a memory model. Modeling and prediction of the different states of semi-static map points are performed that are derived from time series models. The predicted map is then fused with the current map to achieve a better forecast for the next frame if the prediction is not satisfactory enough. Experiments are carried out on an embedded system. The results indicate that the map prediction is reliable and the proposed approach improves the performance of long-term localization and mapping in dynamic environments.

During autonomous navigation, environmental information and map noises at different locations may have dissimilar influence on a vehicle's localization process. This implies that the vehicle's localizability, i.e., the ability to localize itself in an environment using laser range finder (LRF) readings, varies over a given map. It is essential to take this factor into consideration when planning a path to avoid large localization errors or placing the vehicle at risk of failure to perform localization. We propose a localizability constraint (LC)-based path planning method for autonomous vehicles which plans the navigation path according to LRF sensor model of the vehicle in an effort to maintain a satisfactory level of localizability throughout the path, as well as to reduce the overall localization error. Our method is not limited to any specific algorithm in the optimization stage. Paths planned with and without LC are compared, and the influence of the LRF sensor model on planning outcomes is discussed through simulations. By conducting comparative experiments on a “JiaoLong” intelligent wheelchair in both indoor and outdoor environments, we show that the proposed method effectively lowers the localization error along the planned paths.

Robot malfunction is inevitable in practical applications of the robot formation control due to uncontrolled crashing, system malfunction or communication loss. In this paper, we study the synchronous formation control problem in the presence of robot malfunctions. Our main idea is to improve the network connectivity and motion synchronism of the robot formation through a series of topology switchings and robot replacements. Firstly, the synchronous formation control method is introduced which enables the robots to tracking their desired trajectories while keeping predefined formation shapes. Secondly, a recursive switched topology control strategy is proposed to restore the formation shape as well as to improve the network connectivity and motion synchronism in the presence of robot malfunctions. Thirdly, the convergence analysis of the proposed control system is presented and a sufficient condition is obtained under an average dwell time scheme. What's more, the proposed approach is fully distributed and the communication delays between neighboring robots also have been taken into consideration. Simulation results demonstrate the effectiveness of the proposed approach.

In recent researches, monocular simultaneous localization and mapping (SLAM) remains a well-known technique for ego-motion tracking however it significantly suffers from scale drift. Depth estimation in a monocular vision system, which is yet a challenging factor, is relevant to this drift issue and hence monocular SLAM remains unsuitable for large scale mapping and localization. This paper presents a novel solution, a wearable and embedded EMoVI-SLAM system, to resolve scale drift through multi-sensor fusion architecture for integrating visual and inertial data, using monocular SLAM as basis of a visual framework. Firstly, the unknown scale parameter in a monocular vision system is addressed based on the IMU measurements, meanwhile gravity direction and gyroscope bias is initialized. Secondly, the estimated pose from monocular visual sensor and the IMU sensor is fused together using Unscented Kalman Filter (UKF). Furthermore, to minimize scale drift, the scale is re-computed after IMU bias errors exceeds the safe threshold limit. Finally, the experiments are carried out by mounting embedded SLAM system on a head-gear in two different test-environments for indoor and outdoor large-scale motion as well as on EuRoC dataset. Experiment results shows that proposed algorithm performs better than the state of the art visual inertial SLAM systems.

Most existing formation control approaches are based on the assumption that the global/relative position and/or velocity measurements of mobile robots are directly available. To extend the application domain and to improve the formation control performance, it is extremely necessary to avoid the use of position and velocity measurements in the design of formation controllers. In this paper, we propose new leader-following formation tracking control schemes for nonholonomic mobile robots with onboard perspective cameras, without using both position and velocity measurements. To address the unavailability issue of position measurements, the leader-follower kinematics model in the image space is developed, which can facilitate the complete elimination of measurement/estimation of the position information. Furthermore, feedback information from the perspective camera of the follower robot is used to design adaptive observers to estimate the leader linear velocity for feedforward compensation, which can handle the absence of velocity measurements such that the proposed schemes can be applied to control formations of mobile robots without mutual communication abilities. By using the Lyapunov stability theory, a rigorous stability analysis based on the nonlinear formation dynamics is provided to show that the global stability of the combined observer-controller closed-loop system can be guaranteed. Both simulation and experimental results are also given to demonstrate the performance of the proposed formation tracking control schemes.

This paper studies the problem of image-based leader-follower formation control for mobile robots, where the controller is designed independently of the leader's motion. An adaptive control scheme, which is suitable for both omnidirectional and perspective cameras, is proposed. The proposed approach avoids the need for accurate calibration of the extrinsic parameters of the omnidirectional camera as well as the intrinsic and extrinsic parameters of perspective camera. Additionally, the coefficients of the plane where the feature point moves relative to the camera frame can be uncertain. These uncertain constant parameters are estimated using an adaptive estimator. Uniform Semi-global Practical Asymptotic Stability (USPAS) of the system is shown using the Lyapunov approach. Experimental results are presented to demonstrate the effectiveness of the proposed control scheme.

In this paper, the task-space cooperative tracking control problem of networked robotic manipulators without task-space velocity measurements is addressed. To overcome the problem without task-space velocity measurements, a novel task-space position observer is designed to update the estimated task-space position and to simultaneously provide the estimated task-space velocity, based on which an adaptive cooperative tracking controller without task-space velocity measurements is presented by introducing new estimated task-space reference velocity and acceleration. Furthermore, adaptive laws are provided to cope with uncertain kinematics and dynamics and rigorous stability analysis is given to show asymptotical convergence of the task-space tracking and synchronization errors in the presence of communication delays under strongly connected directed graphs. Simulation results are given to demonstrate the performance of the proposed approach.

This paper presents a multirobot pairwised transportation (MRPWT) approach for factory automated material and product deliveries. We consider MRPWT from the viewpoint of robotics and incorporate practical factory application constraints in the transportation method design. The proposed MRPWT approach is a two-level hybrid planning method, consisting of an incidental delivery based single robot level planner and a simulated annealing based robot group level planner. Each robot resolves its individual transportation plan incidentally to reduce the transportation cost, whereas the group level planner utilizes predefined random actions to search the task assignment solution space and then incorporates the simulated annealing algorithm to resolve the MRPWT problem as a combinatorial optimization problem. By implementing a distributed auction mechanism, the proposed MRPWT approach can be further extended to resolve the online task allocation or reallocation problem in dynamic environments. Experiments performed on a group of mobile robots successfully demonstrate the effectiveness and the practical applicability of the proposed MRPWT approach for factory automated material and product deliveries.

In this paper, we investigate the self-healing problem of mobile robot formation after some robots have been damaged, and present a gradient-based algorithm which enables mobile robots to restore the topology of the formation through local interactions among neighboring robots. Firstly, in order to optimize the repair path in a distributed manner, a gradient generation and diffusion mechanism is proposed to generate a specific gradient distribution in the formation. Then, utilizing several predefined path selection rules, a path selection algorithm is presented to guarantee the optimality of the selected repair path. Furthermore, several optimization indices are presented to quantitatively characterize the performance of self-healing algorithms. Finally, the effectiveness of the proposed algorithm is validated by numerical simulations and the simulation results show that the proposed algorithm can restore the topology of the formation with the fewer repair robots and lower energy consumptions.

In this paper, we will study the uncalibrated vision-based positioning problem of wheeled mobile robots by using a ceiling-mounted camera. A new image-based visual servoing scheme will be proposed, which can cope with the unknown intrinsic and extrinsic parameters of the camera and the uncertain distance parameter of the feature point from geometric center of the mobile robot. The presented approach is developed via extending the depth-independent interaction matrix framework for robot manipulators to mobile robots such that the nonlinear dependence on unknown parameters can be removed from the image-Jacobian matrix and then, we can linearly parameterize uncertain parameters in the closed-loop system. In this way, an estimation scheme for the online updating of uncertain parameters can be developed very efficiently. To show that image errors can be guaranteed to be asymptotically convergent, stability analysis will be carried out by using Lyapunov theory. To validate the performance of the presented approach, simulation and experimental results will also be provided.

Aim at enhancing dexterous and safe operation in unstructured environment, a cable-driven soft robotic manipulator is designed in this paper. Due to soft material it made of and nearly infinite degree of freedom it owns, the soft robotic manipulator has higher security and dexterity than traditional rigid-link manipulator, which make it suitable to perform tasks in complex environments that is narrow, confined and unstructured. Though the soft robotic manipulator possesses advantages above, it is not an easy thing for it to achieve precise position control. In order to solve this problem, a kinematic model based on piecewise constant curvature hypothesis is proposed. Through building up three spaces and two mappings, the relationship between the length variables of 4 cables and the position and orientation of the soft robotic manipulator end-effector is obtained. Afterwards, a depth-independent image Jacobian matrix is introduced and an image-based visual servo controller is presented. Applied by adaptive algorithm, the controller could estimate unknown position of the feature point online, and then Lyapunov theory is used to prove the stability of the proposed controller. At last, experiments are conducted to demonstrate rationality and validity of the kinematic model and adaptive visual servo controller.

Shared control is a common used method for human and wheelchair cooperation. However, most of the previous shared control methods didn't think much of the effect caused by the difference in the user's control ability. The control weight of a user in these methods is irrelevant to the user's capability or the driving conditions. In this paper, a dynamic shared control method is proposed to adapt wheelchair's assistance to the variations of user performance and the environmental changes. Three evaluation indices including safety, comfort and obedience are designed to evaluate wheelchair performance in real time. A minimax multi-objective optimization algorithm is adopted to calculate the user's control weight. The results of lab experiments and elderly home field tests show that this method can adapt the degree of wheelchair's autonomy to the user's control ability and it makes driving wheelchair much easier for elder people.

This paper presents a new adaptive controller for visual tracking control of a robot manipulator in 3D general motion with a fixed camera whose intrinsic and extrinsic parameters are uncalibrated. In addition to camera parameters, the feature positions in 3D space are also assumed unknown. Based on the fact that the unknown parameters appears linearly in the closed-loop dynamics of the system if the depth-independent interaction matrix is adopted to map the image errors onto the joint space of the manipulator, we developed a new adaptive algorithm to estimated the unknown parameters on-line. With a full consideration of dynamic responses of the robot manipulator, we employ the Lyapunov method to prove asymptotic convergence of the image errors. Experimental results are used to demonstrate the performance of the proposed approach.

To keep up with the wide-ranging, fast-moving robotics field, education must be adaptive and multidisciplinary. In this paper, integration of PSoC technology with educational robotics is presented. Both of the advantage: flexibility in hardware for PSoC and modularized robot components are fused. This integration balances research and implementation fundamentals by reinforcing course work with intensive projects focused on robotics technology. This education fostered students' teamwork skills, while project completion and competition success greatly enhanced the students' self-confidence.

The navigation system of family indoor mobile robot includes localization, path planning, collision avoidance. The hybrid localization method of straight line matching, corner matching and odometry is proposed. The hardware and software configuration is introduced. Robot detects environment using a 2D laser range finder. Line feature extraction process including area divided, iterative end point fit (IEPF) and a least square technique is introduced. Based on line feature, straight lines and corners as geometry features are obtained. The odometry localization algorithm, straight line localization algorithm and corner localization algorithm are discussed. Artificial Potential Field (APF) based path planning algorithms is implemented. As a result stable localization is achieved with position and orientation resolution as 50mm, 5 degree. A good performance for the method is also achieved with cycle time as 120ms. Experiment shows the effectiveness of the hybrid localization method.

This paper presents a new controller for locking a moving object in 3-D space at a particular position (for example the center) on the image plane of a camera mounted on a robot by actively moving the camera. The controller is designed to cope with both unknown robot dynamics parameters and unknown motion of the object. Based on the fact that the unknown position of the moving object appears linearly in the closed-loop dynamics of the system if the depth-independent image Jacobian is used, we developed a nonlinear observer to estimate the 3-D motion of the object on-line and an adaptive algorithm to estimate the robot dynamic parameters. With a full consideration of dynamic responses of the robot, we employed the Lyapunov method to prove asymptotic convergence of the image errors. Experimental results are presented to support the approach in this paper.

Target detecting and dynamic coverage are fundamental tasks in mobile robotics and represent two important features of mobile robots: mobility and perceptivity. This paper establishes the constrained motion model and sensor model of a mobile robot to represent these two features and defines the k -step reachable region to describe the states that the robot may reach. We show that the calculation of the k-step reachable region can be reduced from that of 2(k) reachable regions with the fixed motion styles to k + 1 such regions and provide an algorithm for its calculation. Based on the constrained motion model and the k -step reachable region, the problems associated with target detecting and dynamic coverage are formulated and solved. For target detecting, the k-step detectable region is used to describe the area that the robot may detect, and an algorithm for detecting a target and planning the optimal path is proposed. For dynamic coverage, the k-step detected region is used to represent the area that the robot has detected during its motion, and the dynamic-coverage strategy and algorithm are proposed. Simulation results demonstrate the efficiency of the coverage algorithm in both convex and concave environments.

The objective of self-healing in mobile robot networks is to maintain not only logical topology but also physical one of a network when robots fail. An interaction dynamics model is first established to describe both logical and physical topologies of the network. Considering the mobility of mobile robot networks, we propose a recursive, distributed topology control for self-healing when mobile robots fail, and give a metric of the topology structure for evaluating the performance of recovered network topologies. Then, we prove the stability of motion synchronization with the topology control based on Lyapunov exponent. Finally, the results of simulations have demonstrated the validity of the proposed modeling and control methods.

Motion synchronization in mobile robot networks is a fundamental task in distributed multi-robot collaboration. In this paper, we investigate the robustness of synchronous speeds against robot and communication failures, an important feature of distributed systems. A performance metric is proposed to represent the robustness of the robot teams based on complex networks theories. Moreover, a distributed topology control algorithm to improve the robustness metric of groups of robots by increasing the communication range temporarily is presented. When the new topology is constructed and is stable, the range for communicating is reduced to the original value. Finally, the results of simulation experiments have demonstrated the efficiency of the proposed control algorithm while robots and connections fail in a robot group

Using multi-robot has more advantages than using single robot for unknown environment exploration. But it brings a new problem of task allocation. Market-based method is an economic approach to allocating targets for robots through auction. However, it only considers costs in the local map of each robot. We update the local maps through fusion of both the local sensor data and the bid information, and thus the extended parts in maps enable robots to calculate costs of other robots’ targets. No extra communication is needed. The results of real robot experiments and simulations demonstrate that the improved method is more efficient than the original market-based approach and provide a proper improved method for environment exploration.

Developing an open mobile robot has been a hot topic in the AI area. In an open system, a particular component can be easily added and/or replaced. In this paper, a modular and object oriented approach is used to construct an autonomous mobile robot system. IPC (interprocess communication), which is the key mechanism in the Linux/Unix operating system, is applied to the robot design. Based on the Windows operating system, the robot owns an omnivision, and can finish tasks such as navigation and robot soccer. With a modular design, new components can be easily added to the system. Finally, the robot is applied to several tasks. The experimental results show the good performance of the robot.

A robotic theory is presented to explain the process of protein crystallization. The theory employs the approach of path planning for robot team formation, but imposes two constraints-simplicity and locality because proteins do not have the intelligence to plan complex paths in a global manner. Based on the two constraints, we develop three rules which govern the development of a successful path. The path defines the direction and speed of the motion for all the proteins in forming the crystal. Physical analysis at the molecular level is conducted to verify that the hypothesized motion for crystallization is achievable. Computer simulation and experimental results are presented to show the promise of the new approach in the study of protein crystallization.

For the on-line safe path planning of a mobile robot in unknown environments, the paper proposes a simple Hopfield Neural Network (HNN) planner. Without learning process, the HNN plans a safe path with consideration of "too close" or "too far". For obstacles of arbitrary shape, we prove that the HNN has no unexpected local attractive point and can find a steepest climbing path, if a feasible path(s) exists. To effectively simulate the HNN on sequential processor, we discuss algorithms with O(N) time complexity, and propose the constrained distance transformation-based Gauss-Seidel iteration method to solve the HNN. Simulations and experiments demonstrate the method has high real-time ability and adaptability to complex environments.

The environment of a middle-size autonomous robot soccer system (MARSS) is highly dynamic, competitive and partially observed. To decide quickly, behave smoothly and speedily, the proposed MARSS used a behavior-based implicit planning, and select suitable task according to the hidden state. To overcome imprecise perceptions and actions, several subtle but simple goal-driven behaviors are designed. Tightly integrated with the simple perceptions, these behaviors switch flexibly and robustly by continuous feedback. In the competitive environments, spontaneous interleaving of these behaviors implicitly plans effective behavior sequences to fulfill the game's tasks, and exhibits some important emergent behaviors making the system design more simple, robust and competitive. The method's effectiveness is verified by experiments and games.