LiDARs are widely used in autonomous robots due to their ability to provide accurate environment structural information. However, the large size of point clouds poses challenges in terms of data storage and transmission. In this paper, we propose a novel point cloud compression and transmission framework for resource-constrained robotic applications, called RCPCC. We iteratively fit the surface of point clouds with a similar range value and eliminate redundancy through their spatial relationships. Then, we use Shape-adaptive DCT (SA-DCT) to transform the unfit points and reduce the data volume by quantizing the transformed coefficients. We design an adaptive bitrate control strategy based on QoE as the optimization goal to control the quality of the transmitted point cloud. Experiments show that our framework achieves compression rates of 40×to 80× while maintaining high accuracy for downstream applications. our method significantly outperforms other baselines in terms of accuracy when the compression rate exceeds 70×. Furthermore, in situations of reduced communication bandwidth, our adaptive bitrate control strategy demonstrates significant QoE improvements. The code will be available at https://github.com/HITSZ-NRSL/RCPCC.git.

Visual place recognition is a critical component of robust simultaneous localization and mapping systems. Conventional approaches primarily rely on RGB imagery, but their performance degrades significantly in extreme environments, such as those with poor illumination and airborne particulate interference (e.g., smoke or fog), which significantly degrade the performance of RGB-based methods. Furthermore, existing techniques often struggle with cross-scenario generalization. To overcome these limitations, we propose an RGB-thermal multimodal fusion framework for place recognition, specifically designed to enhance robustness in extreme environmental conditions. Our framework incorporates a dynamic RGB-thermal fusion module, coupled with dual fine-tuned vision foundation models as the feature extraction backbone. Experimental results on public datasets and our self-collected dataset demonstrate that our method significantly outperforms state-of-the-art RGB-based approaches, achieving generalizable and robust retrieval capabilities across day and night scenarios. The code is available at https://github.com/HITSZ-NRSL/RGB-Thermal-VPR.

Utilizing robots for autonomous target search in complex and unknown environments can greatly improve the efficiency of search and rescue missions. However, existing methods have shown inadequate performance due to hardware platform limitations, inefficient viewpoint selection strategies, and conservative motion planning. In this work, we propose HEATS, which enhances the search capability of mobile manipulators in complex and unknown environments. We design a target viewpoint planner tailored to the strengths of mobile manipulators, ensuring efficient and comprehensive viewpoint planning. Supported by this, a whole-body motion planner integrates global path search with local IPC optimization, enabling the mobile manipulator to safely and agilely visit target viewpoints, significantly improving search performance. We present extensive simulated and real-world tests, in which our method demonstrates reduced search time, higher target search completeness, and lower movement cost compared to classic and state-of-the-art approaches. Our method will be open-sourced for community benefit3.

Panoramic observation using fisheye cameras is significant in virtual reality (VR) and robot perception. How-ever, panoramic images synthesized by traditional methods lack depth information and can only provide three degrees-of-freedom (3DoF) rotation rendering in VR applications. To fully preserve and exploit the parallax information within the original fisheye cameras, we introduce MSI-NeRF, which combines deep learning omnidirectional depth estimation and novel view synthesis. We construct a multi-sphere image as a cost volume through feature extraction and warping of the input images. We further build an implicit radiance field using spatial points and interpolated 3D feature vectors as input, which can simultaneously realize omnidirectional depth estimation and 6DoF view synthesis. Leveraging the knowledge from depth estimation task, our method can learn scene appearance by source view supervision only. It does not require novel target views and can be trained conveniently on existing panorama depth estimation datasets. Our network has the generalization ability to reconstruct unknown scenes efficiently using only four images. Experimental results show that our method outperforms existing methods in both depth estimation and novel view synthesis tasks.

Autonomous ground mobile robots rely on their configuration characteristics to prevent tip-overs and collisions, ensuring safe navigation in complex environments. However, complex configurations with specially designed links and joints produce a higher-dimensional workspace and bring significant challenges for path planning, especially in large-scale rough terrains. To address this, we propose a real-time multi-level terrain-aware path planning framework that integrates different levels of terrain awareness into the global and local layers. An implicit map representation is introduced at the global layer to enable efficient terrain analysis and path planning, while an iterative geometric evaluation is designed at the local layer to estimate configuration stability and improve path smoothness. By sharing the global layer information with the local layer, the framework enhances path planning efficiency and adaptability in complex environments. Its modular design supports diverse robot configurations and pathfinding algorithms, enabling effective autonomous navigation in large-scale 3D terrains with online or offline maps. Simulations and real-world experiments demonstrated that our approach outperforms state-of-the-arts across diverse environments, including uneven terrains, multi-layered structures, and complex debris fields. The results highlighted that our approach provides faster and safer path planning, more accurate and robust configuration-stability estimation, and higher success rates in traversing complex 3D environments.

Large-scale dense mapping is vital in robotics, digital twins, and virtual reality. Recently, implicit neural mapping has shown remarkable reconstruction quality. However, incremental large-scale mapping with implicit neural representations remains problematic due to low efficiency, limited video memory, and the catastrophic forgetting phenomenon. To counter these challenges, we introduce the Robot-centric Implicit Mapping (RIM) technique for large-scale incremental dense mapping. This method employs a hybrid representation, encoding shapes with implicit features via a multi-resolution voxel map and decoding signed distance fields through a shallow MLP. We advocate for a robot-centric local map to boost model training efficiency and curb the catastrophic forgetting issue. A decoupled scalable global map is further developed to archive learned features for reuse and maintain constant video memory consumption. Validation experiments demonstrate our method’s exceptional quality, efficiency, and adaptability across diverse scales and scenes over advanced dense mapping methods using range sensors. Our system’s code will be accessible at https://github.com/HITSZ-NRSL/RIM.git.

The outdoor environment has many uneven surfaces that put the robot at risk of sinking or tipping over. Recognizing the type of terrain can help robot avoid risks and choose an appropriate gait. One of the critical problems is how to extract the terrain-related knowledge from sensor data collected as the robot traversed the ground. Many existing vision-based approaches are limited in directly perceiving the intrinsic properties of various terrains. The intuitive approach entails directly analyzing data recorded by the robot’s proprioceptive sensors. However, it faces challenges in being specific to certain robot leg configurations or in the lack of interpretability of the extracted features. In this paper, a terrain attribute extraction algorithm is proposed based on contrastive learning. It leverages the haptic data generated from the interaction between the robot’s legs and terrain to automatically extract terrain attributes. The results demonstrate that the attributes extracted using this method strongly correlate with the actual softness of the terrain. Furthermore, these attributes played an important role in achieving high accuracy in terrain classification tasks.

Target tracking is a fundamental task in the domain of robotics. The effectiveness of target tracking hinges upon various factors, such as tracking distance, occlusions, collision avoidance, etc. However, few existing works can simultaneously tackle these considerations of tracking single and multiple targets in complex environments. In this study, the interaction mechanism of target tracking between the robot, the environment and the targets is analyzed, and a general measure named detectability is introduced to correlate the tracking performance for guiding robotic motion planning. Based on the detectability measure, the robotic motion planning framework based on Model Predictive Control (MPC) is proposed to achieve continuous and robust tracking of single, two and three targets in complex environments. Simulations and experiments are performed and verify the performances of our method better than the state-of-the-art methods.

It is challenging to realize the safe navigation of mobile robots in crowds. Most of the previous studies may lead to unsafe robot navigation in crowds, as safety guarantee is lacked. To solve this problem, we devise a safety filter (SF) that enables realization of safe robot navigation in crowds, and provides safety guarantees by verifying whether the optimal action recommended by an unsafe method is safe and, if not, corrects the action. The three main processes performed by the SF applied to given robot are (1) construction of the safe state constraints of the robot using a safe set; (2) construction of the safe action constraints of the robot based on discrete-time generalized velocity obstacles (DGVOs); and (3) determination of a feasible solution of the SF design problem, or, if none can be found, replacement of the above hard constraints with heuristic soft constraints. We used the SF with a reaction-based method and three learning-based methods in simulation experiments of random and non-random crowds, and the results showed that the SF decreases the collision rates and danger rates and thereby increases the success rates of these methods. We also deployed the SF with three learning-based methods on an mr1000 robot in real-world experiments, and the results showed that the SF enabled the robot using learning-based methods to navigate to its goal without colliding with humans.

Micro aerial vehicles usually have strap-down sensors on the vehicle body, leading to the severe coupling effect between perception and trajectory planning. As a result, visual-inertial simultaneous localization and mapping (VI-SLAM) technologies implemented on MAVs suffer from tracking failure problems, especially in featureless environments. To overcome these challenges, based on MAVs with movable camera mechanisms (e.g., gimbal stabilizer, pan-tilt, or bionic neck-eye system), we proposed two sampling-based algorithms for known and unknown environments respectively. The first active perception planning algorithm based on a scene richness model is developed with a built feature map for the environment. Differ from the first algorithm, the second one is modified for active localization in unknown 3D space. It is basically a time-based sampling-based approach that uses the same scene richness model. In addition, it also achieved a balance between exploitation and exploration. With the above solutions, the robustness of visual perception is improved while avoiding over-exploitation of known information. Simulation and real-world experiments are performed to verify the feasibility of our algorithms.

Autonomous exploration in unknown environments is a fundamental task for robots. Existing approaches mostly were concentrated on the efficiency of the exploration with the assumption of perfect state estimation, but the drift of pose estimation in visual SLAM occurs frequently and is detrimental to robot's localization and exploration performance. In this paper, a perception-aware exploration(PAE) method is proposed for rapidly and safely autonomous exploration in outdoor environments. The adaptive semantic information is proposed to improve the robustness of perception. Based on the perception module, both the selection of exploration goal on a novel weighted information gain and path planning can avoid the areas with high localization uncertainty. In addition, thanks to the proposed pipeline, including scan-based frontier detection, kd-tree based map prediction and suboptimal frontier buffer strategy, the PAE planner can explore the environment with high success rate and high efficiency. Several simulations are performed to verify the effectiveness of our methods.

Autonomous delivery of suspended payloads with MAVs has many applications in rescue and logistics transportation. Robust and online estimation of the payload status is important but challenging especially in outdoor environments. The paper develops a novel real-time system for estimating the payload position; the system consists of a monocular fisheye camera and a novel encoder-based device. A Gaussian fusion-based estimation algorithm is developed to obtain the payload state estimation. Based on the robust payload position estimation, a payload controller is presented to ensure the re-liable tracking performance on aggressive trajectories. Several experiments are performed to validate the high performance of the proposed method.

The problem of consensus in networked agent systems is revisited and applied to vision-based localization. A class of new consensus dynamics is introduced first, and sufficient conditions including the persistence of excitation on the coupling matrix for reaching consensus are derived. As an application of the proposed consensus dynamics, an adaptive localization algorithm then is proposed for autonomous robots equipped with primarily visual sensors in GPS-denied environments. In the context of consensus over an undirected tree topology, the convergence of the proposed localization algorithm is proved. Finally, both numerical simulations and physical experiments are presented to show the effectiveness of the proposed localization algorithm. Our algorithm is simpler to implement and computationally cheaper compared to other localization methods. Moreover, it is immune to error accumulation and long-term stable, and the asymptotical convergence of the estimation errors can be theoretically guaranteed.

This paper focuses on designing, modeling and validating a novel scalable multicopter whose deformation mechanism, called SNIAE-SSE, relies on a combination of simple non-intersecting angulated elements (SNIAEs) and straight scissor-like elements (SSEs). The proposed SNIAE-SSE mechanism has the advantages of single degree-of-freedom, fast actuation capability and large deformation ratio. In this work, enabled by the SNIAE-SSE mechanism, a quadcopter prototype with symmetrical and synchronous deformation is firstly developed, which facilitates a novel and controllably scalable multicopter system for us to analyze its modeling, as well as to validate its flight performance and dynamics during the deformation in several flight missions including hover, throwing, and morphing flying through a narrow window. Experimental results demonstrate that the developed scalable multicopter can maintain its stable flight behavior even both the folding and unfolding body deformations are fast performed, which indicates an excellent capability of the scalable multicopter to rapidly adapt to complex and dynamically changed environments.

Industrial micro aerial vehicles (MAVs) with robotic manipulators have numerous applications in search and rescue tasks that reduce risks to human beings. However, such tasks distinctly require MAVs to have the capability of real-time autonomous navigation only with onboard sensors, especially in GPS-denied applications. This article introduces a new approach to onboard vision-based autonomous state estimation and mapping for MAVs' navigation in unknown environments. The algorithms run on board and do not need an external positioning system to assist autonomous navigation. The state estimator is developed to provide MAV's current pose on the basis of the extended Kalman filter by using image patch features. Inverse depth convergence monitoring and local bundle adjustment are utilized to improve the accuracy. The mapping algorithm for navigation is developed according to a real-time stereo matching method for three-dimensional perception. Finally, we have performed several experiments to demonstrate the effectiveness of the proposed approach.

Cameras and encoders are widely used in mobile robots, and extrinsic parameter calibration of these sensors is crucial in practical performance. The existing approaches mainly rely on manual measurements, accurate computer-aided design (CAD) models, or carefully designed artificial landmarks. This paper presents a novel approach for automatically calibrating the extrinsic parameters of the robotic camera-encoder system. The approach first calculates a coarse estimation of the external parameters as well as the scale of the visual system, via free-scale hand-eye calibration of the camera and odometer. However, the coarse calibration result and scale of the visual system do not satisfy the accuracy requirement for further mobile robots' applications such as localization and navigation. A nonlinear optimization algorithm that considers both bundle adjustment and odometer measurement error functions is developed to refine the extrinsic parameter calibration result. This coarse-fine approach is computationally efficient and can achieve online calibration during the vehicle motion automatically. Furthermore, it can realize the calibration without using any artificial landmark or prior knowledge about CAD models. Finally, comparisons to other classic calibration approaches are performed with a series of simulations and experiments to illustrate the effectiveness of the approach.

The grasping control of an aerial manipulator in practical environments is challenging due to its complex kinematics/dynamics and motion constraints. This paper introduces a lightweight aerial manipulator, which is combined with an X8 coaxial octocopter and a 4-DoF manipulator. To address the grasping control problem, we develop an efficient scheme containing trajectory generation, visual trajectory tracking, and kinematic compensation. The NSGA-II method is utilized to implement the multiobjective optimization for trajectory planning. Motion constraints and collision avoidance are also considered in the optimization. A kinematic compensation-based visual trajectory tracking is introduced to address the coupled nature between manipulator and VAV body. No dynamic parameter calibration is needed. Finally, several experiments are performed to verify the stability and feasibility of the proposed approach.

Flexible continuum robots have exhibited unique advantages in working in an unstructured environment. Many applications require robots to actively control the deformation of soft objects, such as soft tissues in surgery. Thus, this study presents a robust model-predictive deformation control of a soft object using a flexible continuum robot. A linear approximation model for mapping from actuation space of a continuum robot to deformation space of a soft object is established. Jacobian matrix is estimated online by using a robust Geman-McClure estimator. Then, the deformation of the soft object is regulated by using a prediction horizon-based controller with exponential weighting for model uncertainty. The proposed control approach is effective in manipulating a soft object with a flexible continuum robot that is in contact with obstacles.

Most existing trajectory tracking controllers are based on non-skidding and non-slipping assumptions, also assume that full states are accessible, which is unrealistic for real-world applications due to tire-road interaction. This paper presents a novel vision-based approach to achieve high performance tracking control of a Car-Like Mobile Robot (CLMR) with wheel skidding and slippage. A visual estimation algorithm is proposed to provide reliable position, velocity, skidding and slipping information to close the control loop. The stability of the proposed system can be guaranteed by Lyapunov method since the position tracking error and the estimation error converge to zero simultaneously. Simulation is made to validate the effectiveness of the developed controller in the presence of skidding and slipping with online visual estimator.

How to achieve the required contouring tracking accuracy especially during high-speed and large-curvature contouring tasks, has always been an important problem in manufacturing applications. In this paper, a contouring error estimation method based on natural local approximation is used, and then the position-loop cross-coupled controller is proposed to reduce the estimated contouring error. The effectiveness and superiority of the natural local approximation method using on the position-loop cross-coupled control scheme are demonstrated through experiments on a biaxial linear motor drive servo system.

Transportation manipulation of biological cells, where cells are required to move into a fixed or moving region, has recently attracted increasing attention in bioscience and nanomedicine. Currently, the multicell transportation in practical applications is implemented manually, with low precision and efficiency. This paper presents a swarm-inspired approach to automated transportation of multiple cells using robotically controlled optical tweezers. A swarming controller, where holographic optical tweezers function as end-effectors to manipulate the cells, was developed. To ensure that the cells do not escape from the optical traps, the controller was designed by incorporating a saturation control of the cell offset to the laser center. Because the optical tweezers can only be position-controlled, oscillation may easily occur. This problem was solved by integrating artificial first-order kinematics of the optical tweezers into the controller design. Experiments of transporting multiple yeast cells were performed to verify the effectiveness of the proposed approach.

Optical manipulation of biological cells has recently attracted increasing attention in bioscience and nanotechnology, where optical tweezers are used as end-effectors to manipulate the cells with high precision and flexibility. Analysis of the dynamics of the optically trapped cells plays a critical role in many cell manipulation tasks such as the automatic cell transportation and force transducer. This paper presents a novel approach to calibrating the cell dynamics with the adaptive control technology. According to different measurements, two adaptive tracking controllers are designed, based on which the estimated parameters of the cell trapping dynamics (i.e., the rate of viscous coefficient and trapping stiffness) can automatically converge to the true values. Stability of the adaptive controllers and convergence of the estimated parameters are analyzed by using Lyapunov approach. Simulations and experiments of manipulating yeast cells are performed to verify the effectiveness of the proposed approach.

Flocking of micro-scaled particles, attracts increasing attention especially in cell engineering and drug industry, due to its potential application for particle manipulation with high throughput and productivity. This paper presents an efficient approach to flocking micro particles with robotics and optical tweezers technologies. All particles trapped by optical tweezers can be gradually moved towards a pre-defined region. The main contribution of this paper lies in a solution to achieve the flocking manipulation of particles in micro environments. A local potential function is proposed to avoid collision amongst particles and obstacles. Based on the relationship amongst laser power, particle movement velocity, and trapping force, saturation of velocities is employed to bound particle velocities. In this way, the flocking manipulation can be operated with efficiency and safety. Experiments on yeast cells with a robot-tweezers system are finally performed to verify the effectiveness of the proposed approach.

Significant demand for both accuracy and productivity in batch manipulation of microparticles highlights the need to develop an automatic arraying approach to placing groups of particles into a predefined array with right pairs. This paper presents our latest effort to achieve this objective using integrated robotics and holographic optical tweezers technologies, where holographic optical tweezers function as special robot end-effectors to manipulate the microparticles. Based on the physical dynamics of trapping, a potential-field-based controller is developed to drive every pair of particles to the assigned array, while preventing collisions between particles. The significance of the proposed controller lies in the capability of driving two groups of particles into a common array in right pair and controlling the interdistances between the particles in pairs. Experiments are performed to demonstrate the effectiveness of the proposed approach.

This paper presents a novel formation framework for the manipulation of micro-scaled particles with robotics and optical tweezers technologies. An allocation-based formation algorithm is used to calculate particles' trajectories. Along the trajectories, particles are trapped and moved by optical tweezers. Particles can be gradually moved into a pre-defined formation array. The main contribution of this paper lies in the proposal of using multi-agent solution to address the formation problem of particles in micro environment. The proposed framework can be applied to many bio-applications, such as cell sorting, cell transportation, cell-to-cell interaction study, etc., with high throughput and precision. Experiments on micro-scaled particles, with a robot-tweezer manipulation system, are performed to demonstrate the effectiveness of the proposed approach.

Batch manipulation of micro particles attracts increasing attention among researchers in bio-medical fields such as cellular engineering and drug discovery. Significant demand for both accuracy and productivity highlights the need of developing an automatic arraying approach to moving and pairing a swarm of particles to a pre-defined array. This paper presents our latest effort to achieve this objective by using integrated robotics and holographic optical tweezers technologies, where holographic optical tweezers function as special robot end-effectors. A controller is proposed to drive pairs of particles to the assigned regions which are centered at array points. The potential field method is utilized to avoid collisions between particles. Experiments on colloidal particles are performed to demonstrate the effectiveness of the proposed approach.

This paper presents a novel flocking framework for the manipulation of micro-scale particles with robotics and optical tweezers technologies. A region-based flocking algorithm is used to calculate the particles' trajectories. The optical tweezers are used to trap and move the particles along generated trajectories. All particles can be gradually moved into a pre-defined region. The main contribution of this paper lies in the proposal of using multi-agent solution to address the flocking problem of particles in micro environment. The proposed framework can be applied to many bio-applications such as cell sorting, cell property characterization, and so on, with high throughput and precision. Experiments on micro-scale particles with a robot-tweezers system are finally performed to verify the effectiveness of the proposed approach.

This paper investigates the multirobot task allocation (MRTA) problem for a group of heterogeneous mobile robots. The robots and tasks are characterized by resources as required by task execution. The robots are required to generate optimal solutions for the MRTA problem while forming coalitions to meet the resource constraints imposed by tasks. A leader-follower based coalition methodology is developed, with detailed discussions on leader selection, coalition forming and refinement algorithms. It is shown that the resource constrained task allocation problem can be well resolved by the proposed leader-follower coalition algorithms. Simulations performed on a mobile robot group demonstrate the effectiveness of the proposed approach.