Planning paths that maximize information gain for robotic platforms has wide-ranging applications and significant potential impact. To effectively adapt to real-time data collection, informative path planning must be computed online and be responsive to new observations. In this work, we present IA-TIGRIS (Incremental and Adaptive Tree-based Information Gathering Using Informed Sampling), which is an incremental and adaptive sampling-based informative path planner designed for real-time onboard execution. Our approach leverages past planning efforts through incremental refinement while continuously adapting to updated belief maps. We additionally present detailed implementation and optimization insights to facilitate real-world deployment, along with an array of reward functions tailored to specific missions and behaviors. Extensive simulation results demonstrate IA-TIGRIS generates higher-quality paths compared to baseline methods. We validate our planner on two distinct hardware platforms: a hexarotor unmanned aerial vehicle (UAV) and a fixed-wing UAV, each having different motion models and configuration spaces. Our results show up to a 38% improvement in information gain compared to baseline methods, highlighting the planner's potential for deployment in real-world applications. Project website: ia-tigris.github.io.

Robot navigation in dense human crowds poses a significant challenge due to the complexity of human behavior in dynamic and obstacle-rich environments. In this work, we propose a dynamic weight adjustment scheme using a neural network to predict the optimal weights of objectives in an optimization-based motion planner. We adopt a spatial-temporal trajectory planner and incorporate diverse objectives to achieve a balance among safety, efficiency, and goal achievement in complex and dynamic environments. We design the network structure, observation encoding, and reward function to effectively train the policy network using reinforcement learning, allowing the robot to adapt its behavior in real time based on environmental and pedestrian information. Simulation results show improved safety compared to the fixed-weight planner and the state-of-the-art learning-based methods, and verify the ability of the learned policy to adaptively adjust the weights based on the observed situations. The feasibility of the approach is demonstrated in a navigation task using an autonomous delivery robot across a crowded corridor over a 300 m distance. Video: https://youtu.be/nSCbNaaF_VM

Helmet-mounted wearable positioning systems are crucial for enhancing safety and facilitating coordination in industrial, construction, and emergency rescue environments. These systems, including LiDAR-Inertial Odometry (LIO) and Visual-Inertial Odometry (VIO), often face challenges in localization due to adverse environmental conditions such as dust, smoke, and limited visual features. To address these limitations, we propose a novel head-mounted Inertial Measurement Unit (IMU) dataset with ground truth, aimed at advancing data-driven IMU pose estimation. Our dataset captures human head motion patterns using a helmet-mounted system, with data from ten participants performing various activities. We explore the application of neural networks, specifically Long Short-Term Memory (LSTM) and Transformer networks, to correct IMU biases and improve localization accuracy. Additionally, we evaluate the performance of these methods across different IMU data window dimensions, motion patterns, and sensor types. We release a publicly available dataset, demonstrate the feasibility of advanced neural network approaches for helmet-based localization, and provide evaluation metrics to establish a baseline for future studies in this field. Data and code can be found at https://lqiutong.github.io/HelmetPoser.github.io/.

Humans learn from observations and experiences to adjust their behaviours towards better performance. Interacting with such dynamic humans is challenging, as the robot needs to predict the humans accurately for safe and efficient operations. Long-term interactions with dynamic humans have not been extensively studied by prior works. We propose an adaptive human prediction model based on the Theory-of-Mind (ToM), a fundamental social-cognitive ability that enables humans to infer others' behaviours and intentions. We formulate the human internal belief about others using a game-theoretic model, which predicts the future motions of all agents in a navigation scenario. To estimate an evolving belief, we use an Unscented Kalman Filter to update the behavioural parameters in the human internal model. Our formulation provides unique interpretability to dynamic human behaviours by inferring how the human predicts the robot. We demonstrate through longterm experiments in both simulations and real-world settings that our prediction effectively promotes safety and efficiency in downstream robot planning. Code will be available at https://github.com/centiLinda/AToM-human-prediction.git.

Ultra-wideband (UWB) is gaining popularity with devices like AirTags for precise home item localization but faces significant challenges when scaled to large environments like seaports. The main challenges are calibration and localization under obstructed conditions, which are common in logistics environments. Traditional calibration methods, dependent on line-of-sight (LoS), are slow, costly, and unreliable in seaports and warehouses, making large-scale localization a significant pain point in the industry. To overcome these challenges, we propose a one-shot calibration and localization framework based on UWB-LiDAR fusion. Our method uses Gaussian processes to estimate the anchor position from continuous-time LiDAR Inertial Odometry with sampled UWB ranges. This approach ensures accurate and reliable calibration with only one round of sampling in large-scale areas, i.e.,$600 \times 450 ~\mathrm{m}^{2}$. With LoS issues, UWB-only localization can be problematic, even when anchor positions are known. We demonstrate that by applying a UWB-range filter, the search range for LiDAR loop closure descriptors is significantly reduced, improving both accuracy and speed. This concept can be applied to other loop closure detection methods, enabling cost-effective localization in large-scale warehouses and seaports. It significantly improves precision in challenging environments where the UWB-only and LiDAR-Inertial methods fail, as shown in the video https://https://youtu.be/oY8jQKdM7lU. We will open-source our datasets and calibration codes for community use.

Autonomous exploration in unknown environments requires estimating the information gain of an action to guide planning decisions. While prior approaches often compute information gain at discrete waypoints, pathwise integration offers a more comprehensive estimation but is often computationally challenging or infeasible and prone to overestimation. In this work, we propose the Pathwise Information Gain with Map Prediction for Exploration (PIPE) planner, which integrates cumulative sensor coverage along planned trajectories while leveraging map prediction to mitigate overestimation. To enable efficient pathwise coverage computation, we introduce a method to efficiently calculate the expected observation mask along the planned path, significantly reducing computational overhead. We validate PIPE on real-world floorplan datasets, demonstrating its superior performance over state-of-the-art baselines. Our results highlight the benefits of integrating predictive mapping with pathwise information gain for efficient and informed exploration. Website: pipe-planner.github.io

Large-scale LiDAR Bundle Adjustment (LBA) to refine sensor orientation and point cloud accuracy simultaneously for building navigation maps is a fundamental task in logistics, intelligent transportation, and robotics. In the context of autonomous delivery and smart mobility, the 3D map obtained by accurate and robust LBA plays a pivotal role in enabling reliable localization and navigation across complex, large-scale urban environments. Unlike pose-graph-based methods that rely solely on pairwise relationships between LiDAR frames, LBA leverages raw LiDAR correspondences to achieve more precise results, especially when initial pose estimates are unreliable for low-cost sensors. However, existing LBA methods face challenges such as simplistic planar correspondences, extensive observations, and dense normal matrices in the least-squares problem, which limit robustness, efficiency, and scalability. To address these issues, we propose a Graph Optimality-aware Stochastic Optimization scheme with Progressive Spatial Smoothing, namely PSS-GOSO, to achieverobust,efficient, andscalableLBA. The Progressive Spatial Smoothing (PSS) module extractsrobustLiDAR feature association exploiting the prior structure information obtained by the polynomial smooth kernel. The Graph Optimality-aware Stochastic Optimization (GOSO) module first sparsifies the graph according to optimality for anefficientoptimization. GOSO then utilizes stochastic clustering and graph marginalization to solve the large-scale state estimation problem for ascalableLBA. We validate PSS-GOSO across diverse scenes captured by various platforms, demonstrating its superior performance compared to existing methods. Moreover, the resulting point cloud maps are used for automatic last-mile delivery in large-scale complex scenes, showcasing the practical benefits of our method in modern intelligent transportation systems. The project page can be found at:https://kafeiyin00.github.io/PSS-GOSO/

Inter-robot relative positions are crucial for executing various multirobot missions, such as formation maneuvering and collaborative inspection. However, the current sensing technology usually provides part of relative position information, such as inter-robot distances, bearings and angles. This prompts the study of determining inter-robot relative positions, i.e., relative localization, from these partial measurements. Based on the existing results of static networks' localizability and mobile robots' relative localization, we propose a novel concept,relative localizabilityto describe whether a multirobot system isrelatively localizable. Given each robot's self-displacement measurements and inter-robot partial measurements in$d$($d\leq 4$) sampling instants, we show that a multirobot system's relative localization can be achieved in a purelyalgebraicanddistributedmanner, in which the multirobot system is said to be$d$-step relatively localizable. To make the results more general, we consider that the multirobot system consists of landmarks, leaders, and followers, and that the inter-robot measurements can be distances, bearings or angles. When robots' coordinate frames have different orientations, we show that the given local measurements can be used to determine robots' relative positions and their coordinate frames' relative orientations simultaneously. Simulations and experiments of relative localization for ground robots are conducted to validate the obtained results.

Inspired by the behavior of birds, we present AirCrab, a hybrid aerial ground manipulator (HAGM) with a single active wheel and a 3-degree of freedom (3-DoF) manipulator. AirCrab leverages a single point of contact with the ground to reduce position drift and improve manipulation accuracy. The single active wheel enables locomotion on narrow surfaces without adding significant weight to the robot. To realize accurate attitude maintenance using propellers on the ground, we design a control allocation method for AirCrab that prioritizes attitude control and dynamically adjusts the thrust input to reduce energy consumption. Experiments verify the effectiveness of the proposed control method and the gain in manipulation accuracy with ground contact. A series of operations to complete the letters ‘NTU’ demonstrates the capability of the robot to perform challenging hybrid aerial-ground manipulation missions.

This article investigates the similar formation control problem for multirobot systems. Specifically, we propose an integrated relative localization and similar formation control scheme to navigate multirobot systems to a desired configuration, which is a similar transformation of a given template, based on interrobot and robot-landmark range measurements and odometry measurements of robots themselves. To achieve the exact relative localization, a persistent excitation (P.E.) signal is introduced in the controller which, however, perturbs the motion of each robot and affects the formation accuracy. To resolve the conflict, an autonomous system with its output regulated by a carefully designed function of range measurements is introduced to generate the persistent excitation. It is proved that the similar formation control problem can be solved by our proposed scheme with global asymptotic convergence for directed acyclic graphs (DAGs). Both numerical simulation and physical experiment are presented to verify and validate the effectiveness of our theoretical findings.

In this paper, we present the dynamic model and control of DoubleBee, a novel hybrid aerial-ground vehicle consisting of two propellers mounted on tilting servo motors and two motor-driven wheels. DoubleBee exploits the high energy efficiency of a bicopter configuration in aerial mode, and enjoys the low power consumption of a two-wheel self-balancing robot on the ground. Furthermore, the propeller thrusts act as additional control inputs on the ground, enabling a novel decoupled control scheme where the attitude of the robot is controlled using thrusts and the translational motion is realized using wheels. A prototype of DoubleBee is constructed using commercially available components. The power efficiency and the control performance of the robot are verified through comprehensive experiments. Challenging tasks in indoor and outdoor environments demonstrate the capability of DoubleBee to traverse unstructured environments, fly over and move under barriers, and climb steep and rough terrains.

In this study, we introduce AV-PedAware, a self-supervised audio-visual fusion system designed to improve dynamic pedestrian awareness for robotics applications. Pedestrian awareness is a critical requirement in many robotics applications. However, traditional approaches that rely on cameras and LIDARs to cover multiple views can be expensive and susceptible to issues such as changes in illumination, occlusion, and weather conditions. Our proposed solution replicates human perception for 3D pedestrian detection using low-cost audio and visual fusion. This study represents the first attempt to employ audio-visual fusion to monitor footstep sounds for the purpose of predicting the movements of pedestrians in the vicinity. The system is trained through self-supervised learning based on LIDAR-generated labels, making it a cost-effective alternative to LIDAR-based pedestrian awareness. AV-PedAware achieves comparable results to LIDAR-based systems at a fraction of the cost. By utilizing an attention mechanism, it can handle dynamic lighting and occlusions, overcoming the limitations of traditional LIDAR and camera-based systems. To evaluate our approach's effectiveness, we collected a new multimodal pedestrian detection dataset and conducted experiments that demonstrate the system's ability to provide reliable 3D detection results using only audio and visual data, even in extreme visual conditions. We will make our collected dataset and source code available online for the community to encourage further development in the field of robotics perception systems.

Despite recent progress in trajectory planning for multiple robots and a single tethered robot, trajectory planning for multiple tethered robots to reach their individual targets without entanglements remains a challenging problem. In this article, a complete approach is presented to address this problem. First, a multirobot tether-aware representation of homotopy is proposed to efficiently evaluate the feasibility and safety of a potential path in terms of 1) the cable length required to reach a target following the path, and 2) the risk of entanglements with the cables of other robots. Then the proposed representation is applied in a decentralized and online planning framework, which includes a graph-based kinodynamic trajectory finder and an optimization-based trajectory refinement, to generate entanglement-free, collision-free, and dynamically feasible trajectories. The efficiency of the proposed homotopy representation is compared against the existing single and multiple tethered robot planning approaches. Simulations with up to eight UAVs show the effectiveness of the approach in entanglement prevention and its real-time capabilities. Flight experiments using three tethered UAVs verify the practicality of the presented approach. The software implementation is publicly available online.1

In this article, we consider the problem of using a multirobot system to conduct sweep coverage over a region with uneven and unknown workload distribution. Uneven workload distribution means that a robot has to spend different amounts of time covering a unit area at different locations in the region. Unknown workload distribution means that the amount of workload at any location is unknown prior to the operation, hence online sensing and allocation of workload is needed for better efficiency. In this work, we adopt the formulation in which the entire region is separated into multiple stripes, and a discrete-time distributed workload allocation algorithm is used to allocate workload on a stripe to each robot. Previous works that adopt similar formulations do not provide rigorous stability analysis and experimental verification and lack consideration of practical aspects, such as limited sensor range. This work addresses these weaknesses and bridges the gap between theory and practice. First, compared with the existing works, the convergence of the distributed workload allocation algorithm to the optimal workload assignment is established under a more realistic assumption, and less conservative error bounds are derived, which serve as a better indicator of the effectiveness of the algorithm. Second, we propose a new algorithm that addresses the limited sensor range of robots, which is an important constraint in applications, such as agricultural spraying and building inspection. The stability analysis and error bound of the proposed algorithm are also provided. Third, realistic simulations and actual flight experiments using unmanned aerial vehicles are carried out to demonstrate the practicality and validate the theoretical results.

In this article, we focus on enabling the autonomous perception and control of a small unmanned aerial vehicle (UAV) for a façade inspection task. Specifically, we consider the perception as a planar object pose estimation problem by simplifying the building structure as a concatenation of planes, and the control as an optimal reference tracking control problem. First, a vision-based adaptive observer is proposed for plane pose estimation which converges fast and is insensitive to noise under very mild observation conditions. Second, a model predictive controller (MPC) is designed to achieve stable plane following and smooth transition in a multiple-plane scenario, while the persistent excitation (PE) condition of the observer and the maneuver constraints of the UAV are satisfied. The stability of the observer and the MPC controller is also investigated to ensure theoretical completeness. The proposed autonomous plane pose estimation and plane tracking methods are tested in both simulation and practical building façade inspection scenarios, which demonstrate their effectiveness and practicability.

In recent years, onboard self-localization (OSL) methods based on cameras or lidar have achieved many significant progresses. However, some issues such as estimation drift and robustness in low-texture environment still remain inherent challenges for OSL methods. On the other hand, infrastructure-based methods can generally overcome these issues, but at the expense of some installation cost. This poses an interesting problem of how to effectively combine these methods, so as to achieve localization with long-term consistency as well as flexibility compared to any single method. To this end, we propose a comprehensive optimization-based estimator for the 15-D state of an unmanned aerial vehicle (UAV), fusing data from an extensive set of sensors: inertial measurement unit (IMU), ultrawideband (UWB) ranging sensors, and multiple onboard visual-inertial and lidar odometry subsystems. In essence, a sliding window is used to formulate a sequence of robot poses, where relative rotational and translational constraints between these poses are observed in the IMU preintegration and OSL observations, while orientation and position are coupled in thebody-offsetUWB range observations. An optimization-based approach is developed to estimate the trajectory of the robot in this sliding window. We evaluate the performance of the proposed scheme in multiple scenarios, including experiments on public datasets, high-fidelity graphical-physical simulation, and field-collected data from UAV flight tests. The result demonstrates that our integrated localization method can effectively resolve the drift issue, while incurring minimal installation requirements.

In recent years, thanks to the continuously reduced cost and weight of 3D lidar, the applications of this type of sensor in the community have become increasingly popular. Despite many progresses, estimation drift and tracking loss are still prevalent concerns associated with these systems. However, in theory these issues can be resolved with the use of some observations to fixed landmarks in the operation environments. This motivates us to investigate a sensor fusion scheme of lidar and inertia measurements with Ultra-Wideband (UWB) range measurements to such landmarks, which can be easily deployed in the environments with minimal cost and time. Hence, data from IMU, lidar and UWB are tightly-coupled with the robot's states on a sliding window based on their timestamps. Then, we construct a cost function comprising of factors from UWB, lidar and IMU preintegration measurements. Finally an optimization process is carried out to estimate the robot's position and orientation. It is demonstrated through some real world experiments that the method can effectively resolve the drift issue, while only requiring two or three anchors deployed in the environment.

In this article, we investigate the problem of controlling a multirobot team to follow a leader in formation, supported by a relative position estimate derived from distance and self-displacement measurements, thus waiving the need of external localization infrastructure. The main challenge of the problem, which is to simultaneously fulfill both relative localization and control tasks, is efficiently and novelly resolved by embedding a distance-displacement-based persistently excited adaptive relative localization technique into a time-varying formation with bounded control input (PEARL-TVF). By assuming that the leader is globally reachable and by selecting proper parameters, it is shown that the PEARL-TVF ensures exponentially convergent localization, which leads to exponentially convergent formation when the leader's behavior is deterministic, and bounded formation error for a nondeterministic leader. Numerical simulations and experiments on quadcopters are provided to verify the theoretical findings.

Though vision-based techniques have become quite popular for autonomous docking of Unmanned Aerial Vehicles (UAVs), due to limited field of view (FOV), the UAV must rely on other methods to detect and approach the target before vision can be used. In this paper we propose a method combining Ultra-wideband (UWB) ranging sensor with vision-based techniques to achieve both autonomous approaching and landing capabilities in GPS-denied environments. In the approaching phase, a robust and efficient recursive least-square optimization algorithm is proposed to estimate the position of the UAV relative to the target by using the distance and relative displacement measurements. Using this estimate, UAV is able to approach the target until the landing pad is detected by an onboard vision system, then UWB measurements and vision-derived poses are fused with onboard sensor of UAV to facilitate an accurate landing maneuver. Real-world experiments are conducted to demonstrate the efficiency of our method.

As recalling an Unmanned Aerial Vehicle (UAV) after completing a mission requires quite a lot of attention and skill from its operator, in this paper we propose a method to empower UAV with the capability to autonomously return to base and perform precision landing after completing a mission. The main challenge being tackled in this work is that while the vision-based landing technique is already mature, due to GPS error, UAV can only return to within several meters of home position after completing a mission and may fail to detect the visual marker. To resolve this problem, we employ Ultra-wideband (UWB) ranging measurements to localize and approach the home station. Once the UAV detects the visual marker, both UWB and visual tracking information are fused with onboard sensor to achieve even more accurate positioning. Real-life experiment is used to demonstrate the efficacy of the proposed scheme.

In this paper we study the distance-based docking problem of unmanned aerial vehicles (UAVs) by using a single landmark placed at an arbitrarily unknown position. To solve the problem, we propose an integrated estimation-control scheme to simultaneously achieve the relative localization and navigation tasks for discrete-time integrators under bounded velocity: a nonlinear adaptive estimation scheme to estimate the relative position to the landmark, and a delicate control scheme to ensure both the convergence of the estimation and the asymptotic docking at the given landmark. A rigorous proof of convergence is provided by invoking the discrete-time LaSalle's invariance principle, and we also validate our theoretical findings on quadcopters equipped with ultra-wideband ranging sensors and optical flow sensors in a GPS-less environment.