Terrestrial-aerial bimodal vehicles (TABVs) can fly to avoid obstacles and move safely on the ground to save energy, offering enhanced adaptability and flexibility in various challenging environments. However, a robust localization approach becomes a bottleneck to stably applying the TABVs in real-world tasks. Besides the general limitations of visual SLAM methods, large FoV differences between the two modes, abrupt motion strikes in mode transitions, and unstable attitude in ground mode pose great challenges. In this paper, we present an environment-aware robust localization system specifically designed for passive-wheel-based TABVs, which feature two passive wheels alongside a standard quadrotor. The localization system tightly integrates data from multiple sensors, including a stereo camera, Inertial Measurement Units (IMUs), encoders, and single-point laser distance sensors. First, we introduce a terrain-aware odometer model that accurately estimates terrain slope and vehicle's velocity. Then, we propose an anomaly-aware method that senses anomalous sensors and dynamically adjusts the optimization weights accordingly. By explicitly estimating the environmental conditions, such as ground terrain slopes and visual information qualities, the robot can achieve accurate and robust localization results on the ground. To validate our localization approach, we conducted extensive experiments across various challenging scenarios, demonstrating the effectiveness and reliability of our system for real-world applications.

Terrestrial and aerial bimodal vehicles have gained significant interest due to their energy efficiency and versatile maneuverability across different domains. However, most existing passive-wheeled bimodal vehicles rely on attitude regulation to generate forward thrust, which inevitably results in energy waste on producing lifting force. In this work, we propose a novel passive-wheeled bimodal vehicle called TrofyBot that can rapidly change the thrust direction with a single servo motor and a transformable parallelogram linkage mechanism (TPLM). Cooperating with a bidirectional force generation module (BFGM) for motors to produce bidirectional thrust, the robot achieves flexible mobility as a differential driven rover on the ground. This design achieves 95.37% energy saving efficiency in terrestrial locomotion, allowing the robot continuously move on the ground for more than two hours in current setup. Furthermore, the design obviates the need for attitude regulation and therefore provides a stable sensor field of view (FoV). We model the bimodal dynamics for the system, analyze its differential flatness property, and design a controller based on hybrid model predictive control for trajectory tracking. A prototype is built and extensive experiments are conducted to verify the design and the proposed controller, which achieves high energy efficiency and seamless transition between modes.

Ultra-wideband (UWB) has shown promising potential in GPS-denied localization thanks to its lightweight and drift-free characteristics, while the accuracy is limited in real scenarios due to its sensitivity to sensor arrangement and non-Gaussian pattern induced by multi-path or multi-signal interference, which commonly occurs in many typical applications like long tunnels. We introduce a novel neural fusion framework for ranging inertial odometry which involves a graph attention UWB network and a recurrent neural inertial network. Our graph net learns scene-relevant ranging patterns and adapts to any number of anchors or tags, realizing accurate positioning without calibration. Additionally, the integration of least squares and the incorporation of nominal frame enhance overall performance and scalability. The effectiveness and robustness of our methods are validated through extensive experiments on both public and self-collected datasets, spanning indoor, outdoor, and tunnel environments. The results demonstrate the superiority of our proposed IR-ULSG in handling challenging conditions, including scenarios outside the convex envelope and cases where only a single anchor is available.

With the increasing integration of robots into human life, their role in architectural spaces where people spend most of their time has become more prominent. While motion capabilities and accurate localization for automated robots have rapidly developed, the challenge remains to generate efficient, smooth, comprehensive, and high-quality trajectories in these areas. In this paper, we propose a novel efficient planner for ground robots to autonomously navigate in large complex multi-layered architectural spaces. Considering that traversable regions typically include ground, slopes, and stairs, which are planar or nearly planar structures, we simplify the problem to navigation within and between complex intersecting planes. We first extract traversable planes from 3D point clouds through segmenting, merging, classifying, and connecting to build a plane-graph, which is lightweight but fully represents the traversable regions. We then build a trajectory optimization based on motion state trajectory and fully consider special constraints when crossing multi-layer planes to maximize the robot's maneuverability. We conduct experiments in simulated environments and test on a CubeTrack robot in real-world scenarios, validating the method's effectiveness and practicality.

Autonomous navigation of car-like robots on uneven terrain poses unique challenges compared to flat terrain, particularly in traversability assessment and terrain-associated kinematic modelling for motion planning. This paper introduces SEB-Naver, a novel SE(2)-based local navigation framework designed to overcome these challenges. First, we propose an efficient traversability assessment method for SE(2) grids, leveraging GPU parallel computing to enable real-time updates and maintenance of local maps. Second, inspired by differential flatness, we present an optimization-based trajectory planning method that integrates terrain-associated kinematic models, significantly improving both planning efficiency and trajectory quality. Finally, we unify these components into SEB-Naver, achieving real-time terrain assessment and trajectory optimization. Extensive simulations and real-world experiments demonstrate the effectiveness and efficiency of our approach. The code is at https://github.com/ZJU-FAST-Lab/seb_naver.

Terrain analysis is critical for the practical application of ground mobile robots in real-world tasks, especially in outdoor unstructured environments. In this paper, we propose a novel spatial-temporal traversability assessment method, which aims to enable autonomous robots to effectively navigate through complex terrains. Our approach utilizes sparse Gaussian processes (SGP) to extract geometric features (curvature, gradient, elevation, etc.) directly from point cloud scans. These features are then used to construct a high-resolution local traversability map. Then, we design a spatial-temporal Bayesian Gaussian kernel (BGK) inference method to dynamically evaluate traversability scores, integrating historical and real-time data while considering factors such as slope, flatness, gradient, and uncertainty metrics. GPU acceleration is applied in the feature extraction step, and the system achieves real-time performance. Extensive simulation experiments across diverse terrain scenarios demonstrate that our method outperforms SOTA approaches in both accuracy and computational efficiency. Additionally, we develop an autonomous navigation framework integrated with the traversability map and validate it with a differential driven vehicle in complex outdoor environments. Our code will be open-source for further research and development by the community, https://github.com/ZJU-FAST-Lab/FSGP_BGK.

Ultra-wideband (UWB)-vision fusion localization has achieved extensive applications in the domain of multiagent relative localization. The challenging matching problem between robots and visual detection renders existing methods highly dependent on identity-encoded hardware or delicate tuning algorithms. Overconfident yet erroneous matches may bring about irreversible damage to the localization system. To address this issue, we introduce Mr. Virgil, an end-to-end learning multi-robot visual-range relative localization framework, consisting of a graph neural network for data association between UWB rangings and visual detections, and a differentiable pose graph optimization (PGO) back-end. The graph-based front-end supplies robust matching results, accurate initial position predictions, and credible uncertainty estimates, which are subsequently integrated into the PGO back-end to elevate the accuracy of the final pose estimation. Additionally, a decentralized system is implemented for real-world applications. Experiments spanning varying robot numbers, simulation and real-world, occlusion and non-occlusion conditions showcase the stability and exactitude under various scenes compared to conventional methods. Our code is available at: https://github.com/HiOnes/Mr-Virgil.

The tractor-trailer robot consists of a drivable tractor and one or more non-drivable trailers connected via hitches. Compared to typical car-like robots, the addition of trailers provides greater transportation capability. However, this also complicates motion planning due to the robot’s complex kinematics, high-dimensional state space, and deformable structure. To efficiently plan safe, time-optimal trajectories that adhere to the kinematic constraints of the robot and address the challenges posed by its unique features, this paper introduces a lightweight, compact, and high-order smooth trajectory representation for tractor-trailer robots. Based on it, we design an efficiently solvable spatial-temporal trajectory optimization problem. To deal with deformable structures, which leads to difficulties in collision avoidance, we fully leverage the collision-free regions of the environment, directly applying deformations to trajectories in continuous space. This approach not requires constructing safe regions from the environment using convex approximations through collision-free seed points before each optimization, avoiding the loss of the solution space, thus reducing the dependency of the optimization on initial values. Moreover, a multi-terminal fast path search algorithm is proposed to generate the initial values for optimization. Extensive simulation experiments demonstrate that our approach achieves severalfold improvements in efficiency compared to existing algorithms, while also ensuring lower curvature and trajectory duration. Real-world experiments involving the transportation, loading and unloading of goods in both indoor and outdoor scenarios further validate the effectiveness of our method. The source code is accessible at https://github.com/Tracailer/Tracailer.

Differential drive robots are widely used in various scenarios thanks to their straightforward principle, from household service robots to disaster response field robots. The nonholonomic dynamics and possible lateral slip of these robots lead to difficulty in getting feasible and high-quality trajectories. Although there are several types of driving mechanisms for real-world applications, they all share a similar driving principle, which involves controlling the relative motion of independently actuated tracks or wheels to achieve both linear and angular movement. Therefore, a comprehensive trajectory optimization to compute trajectories efficiently for various kinds of differential drive robots is highly desirable. In this paper, we propose a universal trajectory optimization framework, enabling the generation of high-quality trajectories within a restricted computational timeframe for these robots. We introduce a novel trajectory representation based on polynomial parameterization of motion states or their integrals, such as angular and linear velocities, which inherently matches the robots’ motion to the control principle. The trajectory optimization problem is formulated to minimize computation complexity while prioritizing safety and operational efficiency. We then build a full-stack autonomous planning and control system to demonstrate its feasibility and robustness. We conduct extensive simulations and real-world testing in crowded environments with three kinds of differential drive robots to validate the effectiveness of our approach.Note to Practitioners—The Differential drive robot, known for its simple mechanics and high maneuverability, is widely used in many applications. However, current methods have limitations in practice when high-performance motion is needed. Due to the state representation in Cartesian space, path planning makes it difficult to consider nonholonomic constraints directly. The existing trajectory optimization cannot effectively constrain the angular velocity and it is difficult to model forward and backward motion into a continuous trajectory. This paper provides a novel trajectory representation that inherently utilizes the motion performance of differential drive robots, which ensures its universality for different platforms, and reduces the time required to generate trajectories to ensure real-time performance. Based on this, we propose a robust planning and control framework to achieve efficient navigation. We release the source code athttps://zju-fast-lab.github.io/DDR-opt/facilitating expansion and deployment for practitioners. We validate this framework through extensive experiments, demonstrating its capability to navigate challenging environments.

Perception is necessary for autonomous navigation in an unknown area crowded with obstacles. It’s challenging for a robot to navigate safely without any sensors that can sense the environment, resulting in a blind robot, and becomes more difficult when comes to a group of robots. However, it could be costly to equip all robots with expensive perception or SLAM systems. In this paper, we propose a novel system named ColAG, to solve the problem of autonomous navigation for a group of blind UGVs by introducing cooperation with one UAV, which is the only robot that has full perception capabilities in the group. The UAV uses SLAM for its odometry and mapping while sharing this information with UGVs via limited relative pose estimation. The UGVs plan their trajectories in the received map and predict possible failures caused by the uncertainty of its wheel odometry and unknown risky areas. The UAV dynamically schedules waypoints to prevent UGVs from collisions, formulated as a Vehicle Routing Problem with Time Windows to optimize the UAV’s trajectories and minimize time when UGVs have to wait to guarantee safety. We validate our system through extensive simulation with up to 7 UGVs and real-world experiments with 3 UGVs.

Autonomous target tracking with quadrotors has wide applications in many scenarios, such as cinematographic follow-up shooting or suspect chasing. Target motion prediction is necessary when designing the tracking planner. However, the widely used constant velocity or constant rotation assumption can not fully capture the dynamics of the target. The tracker may fail when the target happens to move aggressively, such as sudden turn or deceleration. In this paper, we propose an intention-aware planner by additionally considering the intention of the target to enhance safety and robustness in aerial tracking applications. Firstly, a designated intention prediction method is proposed, which combines a user-defined potential assessment function and a state observation function. A reachable region is generated to speci cally evaluate the turning intentions. Then we design an intention-driven hybrid A* method to predict the future possible positions for the target. Finally, an intention-aware optimization approach is designed to generate a spatial-temporal optimal trajectory, allowing the tracker to perceive unexpected situations from the target. Benchmark comparisons and real-world experiments are conducted to validate the performance of our method.

Tracked robots with reconfigurable mechanisms exhibit great maneuverability due to their adaptability to complex ground conditions. Reconfigurable tracked robots with geometry-changing tracks show further obstacle-crossing capabilities with compact dimensions. However, existing systems face deployment limitations due to either complex transmission mechanisms or unsustainable designs when maintaining the tension in the tracks. To address these challenges, we introduce a novel design of a reconfigurable tracked robot with geometry-changing tracks, which achieves strong terrain traversability with good mechanical properties. We achieve the elliptical trajectory of key planetary wheels through a novel Quad-slider Elliptical Trammel Mechanism (Qs-ETM), allowing the tracks to maintain fixed tension while changing their geometry. Furthermore, the combination of direct drive motors significantly enhances its mechanical properties and agility. A detailed analysis of the kinematic and dynamic characteristics has been conducted and proved with a series of simulations. We built a fully functional prototype of the design and tested it in real-world experiments to validate its advantages. The result shows that our design can reduce the torque required by up to 68.3% and the shear stress of the flipper by up to 67.1%.

Perching on the moving platforms is a promising solution to enhance the endurance and operational range of quadrotors, which could benefit the efficiency of a variety of air ground cooperative tasks. To ensure robust perching, tracking with a steady relative state and reliable perception is a prerequisite. This paper presents an adaptive dynamic tracking and perching scheme for autonomous quadrotors to achieve tight integration with moving platforms. For reliable perception of dynamic targets, we introduce elastic visibility aware planning to actively avoid occlusion and target loss. Additionally, we propose a flexible terminal adjustment method that adapts the changes in flight duration and the couple d terminal states, ensuring full state synchronization with the time varying perching surface at various angles. A relaxation strategy is developed by optimizing the tangential relative speed to address the dynamics and safety violations brought by hard bo undary conditions. Moreover, we take SE(3) motion planning into account to ensure no collision until the contact moment. Furthermore, we propose an efficient spatiotemporal trajectory optimization framework considerin g full state dynamics The proposed method is extensively tested through benchmark comparisons and ablation studies. To facilitate the application of academic research to industry and to validate the efficiency under strictly limited computational resources, we deploy our system on a commercial drone (DJI MAVIC3) with a full size sport utility vehicle (SUV). We conduct extensive real world experiments, where the drone successfully tracks and perches at 30 km/h (8.3 m/ s) on the top of the SUV, and at 3.5∼m/s with 60° inclined into the trunk of the SUV.

With the development of robotics, ground robots are no longer limited to planar motion. Passive height variation due to complex terrain and active height control provided by special structures on robots require a more general navigation planning framework beyond 2D. Existing methods rarely considers both simultaneously, limiting the capabilities and applications of ground robots. In this paper, we proposed an optimization-based planning framework for ground robots considering both active and passive height changes on the z-axis. The proposed planner first constructs a penalty field for chassis motion constraints defined in$\mathbb{R}^{3}$such that the optimal solution space of the trajectory is continuous, resulting in a high-quality smooth chassis trajectory. Also, by constructing custom constraints in the z-axis direction, it is possible to plan trajectories for different types of ground robots which have z-axis degree of freedom. We performed simulations and real-world experiments to verify the efficiency and trajectory quality of our algorithm.

Service robots have attracted extensive attention due to specially designed functions, such as mobile manipulators or robots with extra structures. For robots that have changing shapes, autonomous navigation in the real world presents new challenges. In this paper, we propose a trajectory optimization method for differential-drive mobile robots with controllable changing shapes in dense 3D environments. We model the whole-body trajectory as a polynomial trajectory that satisfies the nonholonomic dynamics of the base and dynamics of the extra joints. These constraints are converted into soft constraints, and an activation function for dense sampling is applied to avoid nonlinear mutations. In addition, we guarantee the safety of full shape by limiting the system's distance from obstacles. To comprehensively simulate a large extent of height and width changes, we designed a novel Shape-Changing Robot with a Differential Base (SCR-DB). Our global trajectory optimization gives a smooth and collision-free trajectory for SCR-DB at a low computational cost. We present vast simulations and real-world experiments to validate our performance, including coupled whole-body and independent differential-driven vehicle motion planning.

For letting mobile robots travel flexibly through complicated environments, increasing attention has been paid to the whole-body collision evaluation. Most existing works either opt for the conservative corridor-based methods that impose strict requirements on the corridor generation, or ESDF-based methods that suffer from high computational overhead. It is still a great challenge to achieve fast and accurate whole-body collision evaluation. In this paper, we propose a Robo-centric ESDF (RC-ESDF) that is pre-built in the robot body frame and is capable of seamlessly applied to any-shape mobile robots, even for those with non-convex shapes. RC-ESDF enjoys lazy collision evaluation, which retains only the minimum information sufficient for whole-body safety constraint and significantly speeds up trajectory optimization. Based on the analytical gradients provided by RC-ESDF, we optimize the position and rotation of robot jointly, with whole-body safety, smoothness, and dynamical feasibility taken into account. Extensive simulation and real-world experiments verified the reliability and generalizability of our method.

Autonomous navigation of ground robots on uneven terrain is being considered in more and more tasks. However, uneven terrain will bring two problems to motion planning: how to assess the traversability of the terrain and how to cope with the dynamics model of the robot associated with the terrain. The trajectories generated by existing methods are often too conservative or cannot be tracked well by the controller since the second problem is not well solved. In this paper, we propose terrain pose mapping to describe the impact of terrain on the robot. With this mapping, we can obtain the SE(3) state of the robot on uneven terrain for a given state in SE(2). Then, based on it, we present a trajectory optimization framework for car-like robots on uneven terrain that can consider both of the above problems. The trajectories generated by our method conform to the dynamics model of the system without being overly conservative and yet able to be tracked well by the controller. We perform simulations and real-world experiments to validate the efficiency and trajectory quality of our algorithm.

Mutual localization plays a crucial role in multi-robot cooperation. CREPES, a novel system that focuses on six degrees of freedom (DOF) relative pose estimation for multi-robot systems, is proposed in this paper. CREPES has a compact hardware design using active infrared (IR) LEDs, an IR fish-eye camera, an ultra-wideband (UWB) module and an inertial measurement unit (IMU). By leveraging IR light communication, the system solves data association between visual detection and UWB ranging. Ranging measurements from the UWB and directional information from the camera offer relative 3-DOF position estimation. Combining the mutual relative position with neighbors and the gravity constraints provided by IMUs, we can estimate the 6-DOF relative pose from a single frame of sensor measurements. In addition, we design an estimator based on the error-state Kalman filter (ESKF) to enhance system accuracy and robustness. When multiple neighbors are available, a Pose Graph Optimization (PGO) algorithm is applied to further improve system accuracy. We conduct enormous experiments to demonstrate CREPES’ accuracy between robot pairs and a team of robots, as well as performance under challenging conditions.

Terrestrial and aerial bimodal vehicles have gained widespread attention due to their cross-domain maneuverability. Nevertheless, their bimodal dynamics significantly increase the complexity of motion planning and control, thus hindering robust and efficient autonomous navigation in unknown environments. To resolve this issue, we develop a model-based planning and control framework for terrestrial aerial bi-modal vehicles. This work begins by deriving a unified dynamic model and the corresponding differential flatness. Leveraging differential flatness, an optimization-based trajectory planner is proposed, which takes into account both solution quality and computational efficiency. Moreover, we design a tracking controller using nonlinear model predictive control based on the proposed unified dynamic model to achieve accurate trajectory tracking and smooth mode transition. We validate our framework through extensive benchmark comparisons and experiments, demonstrating its effectiveness in terms of planning quality and control performance.

Formation flight has a vast potential for aerial robot swarms in various applications. However, the existing methods lack the capability to achieve fully autonomous large-scale formation flight in dense environments. To bridge the gap, we present a complete formation flight system that effectively integrates real-world constraints into aerial formation navigation. This article proposes a differentiable graph-based metric to quantify the overall similarity error between formations. This metric is invariant to rotation, translation, and scaling, providing more freedom for formation coordination. We design a distributed trajectory optimization framework that considers formation similarity, obstacle avoidance, and dynamic feasibility. The optimization is decoupled to make large-scale formation flights computationally feasible. To improve the elasticity of formation navigation in highly constrained scenes, we present a swarm reorganization method that adaptively adjusts the formation parameters and task assignments by generating local navigation goals. A novel swarm agreement strategy called global-remap-local-replan and a formation-level path planner is proposed in this article to coordinate the global planning and local trajectory optimizations.To validate the proposed method, we design comprehensive benchmarks and simulations with other cutting-edge works in terms of adaptability, predictability, elasticity, resilience, and efficiency. Finally, integrated with palm-sized swarm platforms with onboard computers and sensors, the proposed method demonstrates its efficiency and robustness by achieving the largest scale formation flight in dense outdoor environments.

Many applications related to the Internet of Things, such as tracking people or objects, robotics, and monitoring require the localization of large networks of devices in dynamic, GPS-denied environments. Ultrawideband (UWB) technology is a common choice because of its precise ranging capability. However, allowing access and effective use of the shared UWB medium with a constantly changing set of devices faces some particular challenges: high frequency of ranging measurements by the devices to improve system accuracy; network topology changes requiring rapid adaptation; and decentralized operation to avoid single points of failure. In this article, we propose a novel time-division multiple access (TDMA) algorithm that can quickly schedule the use of the UWB medium by a large network of devices without collisions in local network neighborhoods and avoiding conflicts with hidden terminals, all the while maximizing network usage. Using exclusively the UWB radio network, we realize a decentralized system for synchronization, dynamic TDMA scheduling, and precise relative positioning on a multihop network. Our system does not have special nodes (all nodes are equal) and it is sufficiently scalable for real-world applications. Our method can be applied to implement device localization services in large spaces without GPS and complex topologies, such as malls, museums, mines, etc. We demonstrate our method in simulation and on real hardware in an underground parking lot, showing the effectiveness of its TDMA schedule for relative localization.

Accurate localization and tracking are a fundamental requirement for robotic applications. Localization systems like GPS, optical tracking, simultaneous localization and mapping (SLAM) are used for daily life activities, research, and commercial applications. Ultra-wideband (UWB) technology provides another venue to accurately locate devices both indoors and outdoors. In this paper, we study a localization solution with a single UWB anchor, instead of the traditional multi-anchor setup. Besides the challenge of a single UWB ranging source, the only other sensor we require is a low-cost 9 DoF inertial measurement unit (IMU). Under such a configuration, we propose continuous monitoring of UWB range changes to estimate the robot speed when moving on a line. Combining speed estimation with orientation estimation from the IMU sensor, the system becomes temporally observable. We use an Extended Kalman Filter (EKF) to estimate the pose of a robot. With our solution, we can effectively correct the accumulated error and maintain accurate tracking of a moving robot.

A series of previous work has found that the environmental constraint (EC), which is the natural result of the contact of the robot and the interacting objects, is immensely helpful for the realization of high-precision robotic tasks. However, due to the existence of multifarious errors, such as mechanical error, modeling error and sensing error, there would be discrepancy between the actual constraints and the ideal models. In such case, it is hard to realize manipulation with EC-based strategies. Inspired by human, a preliminary framework which aims at the integration of the coarse sensing information and the environmental constraints is proposed for robotic manipulation. By mapping the sensing information into the new space, where the environmental constraint can be formally described, the region that integrates the sensing information and the environmental constraint is constructed and the conditions to achieve high-precision manipulation are derived. Based on the conditions, a motion planning strategy is proposed to achieve the required task. The effectiveness of this strategy is verified by case studies.

An improved unit module of a novel mobile self-assembling modular robotic system is presented in this paper. Two continuous rotational DoFs are used in each module to implement both valuable self-locomotion and flexible reconfiguration. To achieve efficient exploration, unit module can implement two-dimensional locomotion independently and freely in a range of surface conditions in real world, even in environments with certain terrain challenges. With the help of three active connection mechanisms (ACMs), the module has great potential in assembling and reconfiguration to form complex three-dimensional structures. Preliminary locomotion tests in different environments demonstrate its effective mobility and potential applications for exploration. Several useful and easy realized configurations are explained with simulations at last.

A publicly verifiable scheme for the copyright protection of digital image is proposed in this paper. Combining with some cryptographic techniques such as digital signature and timestamp, the scheme features an idea of registering watermark information to a trusted authority (TA) rather than embedding it into the host image, which overcomes many deficiencies of the conventional watermarking algorithm. In the scheme, generalized brain-state-in-a-box neural network (gBSB) and error diffusion halftoning are employed to extract the robust feature, which is further used to generate the verification information registered to TA, from the original image. Experimental results demonstrate that the feature is adequately robust to make sure that the verifier will be able to extract the logo mark from the attacked image correctly. The proposed scheme is competent to be applied to the copyright protection of digital multimedia.