Point cloud registration aims to align point clouds into a unified coordinate system, which serves as a foundational task for multivehicle collaborative mapping in outdoor environments. However, the significant perspective difference between aerial and ground vehicles results in remarkably low overlap between their point clouds, presenting a substantial challenge for accurate registration. To address this, we propose a real-time registration method with morphology-based representation for aerial-ground vehicles operating in diverse outdoor scenarios. In order to accurately model the ground surface, we perform a morphological operation to extract ground points, ensuring adaptability to diverse scenarios. For the purpose of identifying the overlapping region between aerial–ground point clouds, we stratify the point clouds upward from the ground and locate the regions with the most similar frequency distribution, overcoming the significant aerial–ground perspective difference. Toward real-time registration, we concentrate the graph-based maximal clique searching within the identified overlap regions, reducing the search space and computational cost. Extensive experiments on one self-recorded and one public aerial-ground point cloud datasets demonstrate that our proposed method achieves state-of-the-art registration accuracy while maintaining high computational efficiency.

Ground and aerial robots, with distinct sensing perspectives, acquire heterogeneous point clouds that exhibit limited overlap, presenting significant challenges for collaborative mapping. To address these challenges, this article proposes a robust LiDAR-based aerial-ground collaborative mapping framework for large-scale outdoor environments. Firstly, to perform reliable cross-source place recognition and detect loop closure between aerial-ground robots, a deep network that fuses multi-level bird’s-eye view (BEV) and geometric features is developed to ensure consistent feature extraction and emphasize overlaps between heterogeneous point clouds. Next, an overlap-aware registration method is proposed to align point clouds within a detected loop closure. This method can strategically perform point cloud sparsification based on overlap ratio estimation, and mitigate the adverse effects of interfering points in non-overlapping regions. Furthermore, a graph optimization is implemented to consider all loop closure constraints simultaneously and ensure global map consistency. Comparative experiments on public and self-collected datasets demonstrate the superiority of the proposed approach. We open source code on GitHub1to benefit the community.

Recently, the research on mobile manipulators has attracted increasing attention. Ensuring that mobile manipulators can meet obstacle avoidance constraints and efficiently accomplish assigned tasks in dynamic environments remains a significant challenge. To address this issue, this paper proposes an integrated framework for environment perception, real-time planning, and control optimization. Firstly, we develop a fusion map that combines euclidean signed distance field (ESDF) with clustered point clouds occupying cubes, enabling robots to perceive more precise environmental information in complex and changing conditions. Secondly, we introduce a novel rapid generation strategy for 6-DOF guide point sequences, which directs the mobile manipulator to follow the most efficient path to the target location while making real-time adjustments to avoid dynamic obstacles. Additionally, utilizing optimized nonlinear model predictive control (NMPC), we design a whole-body motion controller for the mobile manipulator to prevent the system from becoming trapped in local optima, thereby allowing the manipulator to adjust its state tracking guide points promptly in complex indoor environments. Finally, the proposed algorithm was implemented on a mobile manipulator with an Ackerman base and tested through both simulations and real-world experiments.

This article proposes a perception-complementarity-driven trajectory generating method for multiple unmanned aerial vehicles (UAVs), which can effectively enhance the visibility of the target for UAVs in unknown environments. rgb0,0,0Traditional methods often rely on prior maps or additional sensors to assist with obstacle avoidance. Nevertheless, these methods are not only costly but also fail to effectively avoid occlusions caused by obstacles. rgb0,0,0Different from existing methods, the trajectory planned by the proposed method not only enables the vision-based UAVs to maintain the perception of obstacles and the target on the one hand, but also preserves topological equivalence with the predicted target trajectory on the other hand. Specifically, a vision-based mutual observation approach among UAVs is proposed to enhance the overall perception capability of the UAVs system. On this basis, a target-guided collaborative trajectory planning method is proposed to ensure the planned collision-free trajectory for other UAV in the formation maintains target visibility. In addition, a trajectory feasibility assessment method is proposed to obtain the collaborative trajectory planned by the UAV at the optimal observation location. Finally, comparative simulations are conducted with three state-of-the-art methods, demonstrating the advantages of the proposed method in maintaining target visibility and tracking efficiency during the vision-based aerial tracking. The real-world experiment demonstrates the feasibility of the proposed method.Note to Practitioners—Most existing vision-based multi-UAV target tracking methods require additional prior maps or laser sensors to assist in obstacle avoidance. In practical tracking scenarios, the limited perception range of cameras poses significant challenges for UAVs in synchronously observing moving target and environmental obstacles. The article proposes a trajectory generation method based on perceptual complementarity to ensure that the generated trajectory effectively perceives surrounding obstacles while enhancing the observation capability of UAVs towards moving target. The key insight of this work is to utilize mutual observation among multiple UAVs to assist in planning collision-free tracking trajectories, ensuring safety during the tracking process. Building upon this, by preserving topological equivalence with the predicted target trajectory, there is an improvement in the target visibility ratio during the tracking process. Furthermore, a trajectory feasibility assessment method is proposed to obtain the optimal collaborative trajectory from the UAV positioned at the optimal location in the formation. The effectiveness of the proposed method in improving flight safety and target visibility is validated through comparative experiments.

Cross-source place recognition is the foundation of collaborative mapping tasks between Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs) to achieve loop closure and global localization in outdoor environments. Due to the dynamic changes in communication bandwidth, the point density of point clouds transmitted from the robots to the server will also change simultaneously, which significantly affects the accuracy of place recognition. This article proposes a hierarchical BEV fusion network for unmanned aerial-ground vehicles with low-overlap and varying-density point clouds, called HBFusion. To improve the accuracy of place recognition for low-overlap point clouds, a multi-resolution voxel feature aggregation module based on sparse convolution is proposed to aggregate voxel features from multi-scale receptive fields to point features, which improves the similarity of global descriptors between low-overlap point clouds using rich point features. To adapt to point density variations caused by dynamic changes in communication bandwidth, we propose a hierarchical BEV fusion module to extract multi-layer BEV features at different heights, which improves the adaptability of global descriptors to point density variations using BEV representations. Extensive experiments performed on GrAco, a public aerial-ground dataset, and DUT-GA, our self-recorded aerial-ground dataset with low-overlap point clouds, indicate that our network achieves state-of-the-art performance in accuracy, even under the condition of varying point densities of point clouds.

This paper proposes a mapping uncertainty-aware point-wise Lidar Inertial Odometry (LIO), which synthesizes the point-wise point-to-plane match and map refreshment into a probabilistic model. As a result, it can address the issue of mismatching during point registration and remove in-frame motion distortion of Lidar sensors. Specifically, the uncertainty-aware map is designed to embody the uncertainty of map geometric features (points and planes), which comes from the Lidar point measurement and pose estimation. Then the map can be modeled in a probabilistic form. In addition, the proposed framework refreshes map at each Lidar point measurement to timely revise geometric features and provide non-delayed map. On the basis, the probabilistic point-to-plane match method is designed to seek a corresponding plane for each Lidar point in point registration, which can enhance the effectiveness of match and provide adaptive observation noises for more accurate state estimation. Comparative experiments on various public datasets are conducted to demonstrate the superior performance of the proposed framework in terms of higher accuracy and better robustness.

Assurance cases are used to communicate and assess confidence in critical system properties, e.g., safety and security. Historically, assurance cases have been manually created documents, validated by engineers through lengthy and error-prone processes. Recently, system assurance practitioners have begun adopting model-based approaches to improve the efficiency and quality of system assurance activities. This becomes increasingly important, for example, to ensure the safety of robotics and autonomous systems (RASs), as they are adopted into society. Such systems can be highly complex, and so it is a challenge to manage the development life-cycle and improve efficiency, including coordination of validation activities, and change impact analysis in interconnected system assurance artifacts. However, adopting model-based approaches require skills in the model management languages, which system assurance practitioners may not be acquainted with. In this article, we contribute an automated validation framework for the model-based assurance cases, which promotes the usage of a constrained natural language (CNL), that can be automatically transformed and executed against engineering models involved in assurance case development. We apply our approach to a case study based on an autonomous underwater vehicle (AUV).

This article proposes a frontier-guided informative planner for unmanned aerial vehicle volumetric exploration and 3-D reconstruction, which can explore a complex unknown environment and provide the accurate truncated signed distance function reconstruction simultaneously. Different from the existing methods, the key insight of the proposed method is that the hybrid surface frontier is proposed to guide both the tree expansion of dynamic rapidly exploring random tree star and the informative trajectory generation. As a result, the proposed planner can achieve more efficient volumetric exploration with higher reconstruction quality. Specifically, hybrid global–local surface frontiers are designed to guide the potential viewpoints sampling and tree expansion, which results in directional exploration. Then, the hybrid surface frontiers are further leveraged to guide the candidate paths generation. On the basis, the path maximizing the new comprehensive gain is selected for the following B-spline trajectory optimization, which can further improve the reconstruction quality. Comparative simulation and real-world experiments are conducted to demonstrate the superior performance of the proposed method including the exploration efficiency and reconstruction quality.

Recent advances in trajectory planning have enabled quadrotor unmanned aerial vehicles (UAVs) to navigate autonomously in complex environments. However, most of the existing methods do not consider the perception quality and the safety simultaneously. This article proposes a perception-aware trajectory planning strategy for quadrotors, which can ensure the safety and localization accuracy. In contrast to the existing methods, the main idea of the proposed method lies in that the yaw angle trajectory is planned to actively obtain more information in the environment to improve the localization accuracy and keep the safe flight simultaneously. Following the mainstream two-stage motion planning framework, a coarse-to-fine graph search strategy is proposed to search for a safe and perception-aware yaw angle path in the first stage. Specifically, a Yaw Safety Corridor (YSC) is proposed to guarantee the safety, which can observe the obstacles directly along the tangent direction of the position trajectory. In addition, a dedicated map Fisher Information Field (FIF) is employed to evaluate the perception quality. In the second stage, a path-guided optimization method is proposed to quickly generate a safe and perception-aware trajectory. Finally, comparative simulation and real-world experiments are conducted to verify the superior performance in terms of the perception quality and the safety of the proposed method.

In this article, a novel visual-inertial-lidar odometry (VILO) system named TCK-VILO is proposed to assist unmanned ground vehicles in reaching high localization performance. We introduce an adaptive sensor reliability evaluation method to configure weights of visual and lidar measurements dynamically in the backend optimization, which can improve both the accuracy and robustness of the localization in real-world outdoor scenarios. A two-stage initialization method is proposed to initialize the system by solving the spatio-temporal alignment of system states. In the TCK-VILO system, a lightweight frontend is designed to update system states while lidar or visual data is fed into the system, and then a tightly-coupled and keyframe-based backend is used to refine system states. We evaluate the proposed TCK-VILO pipeline on the public Newer College dataset and a self-collected real-world dataset. Experimental results show that our system can achieve low drifts in various challenging scenes and outperforms competing state-of-the-art VILO systems.

The perception-aware motion planning method based on the localization uncertainty has the potential to improve the localization accuracy for robot navigation. How-ever, most of the existing perception-aware methods pre-build a global feature map and can not generate the perception- aware trajectory in real time. This paper proposes a topology- guided perception-aware receding horizon trajectory generation method, which contains a topology-guided position trajectory generation and a perception-aware yaw angle trajectory generation. Specifically, a memorable active map is built by selectively storing the visual landmarks. After that, a library of candidate topological trajectories are generated, which are then evaluated in terms of the perception quality based on the active map, smoothness, collision possibility and feasibility. In addition, the yaw angle trajectory is obtained through a front-end multiple refined path search and a back-end path- guided trajectory optimization. Comparative simulation and real-world experiments are carried out to confirm that the proposed method can keep more visual features in view and reduce the localization error.

This article proposes a novel asymptotic robust control approach for rotorcraft unmanned aerial vehicles (UAVs), which can effectively eliminate the impact of external disturbances and the model uncertainties. Different from existing works, the proposed method alleviates the assumption that disturbances should have no variations in the existing observers for uncertainties. In addition, the equilibrium point of the entire observer-controller system is asymptotically stable without the assumption of the boundness of the outer-loop signals or the time-scale separation assumption. Specifically, two observer-based estimators are designed to estimate the model uncertainty and the external disturbance for the force and torque, respectively. On this basis, a nonlinear hierarchical tracking controller is then proposed with the feedforward compensated disturbance term. Despite the nonlinear coupled dynamics and the disturbances, a generic framework for the stability analysis is proposed to yield the asymptotic stability of the equilibrium point of the entire controller-observer system. Comparative experiments are conducted to show the superior performance of the proposed approach in terms of higher tracking accuracy and stronger robustness. Note to Practitioners—Most of the existing observer-based control approaches can only govern the rotorcraft closed-loop system to be ultimately uniformly bounded. The highly coupled dynamics and mismatched uncertainties in practice make the effective asymptotic robust control of rotorcrafts very challenging. A novel robust control approach for rotorcraft UAVs is proposed to yield the asymptotic stability of the equilibrium point despite the nonlinear coupled dynamics and the disturbances. The key insight of this work to guarantee the asymptotic stability of the system is that the nominal signals (i.e., the output of the nominal auxiliary dynamics) are fed back to the controller. In addition, the attitude error signal is proved to be exponentially convergent, which can further help prove the asymptotic stability of the entire controller-observer system. Comparative experiments are conducted to show the applicability of the proposed approach.

Whilst virtualization was once restricted to large-scale computing platforms, and it is now widely deployed on modern embedded computing systems. This has been driven by the availability of hardware support which alleviates the performance penalties incurred by traditional software virtualization technologies. In the domain of hard real-time systems, specialist virtualization technology which respects restricted timing requirements and constraints can be deployed to allow sharing of processors. However, other aspects of the embedded system (I/O, memory, and communication) are harder to analyse. In this paper, we argue that in order to support real-time virtualization on modern embedded systems, additional system-wide hardware support is required. We propose BlueVisor, an analyzable and scalable hardware hypervisor for many-core embedded systems, which enables time-predictable CPU, memory, and I/O virtualization, as well as supporting a fast interrupt handler, and inter-VM communication. We describe the design and implementation of the real-time hypervisor and demonstrate how a BlueVisor-based virtualization system can be leveraged to meet real-time requirements with significant improvement in system performance, and with a low-performance cost when executing different types of software.

An increasingly important trend in the design of safety-critical systems is the integration of components with different levels of criticality onto a common hardware platform. Mixed-criticality systems (MCSs) have been well researched in academia, but can be difficult to implement in industrial scenarios as the theoretical models underpinning the research do not sufficiently consider industrial safety practice and safety standards. In this article, we make the first attempt toward the implementation of the MCS theoretical model in industrial settings. To this end, we identify the pragmatic gaps between theory and practice, and then propose a generic industrial MCS architecture, termedP-MCS(Practical-MCS).P-MCSis built upon the conventional theoretical MCS model with additional considerations of industrial safety requirements: 1) runtime safety analysis, determining preserved applications in each system mode and 2) correct partitioning and isolation of different critical elements. We introduce three implementing methods forP-MCS. Corresponding to the new system architecture, we present a theoretical model and schedulability analysis (with consideration of shared resources) to ensure system predictability. Finally, we evaluate and demonstrateP-MCSin terms of system schedulability, overheads, throughput, and predictability, along with a real-world case study. As shown in the evaluation, the considerations of industrial requirements lead to extra overheads and performance reduction inP-MCS. Such weaknesses can be considerably mitigated by hardware assistance and acceleration.

This article proposes a novel deep-learning framework, called RSSP, for real-time 3-D scene understanding with LiDAR sensors. To this end, we introduce new sparse strided operations based on the sparse tensor representation of point clouds. Compared with conventional convolution operations, the time and space complexity of our sparse strided operations are proportional to the number of occupied voxels${N}$rather than the input spatial size${r} ^{3}$(oftenN$\ll $r3for LiDAR data). This enables our method to process point clouds at high resolutions (e.g., 20483) with a high speed (130 ms for classifying a single frame from Velodyne HDL-64). The main structure includes a CNN model built upon our sparse strided operations and a conditional random field (CRF) model to impose spatial consistency on the final predictions. A highly parallel implementation of our system is presented for both CPU-GPU and CPU-only environments. The efficiency and effectiveness of our approach are demonstrated on two public datasets (Semantic3D.net and KITTI). The experimental results and benchmark tests show that our system can be effectively applied for online 3-D data analyses with comparable or better accuracy than the state-of-the-art methods.

Convolutional neural networks (CNNs) for 3-D data analyses require a large size of memory and fast computation power, making real-time applications difficult. This article proposes a novel OctreeNet (a sparse 3-D CNN) to analyze the sparse 3-D laser scanning data gathered from outdoor environments. It uses a collection of shallow octrees for 3-D scene representation to reduce the memory footprint of 3-D-CNNs and performs point cloud classification on every single octree. Furthermore, the smallest non-trivial and non-overlapped kernel (SNNK) implements convolution directly on the octree structure to reduce dense 3-D convolutions to matrix operations at sparse locations. The proposed neural network implements a depth-first search algorithm for real-time predictions. A conditional random field model is utilized for learning global semantic relationships and refining point cloud classification results. Two public data sets (Semantic3D.net and Oakland) are selected to test the classification performance in outdoor scenes with different spatial sparsity. The experiments and benchmark test results show that the proposed approach can be effectively used in real-time 3-D laser data analyses. Note to Practitioners-This article was motivated by the limitations of existing deep learning technologies for analyzing 3-D laser scanning data. This technology enables robots to infer what the surroundings are, which is closely linked to semantic mapping and navigation tasks. Previous deep neural networks have seldom been used in robotic systems since they require a large amount of memory and fast computation power to apply dense 3-D operations. This article presents a sparse 3-D-Convolutional neural network (CNN) for real-time point cloud classification by exploiting the sparsity of 3-D data. This framework requires no GPUs. The practicality of the proposed method is verified on data sets gathered from different platforms and sensors. The proposed network can be adopted for other classification tasks with laser sensors.

Multiple images have been widely used for scene understanding and navigation of unmanned ground vehicles in long term operations. However, as the amount of visual data in multiple images is huge, the cumulative error in many cases becomes untenable. This paper proposes a novel method that can extract features from a large dataset of multiple images efficiently. Then the membership K-means clustering is used for high dimensional features, and the large dataset is divided into N subdatasets to train N conditional random field (CRF) models based on superpixel. A Softmax subdataset selector is used to decide which one of the N CRF models is chosen as the prediction model for labeling images. Furthermore, some experiments are conducted to evaluate the feasibility and performance of the proposed approach.

A challenging problem in object recognition is to train a robust classifier with small and imbalanced data set. In such cases, the learned classifier tends to overfit the training data and has low prediction accuracy on the minority class. In this paper, we address the problem of class imbalanced object recognition by combining synthetic minorities over-sampling technique (SMOTE) and instance-based transfer boosting to rebalance the skewed class distribution. We present ways of generating synthetic instances under the learning framework of transfer Adaboost. A novel weighted SMOTE technique (WSMOTE) is proposed to generate weighted synthetic instances with weighted source and target instances at each boosting round. Based on WSMOTE, we propose a novel class imbalanced transfer boosting algorithm called WSMOTE-TrAdaboost and experimentally demonstrate its effectiveness on four datasets (Office, Caltech256, SUN2012, and VOC2012) for object recognition application. Bag-of-words model with SURF features and histogram of oriented gradient features are separately used to represent an image. We experimentally demonstrated the effectiveness and robustness of our approaches by comparing it with several baseline algorithms in boosting family for class imbalanced learning.

In this paper, we introduce a new research problem termed online feature transformation learning in the context of multiclass object category recognition. The learning of a feature transformation is viewed as learning a global similarity metric function in an online manner. We first consider the problem of online learning a feature transformation matrix expressed in the original feature space and propose an online passive aggressive feature transformation algorithm. Then these original features are mapped to kernel space and an online single kernel feature transformation (OSKFT) algorithm is developed to learn a nonlinear feature transformation. Based on the OSKFT and the existing Hedge algorithm, a novel online multiple kernel feature transformation algorithm is also proposed, which can further improve the performance of online feature transformation learning in large-scale application. The classifier is trained with k nearest neighbor algorithm together with the learned similarity metric function. Finally, we experimentally examined the effect of setting different parameter values in the proposed algorithms and evaluate the model performance on several multiclass object recognition data sets. The experimental results demonstrate the validity and good performance of our methods on cross-domain and multiclass object recognition application.

This paper investigates the problem of multiclass and multiview 3-D object detection for service robots operating in a cluttered indoor environment. A novel 3-D object detection system using laser point clouds is proposed to deal with cluttered indoor scenes with a fewer and imbalanced training data. Raw 3-D point clouds are first transformed to 2-D bearing angle images to reduce the computational cost, and then jointly trained multiple object detectors are deployed to perform the multiclass and multiview 3-D object detection. The reclassification technique is utilized on each detected low confidence bounding box in the system to reduce false alarms in the detection. The RUS-SMOTEboost algorithm is used to train a group of independent binary classifiers with imbalanced training data. Dense histograms of oriented gradients and local binary pattern features are combined as a feature set for the reclassification task. Based on the dalian university of technology (DUT)-3-D data set taken from various office and household environments, experimental results show the validity and good performance of the proposed method.

Online 3D point cloud classification and scene understanding are crucial tasks for Unmanned Ground Vehicles (UGVs) equipped with multiple laser scanners. Due to the poor performance of traditional 2D image representation model for 3D point clouds, a novel Optimal Bearing Angle (OBA) model is therefore proposed to overcome the limitations of texture information losing and image blurring caused by the UGV's on-the-fly navigation. With the result of 3DSLIC based super-pixel segmentation in OBA images, the center of points belonging to each segmented OBA image patch is assigned as CRF graph node, so that a simplified CRF graph structure is constructed for online contextual classification of 3D laser points in urban environments. Moreover, total 29-dimensional features are extracted from both the raw 3D laser points and the corresponding OBA images. A large number of urban scenes selected from both DUT2 dataset and KAIST dataset are used as testing data in our experiments, and the results show the validity and performance of the proposed method.

Robust scene recognition serves as an essential task for robots to work within a complex dynamic environment. Considering vision device's limited adaptability in the dark environment, a 3D-laser-based scene recognition approach that extracts and matches SIFT features from Bearing Angle images is proposed, which makes it possible to make full use of both global metric information and local scale-invariant features. This approach can not only cope with irregular disturbances of dynamic objects, but also tackle obvious changes of observation location robustly in a semi-structured environment. An large-scale indoor environment with more than 30 offices is selected as the real-world scenes to test the performance of the proposed approach.

This paper proposes the motion control system in a hybrid architecture for middle-size soccer robot. A hybrid architecture combining hierarchical architecture and subsumption architecture is composed of four levels, such as the central control unit, the sensor information processing unit, the motion control system and the execution level. A local path planning algorithm fitting for autonomous soccer robot in dynamic environment is applied, and several actions and behaviors are properly designed for soccer robot motion control. The effectiveness of the proposed motion control system is validated by simulating and implementing it in a real soccer robot platform SmartROB2.