Pre-training techniques play a crucial role in deep learning, enhancing models' performance across a variety of tasks. By initially training on large datasets and subsequently fine-tuning on task-specific data, pre-training provides a solid foundation for models, improving generalization abilities and accelerating convergence rates. This approach has seen significant success in the fields of natural language processing and computer vision. However, traditional pre-training methods necessitate large datasets and substantial computational resources, and they can only learn shared features through prolonged training and struggle to capture deeper, task-specific features. In this paper, we propose a task-oriented pre-training method that begins with generating redundant segmentation proposals using the Segment Anything (SAM) model. We then introduce a Specific Category Enhancement Fine-tuning (SCEF) strategy for fine-tuning the Contrastive Language-Image Pre-training (CLIP) model to select proposals most closely related to the drivable area from those generated by SAM. This approach can generate a lot of coarse training data for pre-training models, which are further fine-tuned using manually annotated data, thereby improving model's performance. Comprehensive experiments conducted on the KITTI road dataset demonstrate that our task-oriented pre-training method achieves an all-around performance improvement compared to models without pre-training (as shown in Fig. 1). Moreover, our pre-training method not only surpasses traditional pre-training approach but also achieves the best performance compared to state-of-the-art self-training methods. The open-source project can be found at https://sites.google.com/view/task-oriented-pre-training.

Crack segmentation is pivotal for structural health monitoring, enabling the timely maintenance of critical infrastructure such as bridges and roads. However, existing deep learning models are often too computationally intensive for deployment on resource-constrained devices. To address this limitation, we introduce UltraFastCrackSeg, a lightweight model designed for real-time crack segmentation that effectively balances high accuracy with low computational demands. Featuring an efficient encoder-decoder architecture, our model significantly reduces parameter count and floating-point operations (FLOPs) compared to current methods, as illustrated in Figure 1. We further enhance performance through a self-supervised pretraining approach that employs a novel, task-oriented masking strategy, thereby improving feature extraction. Experiments across multiple datasets demonstrate that UltraFastCrackSeg achieves state-of-the-art Intersection over Union (IoU) and F1 scores while maintaining a compact model size and high inference speed. Evaluations on a low-power CPU device confirm its capability to achieve up to 80 frames per second (FPS) with ONNX runtime optimization, making it highly suitable for real-time, on-site applications. These findings establish UltraFastCrackSeg as a robust and efficient solution for practical crack detection tasks. Code is available at: https://github.com/weiqingq/UltraFastCrackSeg.

Accurate depth estimation is crucial for 3D scene comprehension in robotics and autonomous vehicles. Fisheye cameras, known for their wide field of view, have inherent geometric benefits. However, their use in depth estimation is restricted by a scarcity of ground truth data and image distortions. We present FisheyeDepth, a self-supervised depth estimation model tailored for fisheye cameras. We incorporate a fisheye camera model into the projection and reprojection stages during training to handle image distortions, thereby improving depth estimation accuracy and training stability. Furthermore, we incorporate real-scale pose information into the geometric projection between consecutive frames, replacing the poses estimated by the conventional pose network. Essentially, this method offers the necessary physical depth for robotic tasks, and also streamlines the training and inference procedures. Additionally, we devise a multi-channel output strategy to improve robustness by adaptively fusing features at various scales, which reduces the noise from real pose data. We demonstrate the superior performance and robustness of our model in fisheye image depth estimation through evaluations on public datasets and real-world scenarios. The project website is available at: https://github.com/guoyangzhaolFisheyeDepth.

Traffic sign is a critical map feature for navigation and traffic control. Nevertheless, current methods for traffic sign recognition rely on traditional deep learning models, which typically suffer from significant performance degradation considering the variations in data distribution across different regions. In this paper, we propose TSCLIP, a robust fine-tuning approach with the contrastive language-image pre-training (CLIP) model for worldwide cross-regional traffic sign recognition. We first curate a cross-regional traffic sign benchmark dataset by combining data from ten different sources. Then, we propose a prompt engineering scheme tailored to the characteristics of traffic signs, which involves specific scene descriptions and corresponding rules to generate targeted text descriptions. During the TSCLIP fine-tuning process, we implement adaptive dynamic weight ensembling (ADWE) to seamlessly incorporate outcomes from each training iteration with the zero-shot CLIP model. This approach ensures that the model retains its ability to generalize while acquiring new knowledge about traffic signs. To the best knowledge of authors, TSCLIP is the first contrastive language-image model used for the worldwide cross-regional traffic sign recognition task. The project website is available at: https://github.com/guoyangzhao/TSCLIP.

LiDAR-based localization serves as a critical component in autonomous systems, yet existing approaches face persistent challenges in balancing repeatability, accuracy, and environmental adaptability. Traditional point cloud registration methods relying solely on offline maps often exhibit limited robustness against long-term environmental changes, leading to localization drift and reliability degradation in dynamic real-world scenarios. To address these challenges, this paper proposes DuLoc, a robust and accurate localization method that tightly couples LiDAR-inertial odometry with offline map-based localization, incorporating a constant-velocity motion model to mitigate outlier noise in real-world scenarios. Specifically, we develop a LiDAR-based localization framework that seamlessly integrates a prior global map with dynamic real-time local maps, enabling robust localization in unbounded and changing environments. Extensive real-world experiments in ultra unbounded port that involve 2,856 hours of operational data across 32 Intelligent Guided Vehicles (IGVs) are conducted and reported in this study. The results attained demonstrate that our system outperforms other state-of-the-art LiDAR localization systems in large-scale changing outdoor environments.

Road curbs are considered as one of the crucial and ubiquitous traffic features, which are essential for ensuring the safety of autonomous vehicles. Current methods for detecting curbs primarily rely on camera imagery or LiDAR point clouds. Image-based methods are vulnerable to fluctuations in lighting conditions and exhibit poor robustness, while methods based on point clouds circumvent the issues associated with lighting variations. However, it is the typical case that significant processing delays are encountered due to the voluminous amount of 3D points contained in each frame of the point cloud data. Furthermore, the inherently unstructured characteristics of point clouds poses challenges for integrating the latest deep learning advancements into point cloud data applications. To address these issues, this work proposes an annotation-free curb detection method leveraging Altitude Difference Image (ADI) (as shown in Fig. 1), which effectively mitigates the aforementioned challenges. Given that methods based on deep learning generally demand extensive, manually annotated datasets, which are both expensive and labor-intensive to create, we present an Automatic Curb Annotator (ACA) module. This module utilizes a deterministic curb detection algorithm to automatically generate a vast quantity of training data. Consequently, it facilitates the training of the curb detection model without necessitating any manual annotation of data. Finally, by incorporating a post-processing module, we manage to achieve state-of-the-art results on the KITTI 3D curb dataset [1] with considerably reduced processing delays compared to existing methods, which underscores the effectiveness of our approach in curb detection tasks. Our code and data will be open-sourced at: https://sites.google.com/view/adi-curb-detection.

Accurate prediction of pedestrian trajectories is crucial for improving the safety of autonomous driving. However, this task is generally nontrivial due to the inherent stochasticity of human motion, which naturally requires the predictor to generate multi-modal prediction. Previous works leverage various generative methods, such as GAN and VAE, for pedestrian trajectory prediction. Nevertheless, these methods may suffer from mode collapse and relatively low-quality results. The denoising diffusion probabilistic model (DDPM) has recently been applied to trajectory prediction due to its simple training process and powerful reconstruction ability. However, current diffusion-based methods do not fully utilize input information and usually require many denoising iterations that lead to a long inference time or an additional network for initialization. To address these challenges and facilitate the use of diffusion models in multi-modal trajectory prediction, we propose GDTS, a novel Goal-Guided Diffusion Model with Tree Sampling for multi-modal trajectory prediction. Considering the "goal-driven" characteristics of human motion, GDTS leverages goal estimation to guide the generation of the diffusion network. A two-stage tree sampling algorithm is presented, which leverages common features to reduce the inference time and improve accuracy for multi-modal prediction. Experimental results demonstrate that our proposed framework achieves comparable state-of-the-art performance with real-time inference speed in public datasets.

The creation of a metric-semantic map, which encodes human-prior knowledge, represents a high-level abstraction of environments. However, constructing such a map poses challenges related to the fusion of multi-modal sensor data, the attainment of real-time mapping performance, and the preservation of structural and semantic information consistency. In this paper, we introduce an online metric-semantic mapping system that utilizes LiDAR-Visual-Inertial sensing to generate a global metric-semantic mesh map of large-scale outdoor environments. Leveraging GPU acceleration, our mapping process achieves exceptional speed, with frame processing taking less than$7ms$, regardless of scenario scale. Furthermore, we seamlessly integrate the resultant map into a real-world navigation system, enabling metric-semantic-based terrain assessment and autonomous point-to-point navigation within a campus environment. Through extensive experiments conducted on both publicly available and self-collected datasets comprising 24 sequences, we demonstrate the effectiveness of our mapping and navigation methodologies. Note to Practitioners—This paper tackles the challenge of autonomous navigation for mobile robots in complex, unstructured environments with rich semantic elements. Traditional navigation relies on geometric analysis and manual annotations, struggling to differentiate similar structures like roads and sidewalks. We propose an online mapping system that creates a global metric-semantic mesh map for large-scale outdoor environments, utilizing GPU acceleration for speed and overcoming the limitations of existing real-time semantic mapping methods, which are generally confined to indoor settings. Our map integrates into a real-world navigation system, proven effective in localization and terrain assessment through experiments with both public and proprietary datasets. Future work will focus on integrating kernel-based methods to improve the map’s semantic accuracy.

Localized neural implicit representation methods have recently been proven effective for shape reconstruction. However, while some recent neural implicit representation-based approaches have investigated part awareness, there is still room for improvement in leveraging the rich geometry information contained in parts, which is crucial for accurate reconstruction. This study aims to enhance the accuracy of shape reconstruction by incorporatingpart awareness. This principle faces a fundamental technical challenge: manually defining parts across various categories is ambiguous and expensive. To address it, we propose a new self-supervised learning paradigm that automatically discovers meaningful parts. Our proposed paradigm has several prominent advantages as compared with the prior arts: (1) It allows masked part modeling thatscaleswell with available data; (2) It is aflexibleformulation that allows a variable number of parts; (3) It allows the fusion ofmulti-scale(global-level and part-level) features at an arbitrarily given coordinate; (4) Thesemantic consistencyof learned parts leads to transferable features. Extensive experiments validate our approach, named Masked PaCONet, showcasing its superiority in qualitative and quantitative results on public benchmarks, even under challenging settings. Codes and models will be released.

Current autonomous driving systems often struggle to balance decision-making and motion control while ensuring safety and traffic rule compliance, especially in complex urban environments. Existing methods may fall short due to separate handling of these functionalities, leading to inefficiencies and safety compromises. To address these challenges, we introduce UDMC, an interpretable and unified Level 4 autonomous driving framework. UDMC integrates decision-making and motion control into a single optimal control problem (OCP), considering the dynamic interactions with surrounding vehicles, pedestrians, road lanes, and traffic signals. By employing innovative potential functions to model traffic participants and regulations, and incorporating a specialized motion prediction module, our framework enhances on-road safety and rule adherence. The integrated design allows for real-time execution of flexible maneuvers suited to diverse driving scenarios. High-fidelity simulations conducted in CARLA exemplify the framework’s computational efficiency, robustness, and safety, resulting in superior driving performance when compared against various baseline models. Our open-source project is available athttps://github.com/henryhcliu/udmc_carla.git.

3D lane detection is essential in autonomous driving (AD) as it extracts structural and traffic information from the road in 3D space, aiding autonomous vehicles in logical, safe, and comfortable path planning and motion control. Given the cost of sensors and the advantages of visual data in color information, 3D lane detection based on monocular vision is an important research direction in the realm of AD that increasingly gains attention in both industry and academia. Nevertheless, recent advancements in visual perception seem inadequate for the development of fully reliable 3D lane detection algorithms, which also hampers the progress of vision-based fully autonomous vehicles. We believe that it still leaves an open and interesting problem for improvement in 3D lane detection algorithms for autonomous vehicles using visual sensors, and significant enhancements are essentially required. This review summarizes and analyzes the current state of achievements in the field of 3D lane detection research. It covers all current monocular-based 3D lane detection processes, discusses the performance of these cutting-edge algorithms, analyzes the time complexity of various algorithms, and highlights the main achievements and limitations of ongoing research efforts. The survey also includes a comprehensive discussion of available 3D lane detection datasets and the challenges that researchers encounter but have not yet resolved. Finally, our work outlines future research directions and invites researchers and practitioners to join this exciting field.

Curb detection is a crucial function in intelligent driving, essential for determining drivable areas on the road. However, the complexity of road environments makes curb detection challenging. This paper introduces CurbNet, a novel framework for curb detection utilizing point cloud segmentation. To address the lack of comprehensive curb datasets with 3D annotations, we have developed the 3D-Curb dataset based on SemanticKITTI, currently the largest and most diverse collection of curb point clouds. Recognizing that the primary characteristic of curbs is height variation, our approach leverages spatially rich 3D point clouds for training. To tackle the challenges posed by the uneven distribution of curb features on the xy-plane and their dependence on high-frequency features along the z-axis, we introduce the Multi-Scale and Channel Attention (MSCA) module, a customized solution designed to optimize detection performance. Additionally, we propose an adaptive weighted loss function group specifically formulated to counteract the imbalance in the distribution of curb point clouds relative to other categories. Extensive experiments conducted on 2 major datasets demonstrate that our method surpasses existing benchmarks set by leading curb detection and point cloud segmentation models. Through the post-processing refinement of the detection results, we have significantly reduced noise in curb detection, thereby improving precision by 4.5 points. Similarly, our tolerance experiments also achieve state-of-the-art results. Furthermore, real-world experiments and dataset analyses mutually validate each other, reinforcing CurbNet’s superior detection capability and robust generalizability. The project website is available at:https://github.com/guoyangzhao/CurbNet/.

LiDAR-based localization is valuable for applications like mining surveys and underground facility maintenance. However, existing methods can struggle when dealing with uninformative geometric structures in challenging scenarios. This paper presents RELEAD, a LiDAR-centric solution designed to address scan-matching degradation. Our method enables degeneracy-free point cloud registration by solving constrained ESIKF updates in the front end and incorporates multisensor constraints, even when dealing with outlier measurements, through graph optimization based on Graduated Non-Convexity (GNC). Additionally, we propose a robust Incremental Fixed Lag Smoother (rIFL) for efficient GNC-based optimization. RELEAD has undergone extensive evaluation in degenerate scenarios and has outperformed existing state-of-the-art LiDAR-Inertial odometry and LiDAR-Visual-Inertial odometry methods.

In recent years, imitation-based driving planners have reported considerable success. However, due to the absence of a standardized benchmark, the effectiveness of various designs remains unclear. The newly released nuPlan addresses this issue by offering a large-scale real-world dataset and a standardized closed-loop benchmark for equitable comparisons. Utilizing this platform, we conduct a comprehensive study on two fundamental yet underexplored aspects of imitation-based planners: the essential features for ego planning and the effective data augmentation techniques to reduce compounding errors. Furthermore, we highlight an imitation gap that has been overlooked by current learning systems. Finally, integrating our findings, we propose a strong baseline model—PlanTF. Our results demonstrate that a well-designed, purely imitation-based planner can achieve highly competitive performance compared to state-of-the-art methods involving hand-crafted rules and exhibit superior generalization capabilities in long-tail cases. Our models and benchmarks are publicly available. Project website https://jchengai.github.io/planTF.

Unmanned vehicles usually rely on Global Positioning System (GPS) and Light Detection and Ranging (LiDAR) sensors to achieve high-precision localization results for navigation purpose. However, this combination with their associated costs and infrastructure demands, poses challenges for widespread adoption in mass-market applications. In this paper, we aim to use only a monocular camera to achieve comparable onboard localization performance by tracking deep-learning visual features on a LiDAR-enhanced visual prior map. Experiments show that the proposed algorithm can provide centimeter-level global positioning results with scale, which is effortlessly integrated and favorable for low-cost robot system deployment in real-world applications.

A Colored point cloud, as a simple and efficient 3D representation, has many advantages in various fields, including robotic navigation and scene reconstruction. This representation is now commonly used in 3D reconstruction tasks relying on cameras and LiDARs. However, fusing data from these two types of sensors is poorly performed in many existing frameworks, leading to unsatisfactory mapping results, mainly due to inaccurate camera poses. This paper presents Omni-Color, a novel and efficient algorithm to colorize point clouds using an independent 360-degree camera. Given a LiDAR-based point cloud and a sequence of panorama images with initial coarse camera poses, our objective is to jointly optimize the poses of all frames for mapping images onto geometric reconstructions. Our pipeline works in an off-the-shelf manner that does not require any feature extraction or matching process. Instead, we find optimal poses by directly maximizing the photometric consistency of LiDAR maps. In experiments, we show that our method can overcome the severe visual distortion of omnidirectional images and greatly benefit from the wide field of view (FOV) of 360-degree cameras to reconstruct various scenarios with accuracy and stability. The code will be released at https://github.com/liubonan123/OmniColor/.

Accurate trajectory prediction is crucial for safe and efficient autonomous driving, but handling partial observations presents significant challenges. To address this, we propose a novel trajectory prediction framework called Partial Observations Prediction (POP) for congested urban road scenarios. The framework consists of two key stages: self-supervised learning (SSL) and feature distillation. POP first employs SLL to help the model learn to reconstruct history representations, and then utilizes feature distillation as the fine-tuning task to transfer knowledge from the teacher model, which has been pre-trained with complete observations, to the student model, which has only few observations. POP achieves comparable results to topperforming methods in open-loop experiments and outperforms the baseline method in closed-loop simulations, including safety metrics. Qualitative results illustrate the superiority of POP in providing reasonable and safe trajectory predictions. Demo videos and code are available at https://chantsss.github.io/POP/.

Visual odometry (VO) system is challenged by complex illumination environments. Image quality and its consistency in the time domain directly determine feature detection and tracking performance, which further affect the robustness and accuracy of the entire system. In this paper, an image acquisition scheme with image bracketing patterns is proposed. Images with different exposure levels are continuously captured to sufficiently explore the scene under varying illumination. An attribute control method is designed to adjust image exposures within the brackets online. Gaussian process regression fits the relationship between image quality metric and exposure via image synthesis technique. The optimal exposures for the next bracket are obtained directly without attempts to ensure a quick response. Experiments show our acquisition system’s effectiveness and performance improvement for VO tasks in complex illumination scenes.

Collaborative state estimation using different heterogeneous sensors is a fundamental prerequisite for robotic swarms operating in GPS-denied environments, posing a significant research challenge. In this paper, we introduce a centralized system to facilitate collaborative LiDAR-ranging-inertial state estimation, enabling robotic swarms to operate without the need for anchor deployment. The system efficiently distributes computationally intensive tasks to a central server, thereby reducing the computational burden on individual robots for local odometry calculations. The server back-end establishes a global reference by leveraging shared data and refining joint pose graph optimization through place recognition, global optimization techniques, and removal of outlier data to ensure precise and robust collaborative state estimation. Extensive evaluations of our system, utilizing both publicly available datasets and our custom datasets, demonstrate significant enhancements in the accuracy of collaborative SLAM estimates. Moreover, our system exhibits remarkable proficiency in large-scale missions, seamlessly enabling ten robots to collaborate effectively in performing SLAM tasks. In order to contribute to the research community, we will make our code open-source and accessible at https://github.com/PengYu-team/Co-LRIO.

The inductively coupled plasma (ICP) source is widely used in the semiconductor manufacturing industry. The main challenge is delivering a high frequency and low harmonic current with multi-level power modulation. The multi-level power mode is required for advanced manufacturing processes and it modulates the output power repeatedly, the energy storage components charge and discharge rapidly, leading to very large transient power losses on switches, especially for the lagging bridge which loses ZVS operation. This paper discussed the matching network design for the ICP source and the steady-state analysis for its phase-shift modulation. Furthermore, the transient power loss mechanism during the multi-level power modulation is elaborated by using multiple harmonic approximation. Finally, the design guidance for ICP source working at multi-level power mode is discussed to achieve high efficiency and high reliability. A 2kW prototyping system is built to validate the proposed analysis and design for the ICP source.

Maps provide robots with crucial environmental knowledge, thereby enabling them to perform interactive tasks effectively. Easily accessing accurate abstract-to-detailed geometric and semantic concepts from maps is crucial for robots to make informed and efficient decisions. To comprehensively model the environment and effectively manage the map data structure, we propose DHP-Mapping, a dense mapping system that utilizes multiple Truncated Signed Distance Field (TSDF) submaps and panoptic labels to hierarchically model the environment. The output map is able to maintain both voxel- and submap-level metric and semantic information. Two modules are presented to enhance the mapping efficiency and label consistency: (1) an inter-submaps label fusion strategy to eliminate duplicate points across submaps and (2) a conditional random field (CRF) based approach to enhance panoptic labels. We conducted experiments with two public datasets including indoor and outdoor scenarios. Our system performs comparably to state-of-the-art (SOTA) methods across geometry and label accuracy evaluation metrics. The experiment results highlight the effectiveness and scalability of our system, as it is capable of constructing precise geometry and maintaining consistent panoptic labels. Our code is publicly available at https://github.com/hutslib/DHP-Mapping.

Monocular 3D object detection is essential for autonomous driving. However, current monocular 3D detection algorithms rely on expensive 3D labels from LiDAR scans, making it difficult to use in new datasets and unfamiliar environments. This study explores training a monocular 3D object detection model using a mix of 3D and 2D datasets. The proposed framework includes a robust monocular 3D model that can adapt to different camera settings, a selective-training strategy to handle varying class annotations in datasets, and a pseudo 3D training method using 2D labels to improve detection ability in scenes with only 2D labels (as shown in Fig. 1). By utilizing this framework, we can train models on a combination of 3D and 2D datasets to improve generalization and performance on new datasets with only 2D labels. Extensive experiments on KITTI, nuScenes, ONCE, Cityscapes, and BDD100K datasets showcase the scalability of our proposed approach. Here is our project page: https://sites.google.com/view/fmaafmono3d.

This paper presents a generic trajectory planning method for wheeled robots with fixed steering axes while the steering angle of each wheel is constrained. In the existing literatures, All-Wheel-Steering (AWS) robots, incorporating modes such as rotation-free translation maneuvers, in-situ rotational maneuvers, and proportional steering, exhibit inefficient performance due to time-consuming mode switches. This inefficiency arises from wheel rotation constraints and inter-wheel cooperation requirements. The direct application of a holonomic moving strategy can lead to significant slip angles or even structural failure. Additionally, the limited steering range of AWS wheeled robots exacerbates non-linearity characteristics, thereby complicating control processes. To address these challenges, we developed a novel planning method termed Constrained AWS (C-AWS), which integrates second-order discrete search with predictive control techniques. Experimental results demonstrate that our method adeptly generates feasible and smooth trajectories for C-AWS while adhering to steering angle constraints. Code and video can be found at https://github.com/Rex-sys-hk/AWSPlanning.

High-definition angular marking maps (HDAM maps) are vital in large-scale environments with variable appearances. In these scenarios, unmanned ground vehicles (UGVs) can use angular markings for localization because they are easy to identify and informative for localization. However, creating such a marking map relies heavily on manual measurement and annotation, which is time-consuming and laborious. Although Inverse Perspective Mapping (IPM) offers a low-cost and automated alternative, its accuracy is compromised by vehicle motion and the arduous pre-calibration of the IPM matrix. To fill these gaps, we propose a pose-guided optimization framework for IPM. This framework enables the automated generation of HDAM maps, while concurrently refining the preliminary IPM matrix. We deployed the proposed method in two different automated ports, and the method yielded HDAM maps with near-centimeter precision. Moreover, the refined IPM matrix matched the accuracy of manual calibrations. The supplementary materials and videos are available at http://liuhongji.site/PGO-IPM/.

Deep neural networks (DNNs) are increasingly utilized in robotic tasks. However, resource-constrained mobile robots often do not have sufficient onboard computing resources or power reserves to run the most accurate and state-of-the-art DNNs. Cloud robotics has the benefit of enabling robots to offload DNNs to cloud servers, which is considered a promising technology to address the issue. However, comprehensive issues exist, including flexibility, convenience, offloading policy, and especially network robustness in its implementations and deployments. Although it is essential to promote cloud robotics to be practical, a cloud robotic system that addresses these issues comprehensively has never been proposed. Accordingly, in this work, we present RoboEC2, a novel cloud robotic system with dynamic network offloading implemented assisted by Amazon EC2. To realize the goal, we present a cloud-edge cooperation framework based on ROS and Amazon Web Services (AWS) and a network offloading approach with a dynamic splitting way. RoboEC2 is capable of executing its network offloading program in any conditions, including disconnected. We model the DNN offloading problem in RoboEC2 to a specific multi-objective optimization problem and address it by proposing the Spotlight Criteria Algorithm (SCA). RoboEC2 is flexible, convenient, and robust. It is the first cloud robotic system with no constraints on time, location, or computing power. Finally, We demonstrate RoboEC2 with analyses and experiments that it performs better in comprehensive metrics compared with the state-of-the-art approach. We open-source the system at https://github.com/RoboEC2/RoboEC2.Note to Practitioners—RoboEC2 is a work that combines cloud computing and robotics. As the deep learning models are becoming larger, robots are becoming more and more difficult to run the state-of-the-art models locally. It has become one of the major problems in robotics. RoboEC2 was proposed to address this problem. It enables more robotics researchers to equip their robots with the power of cloud computing. To be honest, it is very difficult for us to complete this work that is a robotic system with cloud computing. We need to address a lot of difficulties such as network, the cloud platform, algorithms, robot platforms, and conduct various robotic tasks. We have spent more than one year on this system and overcome countless difficulties to complete it. All of what we do is to make robotics developer easier strengthen their robots with cloud. Whether you are an autonomous driving engineer, robotic arm developer, SLAM researcher, mobile robotics researcher, or any other developer working on robotics applications based on ROS and deep learning models, you can use RoboEC2 to make them perform better. You don’t need to worry about networking, because RoboEC2 has solved it perfectly. You don’t need to worry about the serious algorithms in the system, because we provide easily used interact files for you to configure. You just need to tell RoboEC2 which metrics your robotics application needs to focus on. With RoboEC2, all the robotic researchers/developers are capable of enhancing their robotic applications with cloud computing in just a few simple steps and executing them in any network conditions. So, why not?

Predicting accurate depth with monocular images is important for low-cost robotic applications and autonomous driving. This study proposes a comprehensive self-supervised framework for accurate scale-aware depth prediction on autonomous driving scenes utilizing inter-frame poses obtained from inertial measurements. In particular, we introduce a Full-Scale depth prediction network named FSNet. FSNet contains four important improvements over existing self-supervised models: (1) a multichannel output representation for stable training of depth prediction in driving scenarios, (2) an optical-flow-based mask designed for dynamic object removal, (3) a self-distillation training strategy to augment the training process, and (4) an optimization-based post-processing algorithm in test time, fusing the results from visual odometry. With this framework, robots and vehicles with only one well-calibrated camera can collect sequences of training image frames and camera poses, and infer accurate 3D depths of the environment without extra labeling work or 3D data. Extensive experiments on the KITTI dataset, KITTI-360 dataset and the nuScenes dataset demonstrate the potential of FSNet. More visualizations are presented in https://sites.google.com/view/fsnet/homeNote to Practitioners—This paper was motivated by the problem of unsupervised monocular depth for robotic deployment. We notice that PoseNet is not generalizable and by nature monodepth2 only predict depths up to a scale. We believe that we should not expect PoseNet, a ResNet on a concatenation of two images, to produce more reliable poses than the localization module in a robot. So we try our best to completely avoid using PoseNet. This creates much unstability in training, but we managed to fix it in FSNet with multichannel output and self-distillation. We also believe the network should try to directly predict accurate depth with a correct scale at any cases. So our method could produce meaningful results on static frames or scenes with little/no VO points (same as the network’s direct prediction). There are images without VO points in our multi-frame experiment, but our method is robust enough to fix this problem. In future research, we will include multi-frame depth predictions for more accurate depth prediction.

Multiple-object tracking (MOT) is a crucial component in autonomous driving systems. However, inaccurate object detection is always the bottleneck for MOT. Most detectors are not designed to take the temporal information across consecutive frames into consideration. To take advantage of such information, we design a novel data representation, the spatio-temporal (ST) map, which collects a batch of detection results spatio-temporally, and we train a novel network, ST-TrackNet, to assign predicted track IDs to each positive detection across a sequence. With our ST map detection fed into the tracker, the correlation of objects between adjacent frames becomes prominent, which improves the performance of the tracker in the data association step. Moreover, the long-term trajectory in a sequence also helps to refine the detection results. We train and evaluate our network on the KITTI dataset, a CARLA simulation dataset, and a dataset recorded in a factory environment. Our approach generally achieves superior performance over the state-of-the-art. Note to Practitioners—We investigate the MOT problem in this paper. A spatio-temporal pipeline is proposed to provide a solution to this problem. Object detection results produced by off-the-shelf object detectors are used to form the proposed ST maps. In low signal-to-noise ratio (SNR) situations, our proposed framework can achieve more accurate and robust tracking results with more false-positives. Due to the simplicity and modular design of our framework, it can be applied directly after the detection stage to achieve the online tracking task. The proposed method is evaluated on several datasets, and the experimental results demonstrate its effectiveness. Our method can also be used for other autonomous driving applications, such as path planning and trajectory prediction.

Autonomous driving emphasizes precise multi-sensor fusion positioning on limit resource embedded systems. LiDAR-centered sensor fusion system serves as a mainstream navigation system due to its insensitivity to illumination and viewpoint change. However, these types of systems suffer from handling large-scale sequential LiDAR data using limited resources on board, leading LiDAR-centralized sensor fusion unpractical. As a result, hand-crafted features such as plane and edge are leveraged in majority mainstream positioning methods to alleviate this unsatisfaction, triggering a new cornerstone in LiDAR Inertial sensor fusion. However, such super light weight feature extraction, although it achieves real-time constraint in LiDAR-centered sensor fusion, encounters severe vulnerability under high speed rotational or translational perturbation. In this paper, we propose a sparse tensor based LiDAR Inertial fusion method for autonomous driving embedded system. Leveraging the power of sparse tensor, the global geometrical feature is fetched so that the point cloud sparsity defect is alleviated. Inertial sensor is deployed to conquer the time-consuming step caused by the coarse level point-wise inlier matching. We construct our experiments on both representative dataset benchmarks and realistic scenes. The evaluation results show the robustness and accuracy of our proposed solution compared to classical methods.

Considerable research efforts have been devoted to the development of motion planning algorithms, which form a cornerstone of the autonomous driving system (ADS). Nonetheless, acquiring an interactive and secure trajectory for the ADS remains challenging due to the complex nature of interaction modeling in planning. Modern planning methods still employ a uniform treatment of prediction outcomes and solely rely on collision-avoidance strategies, leading to suboptimal planning performance. To address this limitation, this paper presents a novel prediction-based interactive planning framework for autonomous driving. Our method incorporates interaction reasoning into spatio-temporal (s-t) planning by defining interaction conditions and constraints. Specifically, it records and continually updates interaction relations for each planned state throughout the forward search. We assess the performance of our approach alongside state-of-the-art methods in the CommonRoad environment. Our experiments include a total of 232 scenarios, with variations in the accuracy of prediction outcomes, modality, and degrees of planner aggressiveness. The experimental findings demonstrate the effectiveness and robustness of our method. It leads to a reduction of collision times by approximately 17.6% in 3-modal scenarios, along with improvements of nearly 7.6% in distance completeness and 31.7% in the fail rate in single-modal scenarios. For the community’s reference, our code is accessible at https://github.com/ChenYingbing/IR-STP-Planner.

This study explores the application of self-supervised learning (SSL) to the task of motion forecasting, an area that has not yet been extensively investigated despite the widespread success of SSL in computer vision and natural language processing. To address this gap, we introduce Forecast-MAE, an extension of the mask autoencoders framework that is specifically designed for self-supervised learning of the motion forecasting task. Our approach includes a novel masking strategy that leverages the strong interconnections between agents’ trajectories and road networks, involving complementary masking of agents’ future or history trajectories and random masking of lane segments. Our experiments on the challenging Argoverse 2 motion forecasting benchmark show that Forecast-MAE, which utilizes standard Transformer blocks with minimal inductive bias, achieves competitive performance compared to state-of-the-art methods that rely on supervised learning and sophisticated designs. Moreover, it outperforms the previous self-supervised learning method by a significant margin. Code is available at https://github.com/jchengai/forecast-mae.

Surface normal holds significant importance in visual environmental perception, serving as a source of rich geometric information. However, the state-of-the-art (SoTA) surface normal estimators (SNEs) generally suffer from an unsatisfactory trade-off between efficiency and accuracy. To resolve this dilemma, this paper first presents a superfast depth-to-normal translator (D2NT), which can directly translate depth images into surface normal maps without calculating 3D coordinates. We then propose a discontinuity-aware gradient (DAG) filter, which adaptively generates gradient convolution kernels to improve depth gradient estimation. Finally, we propose a surface normal refinement module that can easily be integrated into any depth-to-normal SNEs, substantially improving the surface normal estimation accuracy. Our proposed algorithm demonstrates the best accuracy among all other existing real-time SNEs and achieves the SoTA trade-off between efficiency and accuracy.

The rotation orthonormalization on the special orthogonal group$\text{SO}(n)$, also known as the high dimensional nearest rotation problem, has been revisited. A new generalized simple iterative formula has been proposed that solves this problem in a completely rational manner. Rational operations allow for efficient implementation on various platforms and also significantly simplify the synthesis of large-scale circuitization. The developed scheme is also capable of designing efficient fundamental rational algorithms, for example, quaternion normalization, which outperforms long-exisiting solvers. Furthermore, an$\text{SO}(n)$neural network has been developed for further learning purpose on the rotation group. Simulation results verify the effectiveness of the proposed scheme and show the superiority against existing representatives. Applications show that the proposed orthonormalizer is of potential in robotic pose estimation problems, e.g., hand-eye calibration.

With the fast development of autonomous driving technologies, there is an increasing demand for high-definition (HD) maps, which provide reliable and robust prior information about the static part of the traffic environments. As one of the important elements in HD maps, road lane centerline is critical for downstream tasks, such as prediction and planning. Manually annotating centerlines for road lanes in HD maps is labor-intensive, expensive and inefficient, severely restricting the wide applications of autonomous driving systems. Previous work seldom explores the lane centerline detection problem due to the complicated topology and severe overlapping issues of lane centerlines. In this paper, we propose a novel method named CenterLineDet to detect lane centerlines for automatic HD map generation. Our CenterLineDet is trained by imitation learning and can effectively detect the graph of centerlines with vehicle-mounted sensors (i.e., six cameras and one LiDAR) through iterations. Due to the use of the DETR-like transformer network, CenterLineDet can handle complicated graph topology, such as lane intersections. The proposed approach is evaluated on the large-scale public dataset NuScenes. The superiority of our CenterLineDet is demonstrated by the comparative results. Our code, supplementary materials, and video demonstrations are available at https://tonyxuqaq.github.io/projects/CenterLineDet/.

This paper proposes a novel real-time semantic segmentation network via frequency domain learning, called FDLNet, which revisits the segmentation task from two critical perspectives: spatial structure description and multilevel feature fusion. We first devise an image-size convolution (IS-Conv) as a global frequency-domain learning operator to capture long-range dependency in a single shot. To model spatial structure information, we construct the global structure representation path (GSRP) based on IS-Conv, which learns a unified edge-region representation with affordable complexity. For efficient and lightweight multi-level feature fusion, we propose the factorized stereoscopic attention (FSA) module, which alleviates semantic confusion and reduces feature redundancy by introducing level-wise attention before channel and spatial attention. Combining the above modules, we propose a concise semantic segmentation framework named FDLNet. We experimentally demonstrate the effectiveness and superiority of the proposed method. FDLNet achieves state-of-the-art performance on the Cityscapes, which reports 76.32% mIoU at 150+ FPS and 79.0% mIoU at 41+ FPS. The code is available at https://github.com/qyan0131/FDLNet.

Drivable area segmentation is an essential component of the visual perception system for autonomous driving vehicles. Recent efforts in deep neural networks have sig-nificantly improved semantic segmentation performance for autonomous driving. However, most DNN-based methods need a large amount of data to train the models, and collecting large-scale datasets with manually labeled ground truth is costly, tedious, time consuming and requires the availability of experts, making DNN-based methods often difficult to implement in real world applications. Hence, in this paper, we introduce a novel module named automatic data labeler (ADL), which leverages a deterministic LiDAR-based method for ground plane segmentation and road boundary detection to create large datasets suitable for training DNNs. Furthermore, since the data generated by our ADL module is not as accurate as the manually annotated data, we introduce uncertainty estimation to compensate for the gap between the human labeler and our ADL. Finally, we train the semantic segmentation neural networks using our automatically generated labels on the KITTI dataset [10] and KITTI-CARLA dataset [7]. The experimental results demonstrate that our proposed ADL method not only achieves impressive performance compared to manual labeling but also exhibits more robust and accurate results than both traditional methods and state-of-the-art self-supervised methods.

Object segmentation is a per-pixel label prediction task that targets at providing context analysis for autonomous driving. Moving-object segmentation (MOS) serves as a subbranch of object segmentation, targeting to separating the surrounding objects into binary options: dynamic and static. MOS is vital for the safety-critical task in autonomous driving because dynamic objects are often a true potential threat to self-driving cars compared to static ones. Current methods typically address the MOS problem as a category feature to label the mapping task, which is not rational in reality. For example, a parking car should be considered as static instead of a moving-object category. There is a little systematic theory to differentiate object moving characteristics from nonmoving characteristics in MOS. Furthermore, restricted by limited resources in the embedded system, MOS is often in an offline manner due to huge computational requirements. An online and low computational cost MOS is an urgent demand for the practical safety-critical mission which takes immediate reaction as compulsory. In this article, we propose EmPointMovSeg, an efficient and practical 3-D LiDAR MOS solution for autonomous driving. Leveraging the power of the well-adapted autoregressive system identification (AR-SI) theory, EmPointMovSeg theoretically explains the moving-object feature in large-scale 3-D LiDAR semantic segmentation. An end-to-end sparse tensor-based CNN which balances segmentation accuracy and online process ability is proposed. We construct our experiment on both representative dataset benchmarks and practical embedded systems. The evaluation result shows the effectiveness and accuracy of our proposed solution, conquering the bottleneck in the online large-scale 3-D LiDAR semantic segmentation.

The generalized rigid registration problem in high-dimensional Euclidean spaces is studied. The loss function is minimized with an equivalent error formulation by the Cayley formula. The closed-form linear least-square solution to such a problem is derived which generates the registration covariances, i.e., uncertainty information of rotation and translation, providing quite accurate probabilistic descriptions. Simulation results indicate the correctness of the proposed method and also present its efficiency on computation-time consumption, compared with previous algorithms using singular value decomposition (SVD) and linear matrix inequality (LMI). The proposed scheme is then applied to an interpolation problem on the special Euclidean group SE(n) with covariance-preserving functionality. Finally, experiments on covariance-aided Lidar mapping show practical superiority in robotic navigation.

Self-supervised learning methods for 3D skeleton-based action recognition via contrastive learning have obtained competitive achievements compared to classical supervised methods. Current researches show that adding a Multilayer Perceptron (MLP) to the top of the base encoder can extract high-level and global positive representations. Using a negative memory bank to store negative samples dynamically can balance the ample storage and feature consistency. However, these methods need to consider that the MLP lacks accurate encoding of fine-grained local features, and a memory bank needs rich and diverse negative sample pairs to match positive representations from different encoders. This paper proposes a new method called Cross Momentum Contrast (CrossMoCo), composed of three parts: ST-GCN encoder, ST-GCN encoder with MLP encoder (ST-MLP encoder), and two independent negative memory banks. The two encoders encode the input data into two positive feature pairs. Learning the cross representations of the two positive pairs is helpful for the model to extract both the global and the local information. Two independent negative memory banks update the negative samples according to different positive representations from two encoders, diversifying the negative samples' distribution and making negative representations close to the positive features. The increasing classification difficulty will improve the model's ability of contrastive learning. In addition, the spatiotemporal occlusion mask data augmentation method is used to enhance positive samples' information diversity. This method takes the adjacent skeleton joints that can form a skeleton bone as a mask unit, which can reduce the information redundancy after data augmentation since adjacent joints may carry similar spatiotemporal information. Experiments on the PKU-MMD Part II dataset, the NTU RGB+D 60 dataset, and the NW-UCLA dataset show that the CrossMoCo framework with spatiotemporal occlusion mask data augmentation has achieved a comparable performance.

Real-time kinodynamic trajectory planning in dy-namic environments is critical yet challenging for autonomous driving. In this paper, we propose an efficient trajectory plan-ning system for autonomous driving in complex dynamic sce-narios through iterative and incremental path-speed optimization. Exploiting the decoupled structure of the planning prob-lem, a path planner based on Gaussian process first generates a continuous arc-length parameterized path in the Frenét frame, considering static obstacle avoidance and curvature constraints. We theoretically prove that it is a good generalization of the well-known jerk optimal solution. An efficient s-t graph search method is introduced to find a speed profile along the generated path to deal with dynamic environments. Finally, the path and speed are optimized incrementally and iteratively to ensure kinodynamic feasibility. Various simulated scenarios with both static obstacles and dynamic agents verify the effectiveness and robustness of our proposed method. Experimental results show that our method can run at 20 Hz. The source code is released as an open-source package.

Real-time semantic segmentation is a crucial but challenging dense prediction task for scene parsing. However, the existing CNN-based methods commonly bias the model in favor of speed-boosting compromising spatial resolution due to business requirements and hardware constrains, which impedes the high-accuracy segmentation result. To address the dilemma, we provide a novel Holographic Segmentation Network (HoloSeg), which presents a strong ability of comprehensive information preservation and extraction, and achieves a better trade-off between speed and accuracy. We first design a Lossless Sample Pair (LSP) without any stride for early spatial preservation and later resolution recovery while modeling long-range context dependence. Then, we propose Distributed Pyramid Learning (DPL) to efficiently extract multiscale features and saves a lot of computation. Finally, we propose Resolution Fusion and Restoration (RFR) to fuse multi-level semantic representations across stages and generate output without decoder. Without bells and whistles, HoloSeg achieves state-of-the-art performance on the Cityscapes benchmark which reports 76.24% mIoU at 231 FPS. Code is available online: https://github.com/LiShuTJ/HoloSeg.

3D object detection with only image inputs is an interesting and important problem in computer vision and autonomous driving. Nowadays, most existing monocular 3D object detection algorithms rely solely on the approximation power of convolutional neural networks to learn a mapping from pixels to 3D predictions without knowing the projection matrix of the camera. To endow the networks with camera projection knowledge, we propose the Star-Convolution module for application to image-based 3D detection. The introduced module increases the receptive field of the detector and embeds the camera's projection geometry inside the network while keeping the network end-to-end trainable. We test the module with different baselines in both monocular and stereo 3D object detection, and we achieve significant improvements on both tasks. The code will be published at https://github.com/Owen-Liuyuxuan/visualDet3D.

Disparity and optical flow estimation are respectively 1D and 2D dense correspondence matching (DCM) tasks in nature. Unsupervised domain adaptation (UDA) is crucial for their success in new and unseen scenarios, enabling networks to draw inferences across different domains without manually-labeled ground truth. In this paper, we propose a general UDA framework (UnDAF) for disparity or optical flow estimation. Unlike existing approaches based on adversarial learning that suffers from pixel distortion and dense correspondence mismatch after domain alignment, our UnDAF adopts a straightforward but effective coarse-to-fine strategy, where a co-teaching strategy (two networks evolve by complementing each other) refines DCM estimations after Fourier transform initializes domain alignment. The simplicity of our approach makes it extremely easy to guide adaptation across different domains, or more practically, from synthetic to real-world domains. Extensive experiments carried out on the KITTI and MPI Sintel benchmarks demonstrate the accuracy and robustness of our UnDAF, advancing all other state-of-the-art UDA approaches for disparity or optical flow estimation. Our project page is available at https://sites.google.com/view/undaf.

Pedestrian trajectory prediction is of great importance for downstream tasks, such as autonomous driving and mobile robot navigation. Realistic models of the social interactions within the crowd is crucial for accurate pedestrian trajectory prediction. However, most existing methods do not capture group level interactions well, focusing only on pairwise interactions and neglecting group-wise interactions. In this work, we propose a hierarchical graph convolutional network, HGCN-GJS, for trajectory prediction which well leverages group level interactions within the crowd. Furthermore, we introduce a joint sampling scheme that captures co-dependencies between pedestrian trajectories during trajectory generation. Based on group information, this scheme ensures that generated trajectories within each group are consistent with each other, but enables different groups to act more independently. We demonstrate that our proposed network achieves state of the art performance on all datasets we have considered.

Our goal is to train a neural planner that can capture diverse driving behaviors in complex urban scenarios. We observe that even state-of-the-art neural planners are struggling to perform common maneuvers such as lane change, which is rather natural for human drivers. We propose to explore the multi-modalities in the planning problem and force the neural planner to explicitly consider different policies. This is achieved by generating the future trajectories conditioned on every possible reference line, which could simply be the centerline of the surrounding lanes. We find this simple strategy yet enables the planner to perform rich and complex behaviors. We train our model using real-world driving data and demonstrate the effectiveness of our method through both open-loop and closed-loop evaluations. Project website https://jchengai.github.io/mpnp.

Combining multiple sensors enables a robot to maximize its perceptual awareness of environments and enhance its robustness to external disturbance, crucial to robotic navigation. This paper proposes the FusionPortable benchmark, a complete multi-sensor dataset with a diverse set of sequences for mobile robots. This paper presents three contributions. We first advance a portable and versatile multi-sensor suite that offers rich sensory measurements: 10Hz LiDAR point clouds, 20Hz stereo frame images, high-rate and asynchronous events from stereo event cameras, 200Hz inertial readings from an IMU, and 10Hz GPS signal. Sensors are already temporally synchronized in hardware. This device is lightweight, self-contained, and has plug-and-play support for mobile robots. Second, we construct a dataset by collecting 17 sequences that cover a variety of environments on the campus by exploiting multiple robot platforms for data collection. Some sequences are challenging to existing SLAM algorithms. Third, we provide ground truth for the decouple localization and mapping performance evaluation. We additionally evaluate state-of-the-art SLAM approaches and identify their limitations. The dataset, consisting of raw sensor measurements, ground truth, calibration data, and evaluated algorithms, will be released.

Robot navigation in human crowds remains the challenge of understanding human behaviors in different scenarios. We present an approach for interactive and human-friendly crowd navigation in complex static environments. The planner models the online interactions among the robot, humans, and the static environment based on game theory. It recurrently expands and optimizes the estimated trajectories for the robot and neighboring agents and provides human-friendly navigation commands. We use various indicators to evaluate the social awareness of the planners and show that our method outperforms existing approaches in success rate to reach the goals and compatibility with humans while maintaining low navigation times. The planner is successfully deployed on a real-world quadrupedal robot, demonstrating safe and interactive crowd navigation with real-time performance.

This special issue of the IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING (T-ASE) focuses on how the state-of-the-art achievements and applications in the general area of artificial intelligence in automation for autonomous unmanned systems applications. As Guest Editors, we are very pleased to present the selected 16 articles, whose topics are specifically related to artificial intelligence real-time object detection, recognition, localization, control optimization, motion planning, formation control, adaptive control, and autonomous decision-making.

Indoor localization with high accuracy and efficiency has attracted much attention. Due to visible light communication (VLC), the LED lights in buildings, once modulated, hold great potential to be ubiquitous indoor localization infrastructure. However, this entails retrofitting the lighting system and is hence costly in wide adoption. To alleviate this problem, we propose to exploit modulated LEDs and existing unmodulated lights as landmarks. On this basis, we present a novel inertial-aided visible light positioning (VLP) system for lightweight indoor localization on resource-constrained platforms, such as service robots and mobile devices. With blob detection, tracking, and VLC decoding on rolling-shutter camera images, a visual front end extracts two types of blob features, i.e., mapped landmarks (MLs) and opportunistic features (OFs). These are tightly fused with inertial measurements in a stochastic cloning sliding-window extended Kalman filter (EKF) for localization. We evaluate the system by extensive experiments. The results show that it can provide lightweight, accurate, and robust global pose estimates in real time. Compared with our previous ML-only inertial-aided VLP solution, the proposed system has superior performance in terms of positional accuracy and robustness under challenging light configurations, such as sparse ML/OF distribution. Note to Practitioners—This article is motivated by the problem that many existing visible light positioning (VLP) systems require high-cost environmental modifications, i.e., replacing a large portion of original lights with modulated LEDs as beacons. To reduce costs in wide adoption, we seek to use fewer modulated LEDs if possible. Accordingly, we present a novel inertial-aided VLP system that uses both modulated LEDs and unmodulated lights as landmarks. Like in other VLP systems, the successfully decoded LEDs provide absolute pose measurements for global localization. Unmodulated lights and the LEDs with decoding failures provide relative motion constraints, allowing the reduction of pose drift during the outage of modulated LEDs. Due to the tightly coupled sensor fusion by filtering, the system can provide efficient and accurate localization when modulated LEDs are sparse. The system is lightweight to run on resource-constrained platforms. For practical deployment of our system at scale, creating LED maps accurately and efficiently remains a problem. It is desired to develop automated LED mapping solutions in future work.

This article investigates the multirobot cooperative navigation problem based on raw visual observations. A fully end-to-end learning framework is presented, which leverages graph neural networks to learn local motion coordination and utilizes deep reinforcement learning to generate visuomotor policy that enables each robot to move to its goal without the need of environment map and global positioning information. Experimental results show that, with a few tens of robots, our approach achieves comparable performance with the state-of-the-art imitation learning-based approaches with bird-view state inputs. We also illustrate our generalizability to crowded and large environments and our scalability to ten times number of the training robots. In addition, we demonstrate that our model trained for multirobot case can also improve the success rate in the single-robot navigation task in unseen environments. Note to Practitioners—With the development of intelligent industrial and logistic systems, robotic transportation systems are widely implemented. However, existing multirobot path coordination and navigation approaches are basically under some unreasonable assumptions, which are very hard to be implemented in practical scenarios. This article aims to greatly promote the real application of learning-based multirobot cooperative navigation approach, in order to achieve the following. First, we introduce an end-to-end reinforcement learning framework instead of the commonly used imitation learning strategy, as the latter one needs exhaustive training data to cover all the scenarios and does not have the required generalizability. Second, we directly use the raw sensor data instead of the commonly used bird-eye-view semantic observations, as the latter one is generally not representative of practical application scenario from the robot perspective and cannot solve the occlusion issue. Third, we interpret our learned model to illustrate which parts of the input and shared observations contribute most to the robots’ final actions. The above interpretability ensures predictability (thus safety) of our visuomotor policy in practical applications. Our learned visuomotor policy has the ability to coordinate dozens of robots by only using raw visual observations in unknown environments without map nor global localization information, this is the first time in the literature. Our future work includes solving the sim-to-real issue and conducting physical experiments.

Due to the huge volume of point cloud data, storing and transmitting it is currently difficult and expensive in autonomous driving. Learning from the high-efficiency video coding (HEVC) framework, we propose a novel compression scheme for large-scale point cloud sequences, in which several techniques have been developed to remove the spatial and temporal redundancy. The proposed strategy consists mainly of three parts: intracoding, intercoding, and residual data coding. For intracoding, inspired by the depth modeling modes (DMMs), in 3-D HEVC (3-D-HEVC), a cluster-based prediction method is proposed to remove the spatial redundancy. For intercoding, a point cloud registration algorithm is utilized to transform two adjacent point clouds into the same coordinate system. By calculating the residual map of their corresponding depth image, the temporal redundancy can be removed. Finally, the residual data are compressed either by lossless or lossy methods. Our approach can deal with multiple types of point cloud data, from simple to more complex. The lossless method can compress the point cloud data to 3.63% of its original size by intracoding and 2.99% by intercoding without distance distortion. Experiments on the KITTI dataset also demonstrate that our method yields better performance compared with recent well-known methods.Note to Practitioners—This article deals with the problem of efficient compression of point cloud sequences that come from light detection and ranging (LiDARs) mounted on autonomous mobile robots. The vast amount of point cloud data could be an important bottleneck for transmission and storage. Inspired by the HEVC algorithm, we develop a novel coding architecture for the point cloud sequence. The scans are divided into intraframe and interframe, which are encoded separately using different techniques. Our method can be used for the compression of LiDAR point cloud sequences or dense LiDAR point cloud map and will significantly reduce the transmission bandwidth and storage spaces. We have to admit that although our method is less effective for real-time solutions, it can be highly efficient for off-line applications. Future studies will concentrate on further optimizing the coding algorithm to reduce the computational complexity and trying to find a balance between them.

Potholes are one of the most common forms of road damage, which can severely affect driving comfort, road safety, and vehicle condition. Pothole detection is typically performed by either structural engineers or certified inspectors. However, this task is not only hazardous for the personnel but also extremely time consuming. This article presents an efficient pothole detection algorithm based on road disparity map estimation and segmentation. We first incorporate the stereo rig roll angle into shifting distance calculation to generalize perspective transformation. The road disparities are then efficiently estimated using semiglobal matching. A disparity map transformation algorithm is then performed to better distinguish the damaged road areas. Subsequently, we utilize simple linear iterative clustering to group the transformed disparities into a collection of superpixels. The potholes are finally detected by finding the superpixels, whose intensities are lower than an adaptively determined threshold. The proposed algorithm is implemented on an NVIDIA RTX 2080 Ti GPU in CUDA. The experimental results demonstrate that our proposed road pothole detection algorithm achieves state-of-the-art accuracy and efficiency.

In this article, we present a multiagent framework for real-time large-scale 3-D reconstruction applications. In SLAM, researchers usually build and update a 3-D map after applying nonlinear pose graph optimization techniques. Moreover, many multiagent systems are prevalently using odometry information from additional sensors. These methods generally involve extensive computer vision algorithms and are tightly coupled with various sensors. We develop a generic method for the key challenging scenarios in multiagent 3-D mapping based on different camera systems. The proposed framework performs actively in terms of localizing each agent after the first loop closure between them. It is shown that the proposed system only uses monocular cameras to yield real-time multiagent large-scale localization and 3-D global mapping. Based on the initial matching, our system can calculate the optimal scale difference between multiple 3-D maps and then estimate an accurate relative pose transformation for large-scale global mapping.

Joint detection of drivable areas and road anomalies is very important for mobile robots. Recently, many semantic segmentation approaches based on convolutional neural networks (CNNs) have been proposed for pixelwise drivable area and road anomaly detection. In addition, some benchmark datasets, such as KITTI and Cityscapes, have been widely used. However, the existing benchmarks are mostly designed for self-driving cars. There lacks a benchmark for ground mobile robots, such as robotic wheelchairs. Therefore, in this article, we first build a drivable area and road anomaly detection benchmark for ground mobile robots, evaluating existing state-of-the-art (SOTA) single-modal and data-fusion semantic segmentation CNNs using six modalities of visual features. Furthermore, we propose a novel module, referred to as the dynamic fusion module (DFM), which can be easily deployed in existing data-fusion networks to fuse different types of visual features effectively and efficiently. The experimental results show that the transformed disparity image is the most informative visual feature and the proposed DFM-RTFNet outperforms the SOTAs. In addition, our DFM-RTFNet achieves competitive performance on the KITTI road benchmark.

In robotic applications, many pose problems involve solving the homogeneous transformation based on the special Euclidean group SE(n) . However, due to the nonconvexity of SE(n) , many of these solvers treat rotation and translation separately, and the computational efficiency is still unsatisfactory. A new technique called the SE(n)++ is proposed in this article that exploits a novel mapping from SE(n) to SO(n + 1) . The mapping transforms the coupling between rotation and translation into a unified formulation on the Lie group and gives better analytical results and computational performances. Specifically, three major pose problems are considered in this article, that is, the point-cloud registration, the hand-eye calibration, and the SE(n) synchronization. Experimental validations have confirmed the effectiveness of the proposed SE(n)++ method in open datasets.

Road network graphs provide critical information for autonomous-vehicle applications, such as drivable areas that can be used for motion planning algorithms. To find road network graphs, manual annotation is usually inefficient and labor-intensive. Automatically detecting road network graphs could alleviate this issue, but existing works still have some limitations. For example, segmentation-based approaches could not ensure satisfactory topology correctness, and graph-based approaches could not present precise enough detection results. To provide a solution to these problems, we propose a novel approach based on transformer and imitation learning in this article. In view of that high-resolution aerial images could be easily accessed all over the world nowadays, we make use of aerial images in our approach. Taken as input an aerial image, our approach iteratively generates road network graphs vertex-by-vertex. Our approach can handle complicated intersection points with various numbers of incident road segments. We evaluate our approach on a publicly available dataset. The superiority of our approach is demonstrated through comparative experiments. Our work is accompanied by a demonstration video which is available athttps://tonyxuqaq.github.io/projects/RNGDet/.

Autonomous driving in multi-agent dynamic traffic scenarios is challenging: the behaviors of road users are uncertain and are hard to model explicitly, and the ego-vehicle should apply complicated negotiation skills with them, such as yielding, merging and taking turns, to achieve both safe and efficient driving in various settings. Traditional planning methods are largely rule-based and scale poorly in these complex dynamic scenarios, often leading to reactive or even overly conservative behaviors. Therefore, they require tedious human efforts to maintain workability. Recently, deep learning-based methods have shown promising results with better generalization capability but less hand engineering efforts. However, they are either implemented with supervised imitation learning (IL), which suffers from dataset bias and distribution mismatch issues, or are trained with deep reinforcement learning (DRL) but focus on one specific traffic scenario. In this work, we propose DQ-GAT to achieve scalable and proactive autonomous driving, where graph attention-based networks are used to implicitly model interactions, and deep Q-learning is employed to train the network end-to-end in an unsupervised manner. Extensive experiments in a high-fidelity driving simulator show that our method achieves higher success rates than previous learning-based methods and a traditional rule-based method, and better trades off safety and efficiency in both seen and unseen scenarios. Moreover, qualitative results on a trajectory dataset indicate that our learned policy can be transferred to the real world for practical applications with real-time speeds. Demonstration videos are available athttps://caipeide.github.io/dq-gat/.

Light detection and ranging (LiDAR) plays an indispensable role in autonomous driving technologies, such as localization, map building, navigation and object avoidance. However, due to the vast amount of data, transmission and storage could become an important bottleneck. In this article, we propose a novel compression architecture for multi-line LiDAR point cloud sequences based on clustering and convolutional long short-term memory (LSTM) networks. LiDAR point clouds are structured, which provides an opportunity to convert the 3D data to 2D array, represented as range images. Thus, we cast the 3D point clouds compression as a range image sequence compression problem. Inspired by the high efficiency video coding (HEVC) algorithm, we design a novel compression framework for LiDAR data that includes two main techniques: intra-prediction and inter-prediction. For intra-frames, inspired by the depth modeling modes (DMM) adopted in 3D-HEVC, we develop a clustering-based intra-prediction technique, which can utilize the spatial structure characteristics of point clouds to remove the spatial redundancy. For inter-frames, we design a prediction network model using convolutional LSTM cells. The network model is capable of predicting future inter-frames using the encoded intra-frames. As a result, temporal redundancy can be removed. Experiments on the KITTI dataset demonstrate that the proposed method achieves an impressive compression ratio (CR), with 4.10% at millimeter precision, which means the point clouds can compress to nearly 1/25 of their original size. Additionally, compared with the well-known octree, Google Draco, and MPEG TMC13 methods, our algorithm yields better performance in compression ratio.

Combining multiple LiDARs enables a robot to maximize its perceptual awareness of environments and obtain sufficient measurements, which is promising for simultaneous localization and mapping (SLAM). This article proposes a system to achieve robust and simultaneous extrinsic calibration, odometry, and mapping for multiple LiDARs. Our approach starts with measurement preprocessing to extract edge and planar features from raw measurements. After a motion and extrinsic initialization procedure, a sliding window-based multi-LiDAR odometry runs onboard to estimate poses with an online calibration refinement and convergence identification. We further develop a mapping algorithm to construct a global map and optimize poses with sufficient features together with a method to capture and reduce data uncertainty. We validate our approach’s performance with extensive experiments on 10 sequences (4.60-km total length) for the calibration and SLAM and compare it against the state of the art. We demonstrate that the proposed work is a complete, robust, and extensible system for various multi-LiDAR setups. The source code, datasets, and demonstrations are available at:https://ram-lab.com/file/site/m-loam.

Pose estimation problems are fundamental in robotics. Most of these problems are challenging due to the nonconvex nature. This also sets up an obstacle for uncertainty description that is essential for pose integration and quality control. In this article, we show that a large class of related problems can be categorized as the quadratic pose estimation problems (QPEPs) and we propose a general quaternion-based mathematical model to unify these problems. To solve the nonconvex QPEPs, a Gröbner-basis method is investigated to derive their globally optimal and robust solutions. Furthermore, we develop the rules for characterizing the solvability and observability of these solutions. In addition, the uncertainty description, i.e., covariance matrix, as an important piece of information in robotic state estimation frameworks, is analyzed in detail. Theoretical results show that the covariance can be estimated via online optimization, in an efficient and unbiased manner. In this way, both the solution and covariance are guaranteed to be globally optimal. Through simulations and experiments, we show that the proposed QPEP-based solver is not only accurate, robust, and efficient but outperforms the representatives for covariance estimation. The designed algorithms are also assembled as a C++/MATLAB/Octave/ROS library, while these developed interfaces are built for main stream platforms and simultaneous localization and mapping schemes.

Real-time and high-performance segmentation is a crucial but challenging perception task for computationally constrained robots, such as the humanoid NAO robot used in the RoboCup Soccer Standard Platform League. However, most existing convolutional neural network (CNN)-based models for semantic segmentation suffer from massive computational costs, which prevents them from being applied to performing real-time inference with a NAO. In this article, we first publish meticulously annotated datasets for training and evaluating semantic segmentation models. Then, we propose a fast downsampling module that downsamples the image while maintaining the spatial information and a novel dense learning module that learns high-level semantic information while recovering the spatial details. Based on these operations, by using a multiscale fusion method to recover the resolution, we propose a more efficient and real-time segmentation model called RoboSeg primarily aimed at offering better speed and accuracy tradeoffs. Finally, to accommodate practical engineering applications, we offer a promising deployment guideline for the CNN model describing how to deploy it on computational resource-limited robots and achieve real-time performance. The experimental results show that the RoboSeg exceeds the state-of-the-art networks in RoboCup scene segmentation: we attain a mean IoU of 87.35% and a pixel accuracy of 96.88% on our dataset using a model that contains only 0.29M parameters and performs just 0.73 GFLOPs. Under the proposed deployment strategies, the network can run at above 30 FPS on NAO robots with downsampled frames.

3D object detection has been wildly studied in recent years, especially for robot perception systems. However, existing 3D object detection is under a closed-set condition, meaning that the network can only output boxes of trained classes. Unfortunately, this closed-set condition is not robust enough for practical use, as it will identify unknown objects as known by mistake. Therefore, in this paper, we propose an open-set 3D object detector, which aims to (1) identify known objects, like the closed-set detection, and (2) identify unknown objects and give their accurate bounding boxes. Specifically, we divide the open-set 3D object detection problem into two steps: (1) finding out the regions containing the unknown objects with high probability and (2) enclosing the points of these regions with proper bounding boxes. The first step is solved by the finding that unknown objects are often classified as known objects with low confidence, and we show that the Euclidean distance sum based on metric learning is a better confidence score than the naive softmax probability to differentiate unknown objects from known objects. On this basis, unsupervised clustering is used to refine the bounding boxes of unknown objects. The proposed method combining metric learning and unsupervised clustering is called the MLUC network. Our experiments show that our MLUC network achieves state-of-the-art performance and can identify both known and unknown objects as expected.

In autonomous driving scenarios, edge cases require perception algorithms, like 3D object detection, to incrementally learn new data during a long term. To achieve it, previous methods seek help from knowledge distillation and recursively transfer knowledge from old models to new models. However, conflicts exist between the likelihood term and the distillation regularizer on both old and new knowledge. In this paper, we discuss the drawback of knowledge distillation in the task-incremental-learning scenario for 3D object detection and propose a New-Task-Aware Biased Sampling and Knowledge-Distillation-Aware Detection Loss to solve the conflicts. Based on the KITTI dataset, we thoroughly evaluate our proposed method from the aspects of both forward and backward transfer in the task incremental-learning scenario. A great margin of improvement on the whole task sequence (5.6 mAP) demonstrates the effectiveness of our proposed method.

Classical close-set semantic segmentation networks have limited ability to detect out-of-distribution (OOD) objects, which is important for safety-critical applications such as autonomous driving. Incrementally learning these OOD objects with few annotations is an ideal way to enlarge the knowledge base of the deep learning models. In this paper, we propose an open world semantic segmentation system that includes two modules: (1) an open-set semantic segmentation module to detect both in-distribution and OOD objects. (2) an incremental few-shot learning module to gradually incorporate those OOD objects into its existing knowledge base. This open world semantic segmentation system behaves like a human being, which is able to identify OOD objects and gradually learn them with corresponding supervision. We adopt the Deep Metric Learning Network (DMLNet) with contrastive clustering to implement open-set semantic segmentation. Compared to other open-set semantic segmentation methods, our DMLNet achieves state-of-the-art performance on three challenging open-set semantic segmentation datasets without using additional data or generative models. On this basis, two incremental few-shot learning methods are further proposed to progressively improve the DMLNet with the annotations of OOD objects.

Convolutional neural network (CNN)-based stereo matching approaches generally require a dense cost volume (DCV) for disparity estimation. However, generating such cost volumes is computationally-intensive and memory-consuming, hindering CNN training and inference efficiency. To address this problem, we propose SCV-Stereo, a novel CNN architecture, capable of learning dense stereo matching from sparse cost volume (SCV) representations. Our inspiration is derived from the fact that DCV representations are somewhat redundant and can be replaced with SCV representations. Benefiting from these SCV representations, our SCV-Stereo can update disparity estimations in an iterative fashion for accurate and efficient stereo matching. Extensive experiments carried out on the KITTI Stereo benchmarks demonstrate that our SCV-Stereo can significantly minimize the trade-off between accuracy and efficiency for stereo matching. Our project page is https://sites.google.com/view/scv-stereo.

Stereo matching is a key component of autonomous driving perception. Recent unsupervised stereo matching approaches have received adequate attention due to their advantage of not requiring disparity ground truth. These approaches, however, perform poorly near occlusions. To overcome this drawback, in this paper, we propose CoT-Stereo, a novel unsupervised stereo matching approach. Specifically, we adopt a co-teaching framework where two networks interactively teach each other about the occlusions in an unsupervised fashion, which greatly improves the robustness of unsupervised stereo matching. Extensive experiments on the KITTI Stereo benchmarks demonstrate the superior performance of CoT-Stereo over all other state-of-the-art unsupervised stereo matching approaches in terms of both accuracy and speed. Our project webpage is https://sites.google.com/view/cot-stereo.

Modern LiDAR-SLAM (L-SLAM) systems have shown excellent results in large-scale, real-world scenarios. However, they commonly have a high latency due to the expensive data association and nonlinear optimization. This paper demonstrates that actively selecting a subset of features significantly improves both the accuracy and efficiency of an L-SLAM system. We formulate the feature selection as a combinatorial optimization problem under a cardinality constraint to preserve the information matrix's spectral attributes. The stochastic-greedy algorithm is applied to approximate the optimal results in real-time. To avoid ill-conditioned estimation, we also propose a general strategy to evaluate the environment's degeneracy and modify the feature number online. The proposed feature selector is integrated into a multi-LiDAR SLAM system. We validate this enhanced system with extensive experiments covering various scenarios on two sensor setups and computation platforms. We show that our approach exhibits low localization error and speedup compared to the state-of-the-art L-SLAM systems. To benefit the community, we have released the source code: https://ram-lab.com/file/site/m-loam.

Object detection in 3D with stereo cameras is an important problem in computer vision, and is particularly crucial in low-cost autonomous mobile robots without LiDARs. Nowadays, most of the best-performing frameworks for stereo 3D object detection are based on dense depth reconstruction from disparity estimation, making them extremely computationally expensive. To enable real-world deployments of vision detection with binocular images, we take a step back to gain insights from 2D image-based detection frameworks and enhance them with stereo features. We incorporate knowledge and the inference structure from real-time one-stage 2D/3D object detector and introduce a light-weight stereo matching module. Our proposed framework, YOLOStereo3D, is trained on one single GPU and runs at more than ten fps. It demonstrates performance comparable to state-of-the-art stereo 3D detection frameworks without usage of LiDAR data. The code will be published in https://github.com/Owen-Liuyuxuan/visualDet3D.

Recently, deep-learning based approaches have achieved impressive performance for autonomous driving. However, end-to-end vision-based methods typically have limited interpretability, making the behaviors of the deep networks difficult to explain. Hence, their potential applications could be limited in practice. To address this problem, we propose an interpretable end-to-end vision-based motion planning approach for autonomous driving, referred to as IVMP. Given a set of past surrounding-view images, our IVMP first predicts future egocentric semantic maps in bird’s-eye-view space, which are then employed to plan trajectories for self-driving vehicles. The predicted future semantic maps not only provide useful interpretable information, but also allow our motion planning module to handle objects with low probability, thus improving the safety of autonomous driving. Moreover, we also develop an optical flow distillation paradigm, which can effectively enhance the network while still maintaining its real-time performance. Extensive experiments on the nuScenes dataset and closed-loop simulation show that our IVMP significantly outperforms the state-of-the-art approaches in imitating human drivers with a much higher success rate. Our project page is available at https://sites.google.com/view/ivmp.

Path planning is a fundamental capability for autonomous navigation of robotic wheelchairs. With the impressive development of deep-learning technologies, imitation learning-based path planning approaches have achieved effective results in recent years. However, the disadvantages of these approaches are twofold: 1) they may need extensive time and labor to record expert demonstrations as training data; and 2) existing approaches could only receive high-level commands, such as turning left/right. These commands could be less sufficient for the navigation of mobile robots (e.g., robotic wheelchairs), which usually require exact poses of goals. We contribute a solution to this problem by proposing S2P2, a self-supervised goal-directed path planning approach. Specifically, we develop a pipeline to automatically generate planned path labels given as input RGB-D images and poses of goals. Then, we present a best-fit regression plane loss to train our data-driven path planning model based on the generated labels. Our S2P2 does not need pre-built maps, but it can be integrated into existing map-based navigation systems through our framework. Experimental results show that our S2P2 outperforms traditional path planning algorithms, and increases the robustness of existing map-based navigation systems. Our project page is available at https://sites.google.com/view/s2p2.

A novel 3-dimensional (3-D) alignment method for point-cloud registration is proposed where the time-differential information of the measured points is employed. The new problem turns out to be a novel multi-dimensional optimization. Analytical solution to this optimization is then obtained, which sets the ground of further correspondence matching using k-D trees. Finally, via many examples, we show that the new method owns better registration accuracy in real-world experiments.

Navigation on challenging terrain topographies requires the understanding of robots’ locomotion capabilities to produce optimal solutions. We present an integrated framework for real-time autonomous navigation of mobile robots based on elevation maps. The framework performs rapid global path planning and optimization that is aware of the locomotion capabilities of the robot. A GPU-aided, sampling-based path planner combined with a gradient-based path optimizer provides optimal paths by using a neural network-based locomotion cost predictor which is trained in simulation. We show that our approach is capable of planning and optimizing paths three orders of magnitude faster than RRT* on GPU-enabled hardware, enabling real-time deployment on mobile platforms. We successfully evaluate the framework on the ANYmal C quadrupedal robot in both simulations and real-world environments for path planning tasks on multiple complex terrains.

In this paper, we introduce a method for visual relocalization using the geometric information from a 3D surfel map. A visual database is first built by global indices from the 3D surfel map rendering, which provides associations between image points and 3D surfels. Surfel reprojection constraints are utilized to optimize the keyframe poses and map points in the visual database. A hierarchical camera relocalization algorithm then utilizes the visual database to estimate 6-DoF camera poses. Learned descriptors are further used to improve the performance in challenging cases. We present evaluation under real-world conditions and simulation to show the effectiveness and efficiency of our method, and make the final camera poses consistently well aligned with the 3D environment.

Federated Learning (FL) creates an ecosystem for multiple agents to collaborate on building models with data privacy consideration. The method for contribution measurement of each agent in the FL system is critical for fair credits allocation but few are proposed. In this paper, we develop a real-time contribution measurement method FedCM that is simple but powerful. The method defines the impact of each agent, comprehensively considers the current round and the previous round to obtain the contribution rate of each agent with attention aggregation. Moreover, FedCM updates contribution every round, which enable it to perform in real-time. Real-time is not considered by the existing approaches, but it is critical for FL systems to allocate computing power, communication resources, etc. Compared to the state-of-the-art method, the experimental results show that FedCM is more sensitive to data quantity and data quality under the premise of real-time. Furthermore, we developed federated learning open-source software based on FedCM. The software has been applied to identify COVID-19 based on medical images.

Traditional decision and planning frameworks for self-driving vehicles (SDVs) scale poorly in new scenarios, thus they require tedious hand-tuning of rules and parameters to maintain acceptable performance in all foreseeable cases. Recently, self-driving methods based on deep learning have shown promising results with better generalization capability but less hand engineering effort. However, most of the previous learning-based methods are trained and evaluated in limited driving scenarios with scattered tasks, such as lane-following, autonomous braking, and conditional driving. In this paper, we propose a graph-based deep network to achieve scalable self-driving that can handle massive traffic scenarios. Specifically, more than 7,000 km of evaluation is conducted in a high-fidelity driving simulator, in which our method can obey the traffic rules and safely navigate the vehicle in a large variety of urban, rural, and highway environments, including unprotected left turns, narrow roads, roundabouts, and pedestrian-rich intersections. Demonstration videos are available at https: //caipeide.github.io/dignet/.

Freespace detection is a fundamental component of autonomous driving perception. Recently, deep convolutional neural networks (DCNNs) have achieved impressive performance for this task. In particular, SNE-RoadSeg, our previously proposed method based on a surface normal estimator (SNE) and a data-fusion DCNN (RoadSeg), has achieved impressive performance in freespace detection. However, SNE-RoadSeg is computationally intensive, and it is difficult to execute in real time. To address this problem, we introduce SNE-RoadSeg+, an upgraded version of SNE-RoadSeg. SNE-RoadSeg+ consists of 1) SNE+, a module for more accurate surface normal estimation, and 2) RoadSeg+, a data-fusion DCNN that can greatly minimize the trade-off between accuracy and efficiency with the use of deep supervision. Extensive experimental results have demonstrated the effectiveness of our SNE+ for surface normal estimation and the superior performance of our SNE-RoadSeg+ over all other freespace detection approaches. Specifically, our SNE-RoadSeg+ runs in real time, and meanwhile, achieves the state-of-the-art performance on the KITTI road benchmark. Our project page is at https://www.sne-roadseg.site/sne-roadseg-plus.

Road curb detection is important for autonomous driving. It can be used to determine road boundaries to constrain vehicles on roads, so that potential accidents could be avoided. Most of the current methods detect road curbs online using vehicle-mounted sensors, such as cameras or 3-D Lidars. However, these methods usually suffer from severe occlusion issues. Especially in highly-dynamic traffic environments, most of the field of view is occupied by dynamic objects. To alleviate this issue, we detect road curbs offline using high-resolution aerial images in this paper. Moreover, the detected road curbs can be used to create high-definition (HD) maps for autonomous vehicles. Specifically, we first predict the pixel-wise segmentation map of road curbs, and then conduct a series of post-processing steps to extract the graph structure of road curbs. To tackle the disconnectivity issue in the segmentation maps, we propose an innovative connectivity-preserving loss (CP-loss) to improve the segmentation performance. The experimental results on a public dataset demonstrate the effectiveness of our proposed loss function. This paper is accompanied with a demonstration video and a supplementary document, which are available at https://sites.google.com/view/cp-loss.

Abstract The unstable link quality in wireless sensor networks (WSNs) directly affects the success rate of data transmission. The retransmission mechanism is one of the commonly used methods to solve this problem. However, too many retransmissions could lower the communication efficiency. Therefore, to reduce retransmissions while guaranteeing the communication reliability in WSNs, this study introduces the NarrowBand Internet of Things (NB‐IoT) technology, and builds a network including both sensor and NB‐IoT nodes. The NB‐IoT node is designed to support both the radio frequency and the NarrowBand communication modes; thus it can communicate with both sensor nodes and NB‐IoT base stations. Further, the communication reliability metrics considering both the link quality and the number of retransmissions in different communication modes are defined, based on which, an adaptive communication reliability algorithm is proposed to switch the communication modes of nodes. The simulation results verify that the proposed algorithm can achieve higher success rate with less end‐to‐end delay and flexibly control the cost on NB‐IoT communication.

Parafoil airdrop is an important way to deliver goods and materials to area where road vehicles are not easy to reach. However, it is difficult to deliver large quantities of goods and materials to a given location with only one parafoil. Airdropping multiple parafoils is an effective choice for transporting large quantities of goods and materials. To realize the cooperative airdrop of multiple parafoils, a cooperative guidance framework is proposed. First, a trajectory planning algorithm is designed to plan the multiphase trajectory for the parafoil group. Then, a trajectory tracking algorithm is developed for the pilot parafoil in the parafoil group to reliably follow the planned trajectory. Finally, a cooperative formation guidance strategy is designed based on the leader–follower consensus theory. Under this strategy, the position and speed of the follower parafoil can be consensus with those of the leader parafoil. Lyapunov’s theorem proves the stability of this strategy. We evaluate the effectiveness of this framework through simulations. The results demonstrate that our algorithms can realize the precise airdrop of massive goods and materials with upwind landing using multiple parafoils. In addition, the parafoils could be gradually gathered to a desired formation, and safe distances could be maintained between parafoils during the airdrop process.Note to Practitioners—This article was motivated by the problem of airdropping massive goods and materials. Existing methods usually adopt a single heavy parafoil, or use centralized multiparafoil systems. Both these methods have their limitations. For the former, there is an upper limit of the load capacity for a single parafoil. For the latter, the parafoils in the centralized system lack fully autonomous ability. Distributed multiparafoil systems could solve the problem effectively. However, compared to single-parafoil systems, there are still some challenges, for example, the multiparafoil gathering, collision avoidance and cooperative formation, as well as the upwind landing. Fortunately, existing parafoils are equipped with sensors, communication, and control devices, so they could be viewed as agents with autonomous capabilities. In this article, a formation guidance framework for multiple autonomous parafoils is proposed. First, we plan a trajectory for the pilot parafoil. Then, we show how to effectively track the planned trajectory. Finally, we demonstrate how multiple parafoils could coordinate with each other to accomplish airdrop tasks. The simulation results confirm the feasibility of this strategy.

In this article, we focus on resolving the traffic flow prediction, robot path planning, and motion coordination problems in large-scale warehousing robotics systems with thousand-robot networks. The warehousing environment is partitioned into several sectors, and a hierarchical framework is developed, which includes a centralized prediction and planning level and a decentralized local coordination level. In the centralized level, a traffic flow prediction algorithm is first proposed to predict the evolution of the robot density distribution in a future horizon and estimate the future traffic heat value of each sector. Based on this, the sector-level robot path can be generated in the time-expended sector graph by comprehensively considering the traveling distance and the predicted traffic heat value and will be dynamically updated by considering the most recent traffic information. In the coordination level, local cooperative A* algorithm, incorporated with the conflict-based searching strategy, is implemented within each sector to generate conflict-free road-level paths for all the robots in the sector simultaneously, and the rolling planning scheme is utilized in order to immediately react to robot motion uncertainties and communication disconnections. The effectiveness and practical applicability of the proposed approach are validated by large-scale simulations with more than one 1000 robots and real laboratory experiments.Note to Practitioners—Considering practical situations and requirements in industrial warehouses and automated logistics systems, this article resolves the life-long planning and coordination problems of large-scale robot networks and ensures the practical execution performance in the presence of robot motion uncertainties and temporary communication disconnections. Our main idea is to reduce robot congestions and improve warehouse working efficiency by balancing the traffic flow in the whole environment. To achieve this, we present a traffic flow prediction algorithm to estimate the robot density distribution in a future horizon and take this information into consideration in sector-level path planning. The reliability, scalability, and the real-time performance of the proposed solution are achieved by the presented hierarchical system framework and the dynamic planning scheme. The proposed concept and approach can also be used to coordinate other large-scale systems with multirobot or multi-AGV networks. Simulation and experimental results suggest that the proposed solution is effective and practically applicable, but a saturation phenomenon of the system capacity can be observed under a very heavy workload. In the future, we will investigate the relation between the maximum system capacity and the environment structure and make further efforts to optimize the environment structure and road layout in order to improve the warehouse working efficiency.

Semantic segmentation of urban scenes is an essential component in various applications of autonomous driving. It makes great progress with the rise of deep learning technologies. Most of the current semantic segmentation networks use single-modal sensory data, which are usually the RGB images produced by visible cameras. However, the segmentation performance of these networks is prone to be degraded when lighting conditions are not satisfied, such as dim light or darkness. We find that thermal images produced by thermal imaging cameras are robust to challenging lighting conditions. Therefore, in this article, we propose a novel RGB and thermal data fusion network named FuseSeg to achieve superior performance of semantic segmentation in urban scenes. The experimental results demonstrate that our network outperforms the state-of-the-art networks.Note to Practitioners—This article investigates the problem of semantic segmentation of urban scenes when lighting conditions are not satisfied. We provide a solution to this problem via information fusion with RGB and thermal data. We build an end-to-end deep neural network, which takes as input a pair of RGB and thermal images and outputs pixel-wise semantic labels. Our network could be used for urban scene understanding, which serves as a fundamental component of many autonomous driving tasks, such as environment modeling, obstacle avoidance, motion prediction, and planning. Moreover, the simple design of our network allows it to be easily implemented using various deep learning frameworks, which facilitates the applications on different hardware or software platforms.

As a key research issue in mobile crowdsensing (MCS), recent studies on task recommendation have begun to focus on recommending tasks to participants according to the learned participant preferences. The common drawbacks of these studies are that, on the one hand, the factors affecting participant preferences are predefined, which is not practical as the influential factors are quite complex and a full map of participant profiles needs to be preexisted. On the other hand, they do not consider how to update the recommendation dynamically. To overcome these drawbacks, a profile-free and real-time task recommendation method is proposed in this work. First, we apply the recommendation systems to MCS to realize profile-free task recommendations. Second, a participant-task-location tensor is constructed, based on which an improved tensor factorization method is presented to provide task recommendations for participants at a given location. Finally, we design a real-time update algorithm based on the idea of one update at a time to update task recommendation lists for participants in real time. Based on real-world trace data sets, extensive evaluations show that the proposed method has obvious advantages over other baselines in terms of accuracy and time cost.

Existing road pothole detection approaches can be classified as computer vision-based or machine learning-based. The former approaches typically employ 2D image analysis/ understanding or 3D point cloud modeling and segmentation algorithms to detect (i.e., recognize and localize) road potholes from vision sensor data, e.g., RGB images and/or depth/disparity images. The latter approaches generally address road pothole detection using convolutional neural networks (CNNs) in an end-to-end manner. However, road potholes are not necessarily ubiquitous and it is challenging to prepare a large well-annotated dataset for CNN training. In this regard, while computer vision-based methods were the mainstream research trend in the past decade, machine learning-based methods were merely discussed. Recently, we published the first stereo vision-based road pothole detection dataset and a novel disparity transformation algorithm, whereby the damaged and undamaged road areas can be highly distinguished. However, there are no benchmarks currently available for state-of-the-art (SoTA) CNNs trained using either disparity images or transformed disparity images. Therefore, in this paper, we first discuss the SoTA CNNs designed for semantic segmentation and evaluate their performance for road pothole detection with extensive experiments. Additionally, inspired by graph neural network (GNN), we propose a novel CNN layer, referred to as graph attention layer (GAL), which can be easily deployed in any existing CNN to optimize image feature representations for semantic segmentation. Our experiments compare GAL-DeepLabv3+, our best-performing implementation, with nine SoTA CNNs on three modalities of training data: RGB images, disparity images, and transformed disparity images. The experimental results suggest that our proposed GAL-DeepLabv3+ achieves the best overall pothole detection accuracy on all training data modalities. The source code, dataset, and benchmark are publicly available at mias.group/GAL-Pothole-Detection.

Vision-based autonomous driving through imitation learning mimics the behavior of human drivers by mapping driver view images to driving actions. This article shows that performance can be enhanced via the use of eye gaze. Previous research has shown that observing an expert's gaze patterns can be beneficial for novice human learners. We show here that neural networks can also benefit. We trained a conditional generative adversarial network to estimate human gaze maps accurately from driver-view images. We describe two approaches to integrating gaze information into imitation networks: eye gaze as an additional input and gaze modulated dropout. Both significantly enhance generalization to unseen environments in comparison with a baseline vanilla network without gaze, but gaze-modulated dropout performs better. We evaluated performance quantitatively on both single images and in closed-loop tests, showing that gaze modulated dropout yields the lowest prediction error, the highest success rate in overtaking cars, the longest distance between infractions, lowest epistemic uncertainty, and improved data efficiency. Using Grad-CAM, we show that gaze modulated dropout enables the network to concentrate on task-relevant areas of the image.

In this paper, an online foot position compensator (FPC) is proposed for improving the robustness of humanoid walking based on orbital energy conservation and discrete control Lyapunov function (DCLF), with which the asymptotic stability of the humanoid system can be maintained and, thus, the foot placement control is achieved. The online FPC is developed based on linear model predictive control (MPC) by replanning the trajectories of the center of mass (CoM) and properly placing the footsteps to resist external disturbances and recover the walking posture. To further improve the robustness of the humanoid robots to suppress strong external disturbance, a strategy of upper body posture control is proposed. The presented controller stabilizes the humanoid robot by utilizing hip joints to modulate the upper body posture online. Webots simulations and real experiments on a full-body NAO humanoid robot verify the effectiveness of the proposed methods.

Robustness and accuracy for monocular visual odometry (VO) under challenging environments are widely concerned. In this paper, we present a monocular VO system leveraging learned repeatability and description. In a hybrid scheme, the camera pose is initially tracked on the predicted repeatability maps in a direct manner and then refined with the patch-wise 3D-2D association. The local feature parameterization and the adapted mapping module further boost different functionalities in the system. Extensive evaluations on challenging public datasets are performed. The competitive performance on camera pose estimation demonstrates the effectiveness of our method. Additional studies on the local reconstruction accuracy and running time exhibit that our system is capable of maintaining a robust and lightweight backend.

We present LINS, a lightweight lidar-inertial state estimator, for real-time ego-motion estimation. The proposed method enables robust and efficient navigation for ground vehicles in challenging environments, such as feature-less scenes, via fusing a 6-axis IMU and a 3D lidar in a tightly-coupled scheme. An iterated error-state Kalman filter (ESKF) is designed to correct the estimated state recursively by generating new feature correspondences in each iteration, and to keep the system computationally tractable. Moreover, we use a robocentric formulation that represents the state in a moving local frame in order to prevent filter divergence in a long run. To validate robustness and generalizability, extensive experiments are performed in various scenarios. Experimental results indicate that LINS offers comparable performance with the state-of-the-art lidar-inertial odometry in terms of stability and accuracy and has order-of-magnitude improvement in speed.

In this paper, we introduce an approach to tracking the pose of a monocular camera in a prior surfel map. By rendering vertex and normal maps from the prior surfel map, the global planar information for the sparse tracked points in the image frame is obtained. The tracked points with and without the global planar information involve both global and local constraints of frames to the system. Our approach formulates all constraints in the form of direct photometric errors within a local window of the frames. The final optimization utilizes these constraints to provide the accurate estimation of global 6-DoF camera poses with the absolute scale. The extensive simulation and real-world experiments demonstrate that our monocular method can provide accurate camera localization results under various conditions.

The presence of any type of defect on the glass screen of smart devices has a great impact on their quality. We present a robust semi-supervised learning framework for intelligent micro-scaled localization and classification of defects on a 16K pixel image of smartphone glass. Our model features the efficient recognition and labeling of three types of defects: scratches, light leakage due to cracks, and pits. Our method also differentiates between the defects and light reflections due to dust particles and sensor regions, which are classified as non-defect areas. We use a partially labeled dataset to achieve high robustness and excellent classification of defect and non-defect areas as compared to principal components analysis (PCA), multi-resolution and information-fusion-based algorithms. In addition, we incorporated two classifiers at different stages of our inspection framework for labeling and refining the unlabeled defects. We successfully enhanced the inspection depth-limit up to 5 microns. The experimental results show that our method outperforms manual inspection in testing the quality of glass screen samples by identifying defects on samples that have been marked as good by human inspection.

Extrinsic perturbation always exists in multiple sensors. In this paper, we focus on the extrinsic uncertainty in multi-LiDAR systems for 3D object detection. We first analyze the influence of extrinsic perturbation on geometric tasks with two basic examples. To minimize the detrimental effect of extrinsic perturbation, we propagate an uncertainty prior on each point of input point clouds, and use this information to boost an approach for 3D geometric tasks. Then we extend our findings to propose a multi-LiDAR 3D object detector called MLOD. MLOD is a two-stage network where the multi-LiDAR information is fused through various schemes in stage one, and the extrinsic perturbation is handled in stage two. We conduct extensive experiments on a real-world dataset, and demonstrate both the accuracy and robustness improvement of MLOD. The code, data and supplementary materials are available at: https://ram-lab.com/file/site/mlod.

The joint detection of drivable areas and road anomalies is a crucial task for ground mobile robots. In recent years, many impressive semantic segmentation networks, which can be used for pixel-level drivable area and road anomaly detection, have been developed. However, the detection accuracy still needs improvement. Therefore, we develop a novel module named the Normal Inference Module (NIM), which can generate surface normal information from dense depth images with high accuracy and efficiency. Our NIM can be deployed in existing convolutional neural networks (CNNs) to refine the segmentation performance. To evaluate the effectiveness and robustness of our NIM, we embed it in twelve state-of-the-art CNNs. The experimental results illustrate that our NIM can greatly improve the performance of the CNNs for drivable area and road anomaly detection. Furthermore, our proposed NIM-RTFNet ranks 8th on the KITTI road benchmark and exhibits a real-time inference speed.

In this paper, we address the issue of pedestrian tracking in crowd scenarios. People in close social relationships tend to act as a group which is a great challenge to individually discriminate and track pedestrians on a LiDAR system. In this paper, we integrally model groups of people and track them in a recursive framework based on Gaussian Mixture Model (GMM). The model is optimized by an extended Expectation-Maximization (EM) algorithm which can adaptively vary the number of mixture components over scans. Experimental results both qualitatively and quantitatively indicate the reliability and accuracy of our tracker in populated scenarios.

Due to the huge volume of point cloud data, storing or transmitting it is currently difficult and expensive in autonomous driving. Learning from the high efficiency video coding (HEVC) coding framework, we propose an advanced coding scheme for large-scale LiDAR point cloud sequences, in which several techniques have been developed to remove the spatial and temporal redundancy. The proposed strategy consists mainly of intra-coding and inter-coding. For intra-coding, we utilize a cluster-based prediction method to remove the spatial redundancy. For inter-coding, a predictive recurrent network is designed, which is capable of generating future frames according to the previously encoded frames. By calculating the residual error between the predicted and real point cloud data, the temporal redundancy can be removed. Finally, the residual data is quantized and encoded by lossless coding schemes. Experiments are conducted on the KITTI data set with four different scenes to verify the effectiveness and efficiency of the proposed method. Our approach can deal with multiple types of point cloud data from the simple to more complex, and yields better performance in terms of compression ratio compared with octree, Google Draco, MPEG TMC13 and other recently proposed methods.

Opportunistic signals (e.g., WiFi, magnetic fields, and ambient light) have been extensively studied for low-cost indoor localization, especially via fingerprinting. We present an automatic site survey approach to build the signal maps in space-constrained environments (e.g., modern office buildings). The survey can be completed by a single smartphone user during normal walking, say, with a little human intervention. Our approach follows the classical GraphSLAM framework: the front end constructs a pose graph by incorporating the relative motion constraints from the pedestrian dead-reckoning (PDR), the loop-closure constraints by magnetic sequence matching with the WiFi signal similarity validation, and the global heading constraints from the opportunistic magnetic heading measurements; and the back end generates a globally consistent trajectory via graph optimization to provide ground-truth locations for the collected signal fingerprints along the survey path. We then build the signal map (also known as fingerprint database) upon these location-labeled fingerprints by the Gaussian processes regression (GPR) for later online localization. Specifically, we exploit the pseudowall constraints from the GPR variance map of magnetic fields and the observations of ceiling lights to correct the PDR drifts with a particle filter. We evaluate our approach on several data sets collected from both the HKUST academic building and a shopping mall. We demonstrate the real-time localization on a smartphone in an office area, with 50th percentile accuracy of 2.30 m and 90th percentile accuracy of 3.41 m. Note to Practitioners - This paper was motivated by the problem of the efficient signal map construction for fingerprinting-based localization on smartphones. The conventional manual site survey method, known to be time-consuming and labor-intensive, hinders the penetration of fingerprinting methods in practice. This paper suggests a GraphSLAM-based approach to automate this signal map construction process by reducing the survey overhead significantly. A surveyor is merely asked to walk through an indoor venue with an Android smartphone held in hand with a little human intervention. Meanwhile, opportunistic signals (e.g., WiFi and magnetic fields) are captured by smartphone sensors. We construct a GraphSLAM engine to first identify the measurement constraints from these signal observations and then recover the surveyor's walking trajectory by the graph optimization. We can generate signal maps using the captured signals alongside the recovered trajectory. In this paper, we propose a WiFi signal similarity validation method to reduce false positive loop-closures and exploit the magnetic headings to improve the trajectory optimization performance. In addition, we propose to use the generated magnetic field variance map and the lights distribution map for localization. The efficacy of the proposed site survey approach is proven through field experiments, and real-time localization is demonstrated on a smartphone using the generated signal maps. The localization experiment was conducted by a single user with the same Android smartphone that was used in the site survey. Therefore, the usability of signal maps on other devices and the generality to other users have not yet been testified. We will leave these issues in our future work.

3-D registration has always been performed invoking singular value decomposition (SVD) or eigenvalue decomposition (EIG) in real engineering practices. However, these numerical algorithms suffer from uncertainty of convergence in many cases. A novel fast symbolic solution is proposed in this article by following our recent publication in this journal. The equivalence analysis shows that our previous solver can be converted to deal with the 3-D registration problem. Rather, the computation procedure is studied for further simplification of computing without complex-number support. Experimental results show that the proposed solver does not loose accuracy and robustness but improves the execution speed to a large extent by almost 50%-80%, on both a personal computer (PC) and an embedded processor. Note to Practitioners-3-D registration usually has a large computational burden in engineering tasks. The proposed symbolic solution can directly solve the eigenvalue and its associated eigenvector. A lot of computation resources can then be saved for better overall system performance. The deterministic behavior of the proposed solver also ensures long-endurance stability and can help an engineer better design thread timing logic.

Pothole detection is one of the most important tasks for road maintenance. Computer vision approaches are generally based on either 2D road image analysis or 3D road surface modeling. However, these two categories are always used independently. Furthermore, the pothole detection accuracy is still far from satisfactory. Therefore, in this paper, we present a robust pothole detection algorithm that is both accurate and computationally efficient. A dense disparity map is first transformed to better distinguish between damaged and undamaged road areas. To achieve greater disparity transformation efficiency, golden section search and dynamic programming are utilized to estimate the transformation parameters. Otsu's thresholding method is then used to extract potential undamaged road areas from the transformed disparity map. The disparities in the extracted areas are modeled by a quadratic surface using least squares fitting. To improve disparity map modeling robustness, the surface normal is also integrated into the surface modeling process. Furthermore, random sample consensus is utilized to reduce the effects caused by outliers. By comparing the difference between the actual and modeled disparity maps, the potholes can be detected accurately. Finally, the point clouds of the detected potholes are extracted from the reconstructed 3D road surface. The experimental results show that the successful detection accuracy of the proposed system is around 98.7% and the overall pixel-level accuracy is approximately 99.6%.

Unfortunately, we made two minor mistakes in the above paper. First of all, the first graph on row (c) inFig. 11was same as the third graph on row (c) inFig. 11. Secondly, “precision” and “recall” inTable IIIneed to be switched. The correct figure and table have no influence on the discussion and conclusions in the above paper, and they are given here.

This article presents a novel road damage detection algorithm based on unsupervised disparity map segmentation. Firstly, a disparity map is transformed by minimizing an energy function with respect to stereo rig roll angle and road disparity projection model. Instead of solving this energy minimization problem using non-linear optimization techniques, we directly find its numerical solution. The transformed disparity map is then segmented using Otus's thresholding method, and the damaged road areas can be extracted. The proposed algorithm requires no parameters when detecting road damage. The experimental results illustrate that our proposed algorithm performs both accurately and efficiently. The pixel-level road damage detection accuracy is approximately 97.56%. The source code is publicly available at: https://github.com/ruirangerfan/unsupervised_disparity_map_segmentation.git.

In the past several years, there have been several representative attitude determination methods developed using derivative-based optimization algorithms. Optimization techniques e.g. gradient-descent algorithm (GDA), Gauss-Newton algorithm (GNA), Levenberg-Marquadt algorithm (LMA) suffer from local optimum in real engineering practices. A brief discussion on the convexity of this problem is presented recently [1] stating that the problem is neither convex nor concave. In this paper, we give analytic proofs on this problem. The results reveal that the target loss function is convex in the common practice of quaternion normalization, which leads to non-existence of local optimum.

End-to-end behavioral cloning trained by human demonstration is now a popular approach for vision-based autonomous driving. A deep neural network maps drive-view images directly to steering commands. However, the images contain much task-irrelevant data. Humans attend to behaviorally relevant information using saccades that direct gaze towards important areas. We demonstrate that behavioral cloning also benefits from active control of gaze. We trained a conditional generative adversarial network (GAN) that accurately predicts human gaze maps while driving in both familiar and unseen environments. We incorporated the predicted gaze maps into end-to-end networks for two behaviors: following and overtaking. Incorporating gaze information significantly improves generalization to unseen environments. We hypothesize that incorporating gaze information enables the network to focus on task critical objects, which vary little between environments, and ignore irrelevant elements in the background, which vary greatly.

The detection of curved lanes is still challenging for autonomous driving systems. Although current cutting-edge approaches have performed well in real applications, most of them are based on strict model assumptions. Similar to other visual recognition tasks, lane detection can be formulated as a two-dimensional graph searching problem, which can be solved by finding several optimal paths along with line segments and boundaries. In this paper, we present a directed graph model, in which dynamic programming is used to deal with a specific shortest path problem. This model is particularly suitable to represent objects with long continuous shape structure, e.g., lanes and roads. We apply the designed model and proposed an algorithm for detecting lanes by formulating it as the shortest path problem. To evaluate the performance of our proposed algorithm, we tested five sequences (including 1573 frames) from the KITTI database. The results showed that our method achieves an average successful detection precision of 97.5%.

Ego-motion estimation is a fundamental requirement for most mobile robotic applications. By sensor fusion, we can compensate the deficiencies of stand-alone sensors and provide more reliable estimations. We introduce a tightly coupled lidar-IMU fusion method in this paper. By jointly minimizing the cost derived from lidar and IMU measurements, the lidarIMU odometry (LIO) can perform well with considerable drifts after long-term experiment, even in challenging cases where the lidar measurement can be degraded. Besides, to obtain more reliable estimations of the lidar poses, a rotation-constrained refinement algorithm (LIO-mapping) is proposed to further align the lidar poses with the global map. The experiment results demonstrate that the proposed method can estimate the poses of the sensor pair at the IMU update rate with high precision, even under fast motion conditions or with insufficient features.

Due to uneven spatial distribution of ground air quality monitoring stations, the temporal and spatial distribution of air pollution cannot be accurately revealed only based on ground-based data. Moreover, there is uncertainty in the application of established global air quality estimation products due to lack of validation of ground monitoring data. In this study, we chose Beijing-Tianjin-Hebei (BTH) with the serious air pollution in China as the research area. The global resolved datasets and the ground-level PM2.5measurements from China national air quality networks are combined as the basic data to build a spatial ground-level PM2.5estimation product with high spatio-temporal resolution. In order to improve the precision of PM2.5estimation model, multivariate data related to PM2.5was selected as input in modeling. We verified the applicability of the model to monitor PM2.5concentrations in different season. By testing and modifying the seasonal model group the seasonal model group, the better-precision annual model will be built. As a result of its fine resolution and complete spatial coverage, the PM2.5estimation model could provide extensive and insightful benefits for ground-level PM2.5estimation in the BTH area.

Accurate indoor positioning is urgent for critical location-based services. The approach based on visible light communication (VLC) is promising, as it can deliver high accuracy by sharing the LED lighting infrastructure. In this paper, we propose an EKF-based tightly-coupled visual-inertial fusion method for visible light positioning with an IMU and a rolling-shutter camera, aiming for improved positioning robustness under LED shortage. With the proposed method, we can relax the assumption on the minimum number of concurrently observable LEDs required for positioning from three to one. Meanwhile, we can accurately track the sensor pair’s global 3D pose in realtime. We evaluate our method by real-world experiments using a prototyping VLC network. The efficacy for VLC beaconing and 3D pose estimation, as well as the robustness under LED shortage, is verified by extensive experiments.

Imitation learning by behavioral cloning is a prevalent method that has achieved some success in vision-based autonomous driving. The basic idea behind behavioral cloning is to have the neural network learn from observing a human expert's behavior. Typically, a convolutional neural network learns to predict the steering commands from raw driver-view images by mimicking the behaviors of human drivers. However, there are other cues, such as gaze behavior, available from human drivers that have yet to be exploited. Previous researches have shown that novice human learners can benefit from observing experts' gaze patterns. We present here that deep neural networks can also profit from this. We propose a method, gaze-modulated dropout, for integrating this gaze information into a deep driving network implicitly rather than as an additional input. Our experimental results demonstrate that gaze-modulated dropout enhances the generalization capability of the network to unseen scenes. Prediction error in steering commands is reduced by 23.5% compared to uniform dropout. Running closed loop in the simulator, the gaze-modulated dropout net increased the average distance travelled between infractions by 58.5%. Consistent with these results, the gazemodulated dropout net shows lower model uncertainty.

Metric localization plays a critical role in vision-based navigation. For overcoming the degradation of matching photometry under appearance changes, recent research resorted to introducing geometry constraints of the prior scene structure. In this paper, we present a metric localization method for the monocular camera, using the Signed Distance Field (SDF) as a global map representation. Leveraging the volumetric distance information from SDFs, we aim to relax the assumption of an accurate structure from the local Bundle Adjustment (BA) in previous methods. By tightly coupling the distance factor with temporal visual constraints, our system corrects the odometry drift and jointly optimizes global camera poses with the local structure. We validate the proposed approach on both indoor and outdoor public datasets. Compared to the state-of-the-art methods, it achieves a comparable performance with a minimal sensor configuration.

Multiple LiDARs have progressively emerged on autonomous vehicles for rendering a rich view and dense measurements. However, the lack of precise calibration negatively affects their potential applications. In this paper, we propose a novel system that enables automatic multi-LiDAR calibration method without any calibration target, prior environment information, and manual initialization. Our approach starts with a hand-eye calibration by aligning the motion of each sensor. The initial results are then refined by an appearance-based method by minimizing a cost function constructed by point-plane distance. Experimental results on simulated and real-world data demonstrate the reliability and accuracy of our calibration approach. The proposed approach can calibrate a multi-LiDAR system with the rotation and translation errors less than 0. 04rad and 0. 1m respectively for a mobile platform.

This paper was motivated by the problem of how to make robots fuse and transfer their experience so that they can effectively use prior knowledge and quickly adapt to new environments. To address the problem, we present a learning architecture for navigation in cloud robotic systems: Lifelong Federated Reinforcement Learning (LFRL). In the work, we propose a knowledge fusion algorithm for upgrading a shared model deployed on the cloud. Then, effective transfer learning methods in LFRL are introduced. LFRL is consistent with human cognitive science and fits well in cloud robotic systems. Experiments show that LFRL greatly improves the efficiency of reinforcement learning for robot navigation. The cloud robotic system deployment also shows that LFRL is capable of fusing prior knowledge. In addition, we release a cloud robotic navigation-learning website to provide the service based on LFRL: www.shared-robotics.com.

End-to-end visual-based imitation learning has been widely applied in autonomous driving. When deploying the trained visual-based driving policy, a deterministic command is usually directly applied without considering the uncertainty of the input data. Such kind of policies may bring dramatical damage when applied in the real world. In this paper, we follow the recent real-to-sim pipeline by translating the testing world image back to the training domain when using the trained policy. In the translating process, a stochastic generator is used to generate various images stylized under the training domain randomly or directionally. Based on those translated images, the trained uncertainty-aware imitation learning policy would output both the predicted action and the data uncertainty motivated by the aleatoric loss function. Through the uncertainty-aware imitation learning policy, we can easily choose the safest one with the lowest uncertainty among the generated images. Experiments in the Carla navigation benchmark show that our strategy outperforms previous methods, especially in dynamic environments.

The visual-inertial navigation system (VINS) has been a practical approach for state estimation in recent years. In this paper, we propose a general GPS-aided omnidirectional visual-inertial state estimator capable of operating in ubiquitous environments and platforms. Our system consists of two parts: 1) the pre-processing of omnidirectional cameras, IMU, and GPS measurements, and 2) the sliding window based nonlinear optimization for accurate state estimation. We test our system in different conditions including an indoor office, campus roads, and challenging open water surface. Experiment results demonstrate the high accuracy of our approach than state-of-the-art VINSs in all scenarios. The proposed odometry achieves drift ratio less than 0.5% in 1200 m length outdoors campus road in overexposure conditions and 0.65% in open water surface, without a loop closure, compared with a centimeter accuracy GPS reference.

Nowadays, a large amount of labeled data is crucial for deep neural network training. However, data labeling is still a time- and labor-consuming task, especially when labeling 3D point clouds. Meanwhile, object recognition has achieved great success on 2D images, even beyond the ability of humans. In this paper, we propose an effective framework to produce labeled data by using an image detector as a supervisor, and we train the network with a simple trick to eliminate noisy labels. For object-sparse scenes, this method is able to obtain good label data, while for object-dense scenes, we can use our training method to detect some of the corrupted labels. This is realized by building a cohesive camera and LiDAR system (named “Licam”) and performing target frustum region proposal on point clouds using the camera detection result. Efficient and effective vehicle detection is achieved based on this learning and training framework. We examine this method on the KITTI dataset [7] and our own road running data collected from a micro electro mechanical system (MEMS) LiDAR, demonstrating fast and accurate detection results. The results show that our automatic data labeling and training framework is effective and efficient. It provides the ability to obtain large-scale labeled data, and is easy to use for online learning.

Crack is one of the most common road distresses which may pose road safety hazards. Generally, crack detection is performed by either certified inspectors or structural engineers. This task is, however, time-consuming, subjective and labor-intensive. In this paper, a novel road crack detection algorithm which is based on deep learning and adaptive image segmentation is proposed. Firstly, a deep convolutional neural network is trained to determine whether an image contains cracks or not. The images containing cracks are then smoothed using bilateral filtering, which greatly minimizes the number of noisy pixels. Finally, cracks are extracted from the road surface using an adaptive thresholding method. The experimental results illustrate that our network can classify images with an accuracy of 99.92%, and the cracks can be successfully extracted from the images using our proposed thresholding algorithm.

Multiple lidars are used on mobile vehicles for rendering a broad view to enhance the performance of perception systems. However, precise calibration of multiple lidars is challenging since the feature correspondences in scan points are sparse for providing enough constraints. To address this problem, existing methods require fixed calibration targets in scenes or rely exclusively on additional sensors. In this paper, we present a novel method that enables automatic lidar calibration without these restrictions. Three linearly independent planar surfaces appearing in surroundings is utilized to find correspondences. Two components are developed to ensure the extrinsic parameters to be found: a closed-form solver for initialization and an optimizer for refinement by minimizing a nonlinear cost function. Simulation and experimental results demonstrate the accuracy of our calibration approach with the rotation and translation errors smaller than 0.05rad and 0.1m respectively.

Foreground moving object segmentation is a fundamental problem in many computer vision applications. As a solution for foreground segmentation, background modeling has been intensively studied over past years and many effective algorithms have been developed. However, accurate foreground segmentation is still a difficult problem. Currently, most of the algorithms work solely within the color space, in which the segmentation performance is prone to be degraded by a multitude of challenges, such as illumination changes, shadows, automatic camera adjustments, and color camouflage. RGB-D cameras are active visual sensors that provide depth measurements along with color images. We present in this paper an innovative background modeling method by using both the color and depth information from an RGB-D camera. The proposed method is evaluated using a public RGB-D data set. Various experiments confirm that our method is able to achieve superior performance compared with existing well-known methods.

Focusing on generalized sensor combinations, this paper deals with the attitude estimation problem using a linear complementary filter (CF). The quaternion observation model is obtained via a gradient descent algorithm. An additive measurement model is then established according to derived results. The filter is named as the generalized CF where the observation model is simplified as a linear one that is quite different from previous-reported brute-force nonlinear results. Moreover, we prove that representative derivative-based optimization algorithms are essentially equivalent to each other. Derivations are given to establish the state model based on the quaternion kinematic equation. The proposed algorithm is validated under several experimental conditions involving the free-living environment, harsh external field disturbances, and aerial flight test aided by robotic vision. Using the specially designed experimental devices, data acquisition and algorithm computations are performed to give comparisons on accuracy, robustness, time-consumption, and so on with representative methods. The results show that not only the proposed filter can give fast, accurate, and stable estimates in terms of various sensor combinations but also produces robust attitude estimation in the scenario of harsh situations, e.g., irregular magnetic distortion. Note to Practitioners-Multisensor attitude estimation is a crucial technique in robotic devices. Many existing methods focus on the orientation fusion of specific sensor combinations. In this paper, we make the problem more concise. The results given in this paper are very general and can significantly decrease the space consumption and computation burden without losing the original estimation accuracy. Such performance will be of benefit to robotic platforms requiring flexible and easy-to-tune attitude estimation in the future.

We present an approach for mobile robots to learn to navigate in dynamic environments with pedestrians via raw depth inputs, in a socially compliant manner. To achieve this, we adopt a generative adversarial imitation learning (GAIL) strategy, which improves upon a pre-trained behavior cloning policy. Our approach overcomes the disadvantages of previous methods, as they heavily depend on the full knowledge of the location and velocity information of nearby pedestrians, which not only requires specific sensors, but also the extraction of such state information from raw sensory input could consume much computation time. In this paper, our proposed GAIL-based model performs directly on raw depth inputs and plans in real-time. Experiments show that our GAIL-based approach greatly improves the safety and efficiency of the behavior of mobile robots from pure behavior cloning. The real-world deployment also shows that our method is capable of guiding autonomous vehicles to navigate in a socially compliant manner directly through raw depth inputs. In addition, we release a simulation plugin for modeling pedestrian behaviors based on the social force model.

Indoor localization for pedestrians has gained increasing popularity among the rich body of literature for the last decade. In this paper, a low-cost indoor mapping and localization solution is proposed using the opportunistic signals from ambient indoor environments with a smartphone. It is composed of GraphSLAM-based offline mapping and Bayesian filtering-based online localization using generated signal maps. The GraphSLAM front-end is constructed by motion constraints from pedestrian dead-reckoning (PDR), loop-closure constraints identified by magnetic sequence matching with WiFi signal similarity validation, and observation constraints from opportunistic magnetic headings after error rejection. Globally consistent trajectories are created by graph optimization, after which signal maps (e.g., WiFi, magnetic fields, lights) are generated by Gaussian Processes Regression (GPR) for later localization. We propose to use the pseudo-wall constraints from the GPR variance map of magnetic fields and the lights measurements as observations for particle filtering. The proposed method is evaluated on several datasets collected from both the in-compass office buildings and outside public areas. Real-time localization is demonstrated on a smartphone in an office building covering 2000 square meters with the 50- and 90-percentile accuracies being 2.30 m and 3.41 m, respectively.

Indoor localization is critical to many location-aware applications, however, a low-cost solution with guaranteed accuracies has not yet come. Visible Light Communication (VLC-) based localization techniques are very promising to fill this gap. In this paper, we propose Plugo, a novel VLC system with random multiple access towards low-cost indoor localization. Compared to conventional RF-based approaches that rely on dedicated wireless access points as location beacons, the proposed system has the potential to deliver better accuracies with reduced cost. Specifically, we build a handful of compact VLC-compatible LED bulbs out of low-cost offthe-shelf components (around $10 total cost for each assembly) and recover VLC signals using a cheap photodiode receiver. The basic framed slotted Additive Links On-line Hawaii Area (ALOHA) is exploited to achieve random multiple access over the shared optical medium. We show its effectiveness in beacon broadcasting by experiments, and further, demonstrate a preliminary localization result with sound accuracy by using fingerprinting-based methods in a customized testbed.

The development of sharing technology makes it possible for expensive lower limb exoskeleton robots to be extensively employed. However, due to the uniqueness of gait pattern, it is challenging for lower limb exoskeleton robot to adapt to different wearers' gait patterns. Studies have shown that the gait pattern is affected by many physical factors. This paper proposes an individualized gait pattern generation (IGPG) method for sharing lower limb exoskeleton (SLEX) robot. First, the gait sequences are parameterized to extract gait features. Then, the Gaussian process regression with automatic relevance determination is used to establish the mapping relationships between the body parameters and the gait features, and the weights of each body parameters on gait pattern are also given. The gait features of an unknown subject can be predicted based on the training set. Finally, the individualized gait pattern is reconstructed by autoencoder neural network and scaling process based on predicted gait features. The experimental results show that the gait pattern predicted by IGPG is very similar to the subject's actual trajectory and has been successfully applied on the SLEX robot. With the help of sharing technology, the training set will be increased, and the prediction accuracy of individualized gait pattern will also be improved.

This paper studies the problem of image-based leader-follower formation control for mobile robots, where the controller is designed independently of the leader's motion. An adaptive control scheme, which is suitable for both omnidirectional and perspective cameras, is proposed. The proposed approach avoids the need for accurate calibration of the extrinsic parameters of the omnidirectional camera as well as the intrinsic and extrinsic parameters of perspective camera. Additionally, the coefficients of the plane where the feature point moves relative to the camera frame can be uncertain. These uncertain constant parameters are estimated using an adaptive estimator. Uniform Semi-global Practical Asymptotic Stability (USPAS) of the system is shown using the Lyapunov approach. Experimental results are presented to demonstrate the effectiveness of the proposed control scheme.

We present a learning-based mapless motion planner by taking the sparse 10-dimensional range findings and the target position with respect to the mobile robot coordinate frame as input and the continuous steering commands as output. Traditional motion planners for mobile ground robots with a laser range sensor mostly depend on the obstacle map of the navigation environment where both the highly precise laser sensor and the obstacle map building work of the environment are indispensable. We show that, through an asynchronous deep reinforcement learning method, a mapless motion planner can be trained end-to-end without any manually designed features and prior demonstrations. The trained planner can be directly applied in unseen virtual and real environments. The experiments show that the proposed mapless motion planner can navigate the nonholonomic mobile robot to the desired targets without colliding with any obstacles.

Cloud computing enables users to share computing resources on-demand. The cloud computing framework cannot be directly mapped to cloud robotic systems with ad hoc networks since cloud robotic systems have additional constraints such as limited bandwidth and dynamic structure. However, most multirobotic applications with cooperative control adopt this decentralized approach to avoid a single point of failure. Robots need to continuously update intensive data to execute tasks in a coordinated manner, which implies real-time requirements. Thus, a resource allocation strategy is required, especially in such resource-constrained environments. This paper proposes a hierarchical auction-based mechanism, namely link quality matrix (LQM) auction, which is suitable for ad hoc networks by introducing a link quality indicator. The proposed algorithm produces a fast and robust method that is accurate and scalable. It reduces both global communication and unnecessary repeated computation. The proposed method is designed for firm real-time resource retrieval for physical multirobot systems. A joint surveillance scenario empirically validates the proposed mechanism by assessing several practical metrics. The results show that the proposed LQM auction outperforms state-of-the-art algorithms for resource allocation.

Obstacle avoidance is the core problem for mobile robots. Its objective is to allow mobile robots to explore an unknown environment without colliding into other objects. It is the basis for various tasks, e.g. surveillance and rescue, etc. Previous approaches mainly focused on geometric models (such as constructing local cost-maps) which could be regarded as low-level intelligence without any cognitive process. Recently, deep learning has made great breakthroughs in computer vision, especially for recognition and cognitive tasks. It takes advantage of the hierarchical models inspired by human brain structures. However, it is a fact that deep learning, up till now, has seldom been used for controlling and decision making. Inspired by the advantages of deep learning, we take indoor obstacle avoidance as example to show the effectiveness of a hierarchical structure that fuses a convolutional neural network (CNN) with a decision process. It is a highly compact network structure that takes raw depth images as input, and generates control commands as network output, by which a model-less obstacle avoidance behavior is achieved. We test our approach in real-world indoor environments. The new findings and results are reported at the end of the paper.

This paper deals with the path-planning problem for mobile wheeled- or tracked-robot which drive in 2.5-D environments, where the traversable surface is usually considered as a 2-D-manifold embedded in a 3-D ambient space. Specially, we aim at solving the 2.5-D navigation problem using raw point cloud as input. The proposed method is independent of traditional surface parametrization or reconstruction methods, such as a meshing process, which generally has high-computational complexity. Instead, we utilize the output of 3-D tensor voting framework on the raw point clouds. The computation of tensor voting is accelerated by optimized implementation on graphics computation unit. Based on the tensor voting results, a novel local Riemannian metric is defined using the saliency components, which helps the modeling of the latent traversable surface. Using the proposed metric, we prove that the geodesic in the 3-D tensor space leads to rational path-planning results by experiments. Compared to traditional methods, the results reveal the advantages of the proposed method in terms of smoothing the robot maneuver while considering the minimum travel distance.

Indoor localization is a fundamental capability for service robots and indoor applications on mobile devices. To realize that, the cost and performance are of great concern. In this paper, we introduce a lightweight signal encoding and decomposition method for a low-cost and low-power Visible Light Communication (VLC)-based indoor localization system. Firstly, a Gold-sequence-based tiny-length code selection method is introduced for light encoding. Then a correlation-based asynchronous blind light-signal decomposition method is developed for the decomposition of the lights mixed with modulated light sources. It is able to decompose the mixed light-signal package in real-time. The average decomposition time-cost for each frame is 20 ms. By using the decomposition results, the localization system achieves accuracy at 0.56 m. These features outperform other existing low-cost indoor localization approaches, such as WiFiSLAM.

For mobile robots and position-based services, such as healthcare service, precise localization is the most fundamental capability while low-cost localization solutions are with increasing need and potentially have a wide market. A low-cost localization solution based on a novel Visible Light Communication (VLC) system for indoor environments is proposed in this paper. A number of modulated LED lights are used as beacons to aid indoor localization additional to illumination. A Gaussian Process(GP) is used to model the intensity distributions of the light sources. A Bayesian localization framework is constructed using the results of the GP, leading to precise localization. Path-planning is hereby feasible by only using the GP variance field, rather than using a metric map. Dijkstra's algorithm-based path-planner is adopted to cope with the practical situations. We demonstrate our localization system by real-time experiments performed on a tablet PC in an indoor environment.

Cloud technology elevates the potential of robotics with which robots possessing various capabilities and resources may share data and combine new skills through cooperation. With multiple robots, a cloud robotic system enables intensive and complicated tasks to be carried out in an optimal and cooperative manner. Multisensor data retrieval (MSDR) is one of the key fundamental tasks to share the resources. Having attracted wide attention, MSDR is facing severe technical challenges. For example, MSDR is particularly difficult when cloud cluster hosts accommodate unpredictable data requests triggered by multiple robots operating in parallel. In these cases, near real-time responses are essential while addressing the problem of the synchronization of multisensor data simultaneously. In this paper, we present a framework targeting near real-time MSDR, which grants asynchronous access to the cloud from the robots. We propose a market-based management strategy for efficient data retrieval. It is validated by assessing several quality-of-service (QoS) criteria, with emphasis on facilitating data retrieval in near real-time. Experimental results indicate that the MSDR framework is able to achieve excellent performance under the proposed management strategy in typical cloud robotic scenarios.

Mobile wheeled- or tracked-robots drive in 2.5-dimensional (2.5D) environments, where the traversable surface can be considered as a 2D-manifold embedded in a three-dimensional (3D) ambient space. In this work, we aim at solving the 2.5D navigation problem solely on point-cloud. The proposed method is independent of traditional surface parametrization or reconstruction methods, such as a meshing process, which generally has high computational complexity. Instead, we utilize the output of 3D tensor voting framework (TVF) using raw point-clouds. A novel local Riemannian metric is defined based on the saliency components of TVF, which helps the modeling of the latent traversable surface. Using this metric, we prove that the geodesic in the 3D tensor space leads to rational path-planning results. Compared to traditional methods, the results reveal the advantages of the proposed method in terms of facilitating the robot maneuver with minimum movement.

In order to share information in the cloud for multi-robot systems, efficient data transmission is essential for real-time operations such as coordinated robotic missions. As a limited resource, bandwidth is ubiquitously required by applications among physical multi-robot systems. In this paper, we proposed a hierarchical auction-based mechanism, namely LQM (Link Quality Matrix)-auction. It consists of multiple procedures, such as hierarchical auction, proxy scheduling. Note that the proposed method is designed for real-time resource retrieval for physical multi-robot systems, instead of simulated virtual agents. We validate the proposed mechanism through real-time experiments. The results show that LQM-auction is suitable for scheduling a group of robots, leading to optimized performance for resource retrieval.

Chosen the winter crop yield data from agro-meteorological sites and ISDI results during 2000-2009 in Mid-eastern China, the corresponding relationship between drought and winter crop yield was analyzed using confusion matrix and Kappa coefficient. The results shows that the high or low values of ISDI can reflect the crop yield reduction condition caused by the drought or flood; the normal values of ISDI can reflect the crop yield increasing and fluctuation in normal condition. ISDI monitored drought condition has obvious corresponding relationship with crop yield change. This also proved that ISDI can accurate monitor the drought events occurred during winter crop growing season.

Indoor localization is the fundamental capability for indoor service robots and indoor applications on mobile devices. To realize that, the cost of sensors is of great concern. In order to decode the signal carried out by the LED beacons, we propose two reliable solutions using common sensors available on consumer electronic devices. Firstly, we introduce a dedicated analog sensor, which can be directly connected to the microphone input of a computer or a smart phone. It decodes both the signal pattern and signal strength of a beacon. Secondly, we utilize rolling-shutter cameras to decode the signal pattern, providing potential solutions to the localization of hand-held devices with cameras. In contrast to existing widely-applied indoor localization approaches, like vision-based and laser-based methods, our approach reveals its advantages as low-cost, globally consistent and it retains the potential applications using Visible Light Communication(VLC). We also study the characteristics of the proposed solutions under typical indoor conditions by experiments.

Due to the characteristic of the synthetic aperture radar (SAR) image's sensitivity to the target aspect angles, a multiple sparse representation (MSR) method for SAR target configuration recognition is proposed. Making use of the prior information, dictionaries are constructed by using the samples of each configuration to better capture the detail information of the SAR images. The advantage of MSR over sparse representation for detail feature extraction is analyzed. Moreover, to achieve better recognition results, the Dempster–Shafer fusion is carried out to get comprehensive description of the target for configuration recognition. Two mass functions are constructed based on MSR and the sample statistical property. The combined mass function has the advantages of both the detail and global features of the target. Experiments on the moving and stationary target acquisition and recognition data sets validate the effectiveness and superiority of the proposed algorithm.

Scene recognition problems for mobile robots have been extensively studied. This is important for tasks such as visual topological mapping. Usually, sophisticated key-point-based descriptors are used, which can be computationally expensive. In this paper, we describe a lightweight novel scene recognition method using an adaptive descriptor, which is based on color features and geometric information that are extracted from an uncalibrated omnidirectional camera. The proposed method enables a mobile robot to perform online registration of new scenes onto a topological representation automatically and solve the localization problem to topological regions simultaneously, all in real time. We adopt a Dirichlet process mixture model (DPMM) to describe the online inference process. It is based on an approximation of conditional probabilities of the new measurements given incrementally estimated reference models. It enables online inference speeds of up to 50 Hz for a normal CPU. We compare it with state-of-the-art key-point descriptors and show the advantage of the proposed algorithm in terms of performance and computational efficiency. A real-world experiment is carried out with a mobile robot equipped with an omnidirectional camera. Finally, we show the results on extended datasets.

Omnidirectional cameras are widely used for robotic applications in structured environments. However, because of the distorted field of view (FOV), it is hard to describe the primitive features extracted from them robustly. In this paper, we tackle the problem by using Histogram of Gradient (HoG) statistics for the regions of interest (ROI) in the neighborhood of major vertical lines extracted from the panoramic image. As a validation, we compare the proposed algorithm with state-of-the-art based on two widely used data-sets, leading to evidently better performance. We also introduce a scene reconstruction scenario using the proposed descriptor based on 1D Trifocal Tensor framework. The comparative results show the competence of the descriptor.

The constructed Integrated Surface Drought Index (ISDI) has been successfully used for drought monitoring in Mid-eastern China from 2000-2009. In this paper, the precipitation data observed at meteorological stations and soil moisture data (at 10cm, 20cm and 50cm depth) were used to further evaluate the ISDI characteristics for drought monitoring. Correlation analysis indicates that there is clear relationship between ISDI and soil moisture. ISDI has higher correlation with root zone soil moisture than surface and deeper soil moisture, which is crucial indicator for agricultural drought. Temporal trend analysis also shows that ISDI has consistent trend relationship with precipitation and soil moisture. Moreover, ISDI can get comparable results in different regions. The evaluation results provide further evidence of ISDI effectiveness for drought monitoring in Mid-eastern China.

One milestone for autonomous mobile robotics is to endow robots with the capability to compute the plans and motor commands necessary to reach a defined goal position. For indoor or car-like robots moving on flat terrain, this problem is well mastered and open-source software can be deployed to such robots. However, for many applications such as search and rescue, ground robots must handle three-dimensional terrain. In this article, we present a system that is able to plan and execute a path in a complex environment starting from noisy sensor input. In order to cope with the complexity of a high-dimensional configuration space, we separate position and configuration planning. We demonstrate our system on a search and rescue robot with flippers by climbing up and down a difficult curved staircase.

Feature extraction is the key step of synthetic aperture radar (SAR) target configuration recognition. A statistical model embedding the locality preserving property is presented to extract the maximum amount of desired information from the data, which is of crucial help to recognition. The noise, or error, of the SAR image samples is described by a Gaussian mixture distribution, and the locality preserving property is embedded into the statistical model to focus on the problem of configuration recognition. Along with the extraction of the information of interest through the use of the statistical model, also, the preservation of the local structure of the data set is achieved. Parameter estimation is implemented through the expectation–maximization algorithm. Experimental results on the Moving and Stationary Target Acquisition and Recognition data set validate the effectiveness of the proposed method. SAR target configuration recognition is realized with satisfactory accuracy.

Visual homing enables a mobile robot to move to a reference position using only visual information. The approaches that we present in this paper utilize matched image key points (e.g., scale-invariant feature transform) that are extracted from an omnidirectional camera as inputs. First, we propose three visual homing methods that are based on feature scale, bearing, and the combination of both, under an image-based visual servoing framework. Second, considering computational cost, we propose a simplified homing method which takes an advantage of the scale information of key-point features to compute control commands. The observability and controllability of the algorithm are proved. An outlier rejection algorithm is also introduced and evaluated. The results of all these methods are compared both in simulations and experiments. We report the performance of all related methods on a series of commonly cited indoor datasets, showing the advantages of the proposed method. Furthermore, they are tested on a compact dataset of omnidirectional panoramic images, which is captured under dynamic conditions with ground truth for future research and comparison.

Visual Homing is the process by which a mobile robot moves to a Home position using only information extracted from visual data. The approach we present in this paper uses image keypoints (e.g. SIFT) extracted from omnidirectional images and matches the current set of keypoints with the set recorded at the Home location. In this paper, we first formulate three different visual homing problems using uncalibrated omnidirectional camera within the Image Based Visual Servoing (IBVS) framework; then we propose a novel simplified homing approach, which is inspired by IBVS, based only on the scale information of the SIFT features, with its computational cost linear to the number of features. This paper reports on the application of our method on a commonly cited indoor database where it outperforms other approaches. We also briefly present results on a real robot and allude on the integration into a topological navigation framework.

Topological mapping and scene recognition problems are still challenging, especially for online realtime vision-based applications. We develop a hierarchical probabilistic model to tackle them using color information. This work is stimulated by our previous work [1] which defined a lightweight descriptor using color and geometry information from segmented panoramic images. Our novel model uses a Dirichlet Process Mixture Model to combine color and geometry features which are extracted from omnidirectional images. The inference of the model is based on an approximation of conditional probabilities of observations given estimated models. It allows online inference of the mixture model in real-time (at 50Hz), which outperforms other existing approaches. A real experiment is carried out on a mobile robot equipped with an omnidirectional camera. The results show the competence against the state-of-art.

Using data-mining technology, this paper established a new method, named the Integrated Surface Drought Index (ISDI). ISDI integrates traditional meteorological data, remotely sensed indices, and biophysical data, and attempt to describe drought from a more comprehensive perspective. The evaluation results indicated that the construction models for three phases of growth season have very high regression accuracy. The drought condition can be predicted using the independent variables. The practical application of ISDI in mid-eastern China also demonstrated that ISDI has good application accuracy in mid-eastern China. ISDI results were corresponding to the disaster observation records of agro-meteorological sites. It can be potentially extended to nationwide near real time drought monitoring.

In this paper, we present a novel semi-supervised clustering approach based on Markov process. It deals with data which include abundant local constraints. We apply the designed model to a topological region extraction problem, where topological segmentation is constructed based on sparse human inputs (potentially provided by human experts). The model considers human indications as seeds for topological regions, i.e. the partially labeled data. It results in a regional topological segmentation of connected free space.

Visual homing has been widely studied in the past decade. It enables a mobile robot to move to a Home position using only information extracted from visual data. However, integration of homing algorithms into real applications is not widely studied and poses a number of significant challenges. Failures often occur due to moving people within the scene and variations in illumination. We present a novel integrated indoor topological navigation framework, which combines odometry motion with visual homing algorithms. We show robustness to scene variation and real-time performance through a series of tests conducted in four real apartments and several typical indoor scenes, including doorways, offices etc.

The triplet Markov field (TMF) model recently proposed is suitable for dealing with nonstationary synthetic aperture radar (SAR) image segmentation. In this letter, we propose a multiscale TMF model in wavelet domain, named as the wavelet-domain TMF (WTMF) model. In the WTMF model, a multiscale causal WTMF energy function is constructed to capture the intra- and interscale dependences in random fields$(X, U)$. Moreover, multiscale likelihoods of the WTMF model are derived based on a wavelet hidden Markov tree to capture the statistical properties of wavelet coefficients. The proposed model can integrate the global and local information in terms of spatial configuration and image features in a more complete manner. The coarser scale information is utilized to guide the finer scale segmentation, and the coarse-to-fine causal interactions are considered using a Markov chain. Experimental results prove that the proposed model can segment SAR images better than several models previously proposed.

When people communicate with robots, the most intuitive mean is by naming the different regions in the environment. The capability that robots are able to identify different regions highly depends on the unsupervised topological segmentation results. This paper addresses the problem of segmenting a metric map into regions. Nowadays many researches in this direction develop approaches based on spectral clustering. However there are inherent drawbacks of spectral clustering algorithms. In this paper, we first discuss these drawbacks using several testing results; then we propose our approach based on information theory which uses Chow-Liu tree to segment the composed graph according to the weight differences. The results show that our method provides more flexible and faster results in the sense of facilitating semantic mapping or further applications.

The increasing number of ICP variants leads to an explosion of algorithms and parameters. This renders difficult the selection of the appropriate combination for a given application. In this paper, we propose a state-of-the-art, modular, and efficient implementation of an ICP library. We took advantage of the recent availability of fast depth cameras to demonstrate one application example: a 3D pose tracker running at 30 Hz. For this application, we show the modularity of our ICP library by optimizing the use of lean and simple descriptors in order to ease the matching of 3D point clouds. This tracker is then evaluated using datasets recorded along a ground truth of millimeter accuracy. We provide both source code and datasets to the community in order to accelerate further comparisons in this field.

Homing is one of the fundamental functions for both the mobile robot and the flying robot. Furthermore, homing can be introduced into a topological navigation system by cyclically setting Home positions at the keypoints/nodes in a topological map. In this work, we describe a bearing-only homing method based on only few matching keypoints to grant the mobile robot the homing ability. Our method considers the homing problem as a visual servoing problem in 2D plane and even in 3D space, using an omnidirectional camera as the visual sensor. It doesn't require the distance information to the reference feature points. The proof of the convergence for the algorithm is also given. The simulation results confirm the feasibility and robustness of our method.

Many methods are generally recognized as good indicators of drought condition which are based on a transformation of normalized difference vegetation index (NDVI) data i.e. the Vegetation Condition Index (VCI) and phenology metric called Percent of Average Seasonal Greenness (PASG). Because of the immense spatial and temporal variability exhibited by each drought event, the relationship between vegetation condition and precipitation, however, is complex and has not been adequately examined with remote sensing data and meteorological drought index. The objective of this paper is assessing the drought monitoring characteristic of VCI and PASG in crop growing season which are based on 10 years (1999-2008) time-series SPOT VGT-S NDVI sequence. Multi-scale SPI (1-month, 2-month, 3-month, 6-month, 9-month and 12-month) was calculated to detect occurrence of drought using the ten-day precipitation (1960-2008) data set. By analyzing the correlation coefficients between VCI, PASG and multi-scale SPIs, three periods i.e. the time from the last ten-day of May to the middle ten-day of June, the mid-to-late August and the mid-to late September were found to be the best stages using VCI and PASG for drought monitoring. VCI, PASG and 1-month SPI-scale have a better correlation on the whole. VCI presents the better ability of drought monitoring, but PASG maintains a smooth effect on drought monitoring during the crop growing season. Besides, both of them have a very good complementary for each other.

Mobile robots rely on their ability of scene recognition to build a topological map of the environment and perform location-related tasks. In this paper, we describe a novel lightweight scene recognition method using an adaptive descriptor which is based on color features and geometric information for omnidirectional vision. Our method enables the robot to add nodes to a topological map automatically and solve the localization problem of mobile robot in realtime. The descriptor of a scene is extracted in the YUV color space and its dimension is adaptive depending on the segmentation result of the panoramic image. Furthermore, the descriptor is invariant to rotation and slight changes of illumination. The robustness of the scene matching and recognition is tested through real experiments in a dynamic indoor environment. The experiment is carried out on a mobile robot equipped with an omnidirectional camera. In our tests, the average processing time is 30 ms for each frame including feature extraction, matching, and the adding of new nodes.