3D object detection is one of the fundamental perception tasks for autonomous vehicles. Fulfilling such a task with a 4D millimeter-wave radar is very attractive since the sensor is able to acquire 3D point clouds similar to Lidar while maintaining robust measurements under adverse weather. However, due to the high sparsity and noise associated with the radar point clouds, the performance of the existing methods is still much lower than expected. In this paper, we propose a novel Semi-supervised Cross-modality Knowledge Distillation (SCKD) method for 4D radar-based 3D object detection. It characterizes the capability of learning the feature from a Lidar-radar-fused teacher network with semi-supervised distillation. We first propose an adaptive fusion module in the teacher network to boost its performance. Then, two feature distillation modules are designed to facilitate the cross-modality knowledge transfer. Finally, a semi-supervised output distillation is proposed to increase the effectiveness and flexibility of the distillation framework. With the same network structure, our radar-only student trained by SCKD boosts the mAP by 10.38% over the baseline and outperforms the state-of-the-art works on the VoD dataset. The experiment on ZJUODset also shows 5.12% mAP improvements on the moderate difficulty level over the baseline when extra unlabeled data are available.

Federated Learning (FL) has gained widespread attention because it doesn’t require users to share their private data. However, since sensitive information can still be inferred from the uploaded models by users, Differential Privacy (DP) is often employed to safeguard these submitted models. Existing work mainly focused on uniformly allocating privacy budget to each layer of submitted models, which can lead to performance degradation, such as slower convergence rates and reduced generalization capabilities of the global model. To address this issue, this paper proposes a novel Differential Privacy Federated Learning framework, called Personalized Layer Privacy Protection Federated Learning Based on Mutual Information (MIPP-FL). In MIPP-FL, we first calculate the entropies of each layer’s weights of local models to establish a distribution of weights for each training round. Based on the entropies, the mutual information between current and last training rounds can be further obtained. Finally, for a given local model, we dynamically allocate the privacy budget to each layer by considering the calculated mutual information. Theoretically, we have demonstrated that the proposed MIPP-FL framework ensures strict privacy guarantee. Moreover, extensive experiments have shown that our proposed method can improve accuracy and achieve a faster convergence rate than the existing method that allocated the same privacy budgets uniformly to all layers of a local model.

Federated Learning (FL) has attracted significant interest due to its ability to train models without necessitating the sharing of clients’ private data. Nevertheless, as sensitive information may still be inferred from the models uploaded by clients, Differential Privacy (DP) is commonly integrated to enhance the protection of these shared models. Existing works primarily focus on uniformly or non-uniformly allocating privacy budgets across each layer of submitted models, which may lead to performance degradation, such as high computation overhead, slow convergence, and reduced generalization capability of the global model. To address this issue, this paper proposes a Digital Twin Enabled Personalized Client Selection and Privacy Protection FL under Data Heterogeneity (i.e., DTFedCP). In DTFedCP, we first calculate the entropy of each layer’s weights of local models to establish the weight distribution for each training round. Based on the computed entropy, we further derive the mutual information between the current and previous training rounds. Finally, the privacy budget is dynamically allocated to each layer by utilizing the calculated mutual information and measuring client heterogeneity through the Kullback-Leibler(KL) divergence. Theoretically, we have demonstrated that the pro-posed DTFedCP provides rigorous privacy guarantees. Furthermore, extensive experiments under various Dirichlet-based heterogeneous partitions indicate that our proposed method achieves higher accuracy and faster convergence compared to existing methods that uniformly allocate identical privacy budgets across all layers of local models.

Harmonic diffractive optical elements (HDOEs) are characterized by their lightweight and compact size, making them promising candidates for applying in future large-aperture space optical imaging systems. However, its lower focusing efficiency and unavoidable manufacturing errors can result in degraded and blurred imaging. To effectively improve the imaging quality of HDOE optical systems, this study proposes an image super-resolution (SR) method based on plug-and-play (PnP) technology, referred to as HDSR. Specifically, the study first establishes the objective function for image SR and then introduces a Poissonian-Gaussian noise model to describe the noise in HDOE optical imaging systems. Based on this, a denoiser based on a convolutional neural network (CNN) is trained and used as the prior term in the optimization function. In addition, the study proposes a learning-based parameter auto-estimation and updating mechanism to reduce the complexity of manually tuning iterative parameters in the PnP technology. In the experimental section, the study explores and verifies the role and importance of the adopted noise model and parameter estimation mechanism. The results show that the proposed HDSR method significantly enhances the imaging quality of the HDOE optical system. In outdoor scenes, the natural image quality evaluator (NIQE) metric average value after SR using this method is 9.28, which is a 49.62% improvement compared to the bicubic method.

In autonomous driving, 3D LiDAR plays a crucial role in understanding the vehicle’s surroundings. However, the newly emerged, unannotated objects presents few-shot learning problem for semantic segmentation. This paper addresses the limitations of current few-shot semantic segmentation by exploiting the temporal continuity of LiDAR data. Employing a tracking model to generate pseudo-ground-truths from a sequence of LiDAR frames, our method significantly augments the dataset, enhancing the model’s ability to learn on novel classes. However, this approach introduces a data imbalance biased to novel data that presents a new challenge of catastrophic forgetting. To mitigate this, we incorporate LoRA, a technique that reduces the number of trainable parameters, thereby preserving the model’s performance on base classes while improving its adaptability to novel classes. This work represents a significant step forward in few-shot 3D LiDAR semantic segmentation for autonomous driving. Our code is available at https://github.com/BowmanChow/Track-no-forgetting.

As a fundamental task for intelligent robots, visual SLAM has made significant progress in recent years. However, robust SLAM in weakly textured environments remains a challenging task. In this paper, we present a novel visual Robust SLAM for Weak-Textured environments (RWT-SLAM) to address this problem. Unlike existing methods that use detector-based deep networks for interest point detection, we propose extracting distinctive features from a detector-free based network, namely LoFTR, to avoid the difficulty of manual annotations of feature points in weakly textured images. We generate multi-level feature vectors from LoFTR to form dense descriptors for each pixel in the input image. A keypoint localization component is then proposed to measure the saliency of the descriptors and select the distinctive pixels as keypoints. We integrate this new keypoint into the popular ORB-SLAM framework and compare it with the state-of-the-art methods. Extensive experiments on popular TUM RGB-D, OpenLORIS-Scene, as well as our own dataset are carried out. The results demonstrate the superior performance of our method in weakly textured environments.

Large visual models have recently demonstrated their promising performance on zero-shot transfer. However, so far, none of the existing methods explicitly possess the ability to perform zero-shot transfer on parking-slot detection, which results in current deep-learning based methods relying on training datasets, and methods based on traditional computer vision exhibiting poor robustness. In this paper, we propose a large visual model-based parking-slot detection method, which utilizes a large visual model (segment anything) to segment an around-view image and infer parking-slots by analyzing the relationship of marking-points in masks. In addition, we classify real-world parking-slots into two categories, line-based and area-based. The proposed method employs a two-stage approach which has a manually designed post-processing step without training. Multiple experiments have been carried out on public benchmarks, and our method demonstrates the capability for zero-shot transfer. The code will be released at https://github.com/Zhai0123/SAM-PS.

The weight of traditional space optical imaging systems often increases nearly cubically with the increase in aperture size. To overcome this limitation, this study proposes the use of a lightweight harmonic diffractive optical imaging system combined with deep denoiser prior image restoration, to achieve high-resolution, lightweight remote sensing.This research first designed a novel 150-order harmonic diffractive optical element (H-DOE) featuring a 40-mm aperture size and a 320-mm focal length, which is equipped with seven annular zones covering a broad spectral band (500-800 nm). Its slim structure significantly reduces weight and volume, thereby lowering its launch costs. Furthermore, to address the blurring issues encountered in H-DOE imaging tasks and attain enhanced image resolution, this study incorporates an advanced image restoration technique. This technique employs a deep denoiser as a prior module, which is embedded into a model-based image restoration optimization framework. The newly trained deep denoiser utilizes a U-Net architecture integrating a transformer and residual structures and is adept at handling complex noise during the optical imaging process. Experimental results demonstrate that the performance of the proposed image restoration strategy based on a deep denoiser surpasses that of the existing technologies, elevating the resolvable frequency of the modulation transfer function (MTF) of an H-DOE imaging system from 40.58 lp/mm to 98.55 lp/mm, an enhancement of 142.9%. This significant image quality improvement showcases its vast potential for use in future high-resolution, lightweight remote sensing applications.

Semantic segmentation of LiDAR point clouds can provide assistance for precise perception in autonomous driving, but traditional segmentation methods face challenges such as unbalanced class distribution and insufficient labeling. Generalized few-shot learning has been researched on image data, but these methods are difficult to apply directly to LiDAR point clouds. To tackle these challenges, we propose a generalized few-shot semantic segmentation method based on LiDAR point cloud data, enabling us to predict base and novel classes simultaneously. To improve the performance with limited novel class samples, we integrate semantic vectors and leverage the intrinsic relationship between base and novel class vectors to facilitate learning. We conduct comprehensive comparisons with other methods on the SemanticKITTI and constantly surpass them with higher mIoU, demonstrating the effectiveness of our method.

Off-road semantic segmentation with fine-grained labels is necessary for autonomous vehicles to understand driving scenes, as the coarse-grained road detection cannot satisfy off-road vehicles with various mechanical properties. Pixel-wise annotation of fine-grained labels in off-road scenes is very hard because a large part of the pixels could suffer from severe semantic ambiguity. Furthermore, semantic properties of off-road scenes can be very changeable due to various precipitations, temperature, defoliation, etc. To address these challenges, this research proposes an active and contrastive learning-based method. A few image patches are annotated mainly to distinguish semantic differences rather than semantic categories, which can greatly reduce the burden of manual annotation. A feature representation is learnt using the contrastive pairs of image patches, and semantic categories are adaptively modeled from the data. To actively adapt to new scenes, a risk evaluation method is developed to discover and select hard frames with high-risk predictions for supplementary labeling, to update the model efficiently. Extensive experiments and analyses are conducted on self-developed and public datasets. Experimental results demonstrate that fine-grained semantic segmentation can be learned with only dozens of weakly labeled frames, and the model can efficiently adapt across scenes by weak supervision, while achieving competitive performance with the typical fully supervised ones.

In recent years 3D object detection from LiDAR point clouds has made great progress thanks to the development of deep learning technologies. Although voxel or point based methods are popular in 3D object detection, they usually involve time-consuming operations such as 3D convolutions on voxels or ball query among points, making the resulting network inappropriate for time critical applications. On the other hand, 2D view-based methods feature high computing efficiency while usually obtaining inferior performance than the voxel or point based methods. In this work, we present a real-time view-based single stage 3D object detector, namely CVFNet to fulfill this task. To strengthen the cross-view feature learning under the condition of demanding efficiency, our framework extracts the features of different views and fuses them in an efficient progressive way. We first propose a novel Point-Range feature fusion module that deeply integrates point and range view features in multiple stages. Then, a special Slice Pillar is designed to well maintain the 3D geometry when transforming the obtained deep point-view features into bird's eye view. To better balance the ratio of samples, a sparse pillar detection head is presented to focus the detection on the nonempty grids. We conduct experiments on the popular KITTI and NuScenes benchmark, and state-of-the-art performances are achieved in terms of both accuracy and speed.

Road detection or traversability analysis has been a key technique for a mobile robot to traverse complex off-road scenes. The problem has been mainly formulated in early works as a binary classification one, e.g. associating pixels with road or non-road labels. Whereas understanding scenes with fine-grained labels are needed for off-road robots, as scenes are very diverse, and the various mechanical performance of off-road robots may lead to different definitions of safe regions to traverse. How to define and annotate fine-grained labels to achieve meaningful scene understanding for a robot to traverse off-road is still an open question. This research proposes a contrastive learning based method. With a set of human-annotated anchor patches, a feature representation is learned to discriminate regions with different traversability, a method of fine-grained semantic segmentation and mapping is subsequently developed for off-road scene understanding. Experiments are conducted on a dataset of three driving segments that represent very diverse off-road scenes. An anchor accuracy of 89.8% is achieved by evaluating the matching with human-annotated image patches in cross-scene validation. Examined by associated 3D LiDAR data, the fine-grained segments of visual images are demonstrated to have different levels of toughness and terrain elevation, which represents their semantical meaningfulness. The resultant maps contain both fine-grained labels and confidence values, providing rich information to support a robot traversing complex off-road scenes.

This work studies the semantic segmentation of 3D LiDAR data in dynamic scenes for autonomous driving applications. A system of semantic segmentation using 3D LiDAR data, including range image segmentation, sample generation, inter-frame data association, track-level annotation, and semi-supervised learning, is developed. To reduce the considerable requirement of fine annotations, a CNN-based classifier is trained by considering both supervised samples with manually labeled object classes and pairwise constraints, where a data sample is composed of a segment as the foreground and neighborhood points as the background. A special loss function is designed to account for both annotations and constraints, where the constraint data are encouraged to be assigned to the same semantic class. A dataset containing 1838 frames of LiDAR data, 39 934 pairwise constraints and 57 927 human annotations is developed. The performance of the method is examined extensively. The qualitative and quantitative experiments show that the combination of a few annotations and large amount of constraint data significantly enhances the effectiveness and scene adaptability, resulting in greater than 10% improvement.

We formulate camera and LiDAR fusion tracking as a sequential decision-making process. With our deep reinforcement learning framework, we try to optimize the tracking trajectory to be as accurate, smooth, and long as possible. In contrast to traditional fusion algorithms involving complex feature and strategy design and hyperparameters tuned for different scenarios, our fusion agent can learn the confidence of each input by tracking the results from raw observation in a data-driven fashion. Given the input states of different sensors, our approach chooses one input with a higher expected cumulative reward as the observation of a Kalman filter to iteratively predict the target position. The expected cumulative reward is estimated with a convolutional neural network, trained with a modified DQN algorithm, which takes inputs from both LiDAR and a camera. Through case studies and quantitative result evaluation on our dataset from the 4th Ring Road in Beijing, our algorithm is validated to achieve more accurate and robust tracking performance.

We propose a method for off-road drivable area extraction using 3D LiDAR data with the goal of autonomous driving application. A specific deep learning framework is designed to deal with the ambiguous area, which is one of the main challenges in the off-road environment. To reduce the considerable demand for human-annotated data for network training, we utilize the information from vast quantities of vehicle paths and auto-generated obstacle labels. Using these auto-generated annotations, the proposed network can be trained using weakly supervised or semi-supervised methods, which can achieve better performance with fewer human annotations. The experiments on our dataset illustrate the reasonability of our framework and the validity of our weakly and semi-supervised methods.

Localization is an essential technique for many robotic tasks such as mapping and navigation. Scan matching has been fused with other sensors to solve the problem at GPS restricted areas, where an accurate error model describing matching precision at various scenes is indispensable. We proposed an end-to-end method to learn a scene adaptive error model of LiDAR scan matching. A CNN (Convolutional Neural Network) is learnt to map from a LiDAR scan to an information matrix of the matching result, and a localization framework is proposed to fuse the results of LiDAR scan matching based on its error model. Experiments are conducted using both simulated and real world data, where the former is to validate the proposed method of its adaptability at various simple but typical scenes, while the later is to examine the method's practicability at real world environments. We demonstrate the performance of learning covariance error model, and examine the localization accuracy by comparing with other traditional methods. Efficiency of the proposed method is demonstrated.

This work studies a supervised learning method using 3D LiDAR data for autonomous driving applications. A system of semantic segmentation, including range image segmentation, sample generation, track-level annotation and supervised learning, is developed. The formation and content of a data sample is studied intensively to address the specialty of 3D LiDAR data, which can be represented at a Cartesian or a 2D polar coordinate system, and composed of a segment as the foreground and/or the neighborhood points as the background. A CNN-based classifier is trained to map a given sample to an object label. Qualitative and quantitative experiments show that the background information and multiple feature map fusion significantly improve the performance of the classifier.

In this paper, we propose a part-based particle filter for on-road vehicle tracking. The proposed model combines a part-based strategy with a particle filter. By introducing a hidden state representing the center position of the vehicle, particles corresponding to vehicle parts sharing the same motion can be collectively updated in an efficient manner. By using a pre-trained appearance and geometric model, the tracker can distinguish parts with rich information from invalid parts to make more precise predictions. Meanwhile, some prior knowledge about the motion patterns of vehicles in a well-structured on-road environment is learned and can be used to infer measurement and motion models to improve tracking performance and efficiency. Experiments were conducted using the real data collected in Beijing to examine the performance of the method in different situations in terms of both its advantages and challenges. The collected Beijing highway dataset for on-road vehicle tracking will be made publicly available. We compare our method with the state-of-the-art approaches. The results demonstrate that the proposed algorithm is able to handle occlusion and the aspect ratio changes in the on-road vehicle tracking problem.

The 2011 Intelligent Vehicle Future Challenge (11'IVFC) tested self-driving systems in real urban scenarios. The entry of Beijing Institute of Technology: BIT-III finished the 10-kilometer long track in 28 minutes without human operation and obeyed traffic regulations in most circumstances. This paper presented the design and implementation of BIT-III. As a universal self-driving system, BIT-III valued extensibility and featured modularized system architecture. For a better compatibility with diverse sensing devices, BIT-III classified perception to be either OGM (Occupancy Grid Map)-oriented or object-oriented based on the output mode. To work in environments with uncertainties, BIT-III gave first priority to safety and stability in driving, and realized them in the core-level components as the instinct of the system. Even in the unknown environment in the ll'IVFC, BIT-III was able to drive smoothly without crashes.

The first Chinese unmanned ground vehicles competition - The 2009 Future Challenge: Intelligent Vehicles and Beyond (FC'09) pushed China's unmanned vehicles out of laboratories and into application environments. In order to further promote the development of unmanned vehicle technologies, the test and evaluation system for unmanned vehicles needs to be studied. The design method of test environment is proposed in accordance with the definition and classification of test environment elements. Based on the multi-platform and multi-sensor, an omnidirectional video monitoring test system of unmanned vehicles is built. The fuzzy comprehensive evaluation method combined with AHP (analytic hierarchy process) is applied to the comprehensive evaluation of unmanned vehicles. The evaluation examples of unmanned vehicles show that the proposed evaluation system can quantitatively evaluate the overall technical performance and individual technical performance of unmanned vehicles.

The 2009 Future Challenge - Intelligent Vehicle and Beyond (FC'09) was held in Xi'an, China. Our intelligent vehicle named BIT participated in all competitions at this event. This paper describes BIT's system structure and its capabilities. BIT combines a global path planning method and local path planning to drive the vehicle to address the challenges posted by the unknown competition environment. A novel curve tracking strategy based on preview and curve bisector is developed for complex paths such as U-turn. For recognizing traffic lights, Haar feature and AdaBoost algorithm are used to train and obtain traffic light classifiers. Normalization of every candidate region in RGB and HSV spaces is performed and compared with a threshold to fulfill the verification. The experiment describes BIT's performance and the conclusion sets forth the main work in the next step.