In human-machine (autonomous vehicles and robots) interaction scenarios, pedestrians often appear in groups. Pedestrian groups provide richer information compared to individuals, which helps address occlusion problems in pedestrian-machine interactions. However, the randomness and spatiotemporal complexity of pedestrian activity make pedestrian group activity recognition (PGAR) a highly challenging task. This article provides a detailed description of the PGAR task. For the first time, a definition of pedestrian group and activity for autonomous vehicles and robots is provided. Existing datasets and methods are systematically summarized. Furthermore, the unique challenges and trends in PGAR for autonomous vehicles and robots are outlined. Although some related surveys have been published, there has not yet been a survey specifically focused on PGAR in autonomous driving and robotics scenarios. Therefore, the goal of this article is to narrow the gap in this topic and provide a comprehensive reference for researchers in this field.

Rapid sampling from the environment to acquire available frontier points and timely incorporating them into subsequent planning to reduce fragmented regions are critical to improve the efficiency of autonomous exploration. We propose HPHS, a fast and effective method for the autonomous exploration of unknown environments. In this work, we efficiently sample frontier points directly from the LiDAR data and the local map around the robot, while exploiting a hierarchical planning strategy to provide the robot with a global perspective. The hierarchical planning framework divides the updated environment into multiple subregions and arranges the order of access to them by considering the overall revenue of the global path. The combination of the hybrid frontier sampling method and hierarchical planning strategy reduces the complexity of the planning problem and mitigates the issue of region remnants during the exploration process. Detailed simulation and real-world experiments demonstrate the effectiveness and efficiency of our approach in various aspects. The source code will be released to benefit the further research1.

Abstract 3D Gaussian Splatting (3DGS) has emerged as a promising representation for scene reconstruction and novel view synthesis for its explicit representation and real‐time capabilities. This technique thus holds immense potential for use in mapping applications. Consequently, there is a growing need for an efficient and effective camera relocalization method to complement the advantages of 3DGS. This paper presents a camera relocalization method, namely GauLoc, in a scene represented by 3DGS. Unlike previous methods that rely on pose regression or photometric alignment, our proposed method leverages the differential rendering capability provided by 3DGS. The key insight of our work is the proposed implicit featuremetric alignment, which effectively optimizes the alignment between rendered keyframes and the query frames, and leverages the epipolar geometry to facilitate the convergence of camera poses conditioned explicit 3DGS representation. The proposed method significantly improves the relocalization accuracy even in complex scenarios with large initial camera rotation and translation deviations. Extensive experiments validate the effectiveness of our proposed method, showcasing its potential to be applied in many real‐world applications. Source code will be released at https://github.com/xinzhe11/GauLoc .

Vehicle trajectory prediction plays a crucial role in ensuring the driving safety of autonomous vehicles in complex traffic scenes. To accurately predict the trajectory of autonomous vehicles, in this article, we propose a distributed decouple long short-term memory (LSTM) self-trajectory prediction method for autonomous driving. The proposed new recurrent network includes a decouple-LSTM unit and corresponding distributed network architecture. To characterize the closed-loop dynamics of autonomous vehicles, a decouple gate and a control gate are proposed to build the decouple-LSTM unit. The data are processed in different ways according to whether the data participates in the recurrent. The decouple gate filters the data participating in the recurrent, while the control gate handles the data outside the recurrent. By leveraging the decouple-LSTM unit, a distributed network architecture is established, which corresponds with the general vehicle motion control architecture, which effectively models the vehicle motion processes. The proposed method is trained using an actual vehicle dataset and validated through vehicle experiments. The prediction horizon ranges from 0.5 to 3 s. When the prediction horizon is set to 3 s, compared with the LSTM method, the mean square error of the proposed method decreases by 98.0%. Results show that the proposed method significantly improves vehicle trajectory prediction accuracy.

Autonomous vehicles (AVs) bring advantages such as comfort, safety, and eco-friendliness compared to conventional ones. The focus on comfort has become increasingly important as it plays a key role in the widespread acceptance of AVs. Passenger demand for alleviating motion sickness (MS) during travel has also driven extensive research in this area. This manuscript conducts a comprehensive and comparative review of published articles to provide up-to-date research outcomes on MS mitigation. It also examines the optimization methods employed over the past two decades to enhance motion comfort in AVs. The methods for detecting and resolving MS are elaborated upon, and the research gap and challenges are addressed. Furthermore, the manuscript presents the current efficacy of the research and proposes an anti-motion sickness control framework that utilizes a cloud control platform, which might offer potential future research directions in the field of autonomous driving. This study serves as a valuable reference for future efforts and opportunities to improve the motion comfort of AVs.

In this paper, we propose a long-sequence modeling framework, named StreamPETR, for multi-view 3D object detection. Built upon the sparse query design in the PETR series, we systematically develop an object-centric temporal mechanism. The model is performed in an online manner and the long-term historical information is propagated through object queries frame by frame. Besides, we introduce a motion-aware layer normalization to model the movement of the objects. StreamPETR achieves significant performance improvements only with negligible computation cost, compared to the single-frame baseline. On the standard nuScenes benchmark, it is the first online multi-view method that achieves comparable performance (67.6% NDS & 65.3% AMOTA) with lidar-based methods. The lightweight version realizes 45.0% mAP and 31.7 FPS, outperforming the state-of-the-art method (SOLOFusion) by 2.3% mAP and 1.8× faster FPS. Code has been available at https://github.com/exiawsh/StreamPETR.git.

Interactive reinforcement learning has shown promise in learning complex robotic tasks. However, the process can be human-intensive due to the requirement of a large amount of interactive feedback. This paper presents a new method that uses scores provided by humans instead of pairwise preferences to improve the feedback efficiency of interactive reinforcement learning. Our key insight is that scores can yield significantly more data than pairwise preferences. Specifically, we require a teacher to interactively score the full trajectories of an agent to train a behavioral policy in a sparse reward environment. To avoid unstable scores given by humans negatively impacting the training process, we propose an adaptive learning scheme. This enables the learning paradigm to be insensitive to imperfect or unreliable scores. We extensively evaluate our method for robotic locomotion and manipulation tasks. The results show that the proposed method can efficiently learn near-optimal policies by adaptive learning from scores while requiring less feedback compared to pairwise preference learning methods. The source codes are publicly available at https://github.com/SSKKai/Interactive-Scoring-IRL.

For automated driving vehicles, trajectory planning is responsible for obtaining feasible trajectories with velocity profiles according to driving environments. From the perspective of trajectory planning, multiple uncertainties of environments and tracking deviations are two significant factors affecting driving safety. The former disturbs the judgment of trajectory planning on the environments, and the latter reduces the tracking accuracy of planned trajectories. To solve these problems, an improved model predictive control (MPC) trajectory planning method is proposed in this paper. Firstly, a Kalman filter fusion method is carried out to predict obstacle trajectory and their uncertainty, which combines model-based and data-based prediction methods. Based on the prediction results, a tube-based MPC trajectory planning method is applied to plan a reference trajectory with a small tracking deviation. The tube-based MPC is composed of two parts. One is the MPC with tightened constraints that is used to plan a feasible trajectory according to a nominal vehicle system and driving environment. The other is a state feedback control that is proposed to adjust the above planned trajectory to reduce the tracking deviations. To our knowledge, this paper proposes Kalman filter fusion and tube-based MPC planning method for the first time to consider the uncertainties of trajectory prediction and tracking control meanwhile in the planning. The planning method is verified by simulations and experiments in multiple scenes. Results show that the method is suitable for both static and dynamic scenes. Compared with applying the basic prediction method, the lateral deviation of the proposed method from the ideal trajectory is decreased by 46.5%. Compared with the nominal MPC method, the lateral tracking deviations of the proposed method are decreased by 77.42%.

Neural networks have dominated the research of hyperspectral image classification, attributing to the feature learning capacity of convolution operations. However, the fixed geometric structure of convolution kernels hinders long-range interaction between features from distant locations. In this article, we propose a novel spectral–spatial transformer network (SSTN), which consists of spatial attention and spectral association modules, to overcome the constraints of convolution kernels. Also, we design a factorized architecture search (FAS) framework that involves two independent subprocedures to determine the layer-level operation choices and block-level orders of SSTN. Unlike conventional neural architecture search (NAS) that requires a bilevel optimization of both network parameters and architecture settings, the FAS focuses only on finding out optimal architecture settings to enable a stable and fast architecture search. Extensive experiments conducted on five popular HSI benchmarks demonstrate the versatility of SSTNs over other state-of-the-art (SOTA) methods and justify the FAS strategy. On the University of Houston dataset, SSTN obtains comparable overall accuracy to SOTA methods with a small fraction (1.2%) of multiply-and-accumulate operations compared to a strong baseline spectral–spatial residual network (SSRN). Most importantly, SSTNs outperform other SOTA networks using only 1.2% or fewer MACs of SSRNs on the Indian Pines, the Kennedy Space Center, the University of Pavia, and the Pavia Center datasets.

Autonomous vehicles were experiencing rapid development in the past few years. However, achieving full autonomy is not a trivial task, due to the nature of the complex and dynamic driving environment. Therefore, autonomous vehicles are equipped with a suite of different sensors to ensure robust, accurate environmental perception. In particular, the camera-LiDAR fusion is becoming an emerging research theme. However, so far there has been no critical review that focuses on deep-learning-based camera-LiDAR fusion methods. To bridge this gap and motivate future research, this article devotes to review recent deep-learning-based data fusion approaches that leverage both image and point cloud. This review gives a brief overview of deep learning on image and point cloud data processing. Followed by in-depth reviews of camera-LiDAR fusion methods in depth completion, object detection, semantic segmentation, tracking and online cross-sensor calibration, which are organized based on their respective fusion levels. Furthermore, we compare these methods on publicly available datasets. Finally, we identified gaps and over-looked challenges between current academic researches and real-world applications. Based on these observations, we provide our insights and point out promising research directions.

Robust road boundary extraction and completion play an important role in providing guidance to all road users and supporting high-definition (HD) maps. The significant challenges remain in remarkable and accurate road boundary recovery from poor road boundary conditions. This paper presents a novel deep learning framework, named BoundaryNet, to extract and complete road boundaries by using both mobile laser scanning (MLS) point clouds and high-resolution satellite imagery. First, road boundaries are extracted by conducting a curb-based extraction method. Such extracted 3D road boundary lines are used as inputs to feed into a U-shaped network for erroneous boundary denoising. Then, a convolutional neural network (CNN) model is proposed to complete the road boundaries. Next, to achieve more complete and accurate road boundaries, a conditional deep convolutional generative adversarial network (c-DCGAN) with the assistance of road centerlines extracted from satellite images is developed. Finally, according to the completed road boundaries, the inherent road geometries are calculated. The proposed methods were evaluated using satellite imagery and four MLS point cloud datasets with varying densities and road conditions in urban environments. The quality evaluation metrics of 82.88%, 82.43%, 88.86%, and 84.89% were achieved for four data sets. The experimental results indicate that the BoundaryNet model can provide a promising solution for road boundary completion and road geometry estimation.

Accurate road marking extraction and classification play a significant role in the development of autonomous vehicles (AVs) and high-definition (HD) maps. Due to point density and intensity variations from mobile laser scanning (MLS) systems, most of the existing thresholding-based extraction methods and rule-based classification methods cannot deliver high efficiency and remarkable robustness. To address this, we propose a capsule-based deep learning framework for road marking extraction and classification from massive and unordered MLS point clouds. This framework mainly contains three modules. Module I is first implemented to segment road surfaces from 3D MLS point clouds, followed by an inverse distance weighting (IDW) interpolation method for 2D georeferenced image generation. Then, in Module II, a U-shaped capsule-based network is constructed to extract road markings based on the convolutional and deconvolutional capsule operations. Finally, a hybrid capsule-based network is developed to classify different types of road markings by using a revised dynamic routing algorithm and large-margin Softmax loss function. A road marking dataset containing both 3D point clouds and manually labeled reference data is built from three types of road scenes, including urban roads, highways, and underground garages. The proposed networks were accordingly evaluated by estimating robustness and efficiency using this dataset. Quantitative evaluations indicate the proposed extraction method can deliver 94.11% in precision, 90.52% in recall, and 92.43% in F1-score, respectively, while the classification network achieves an average of 3.42% misclassification rate in different road scenes.

Although significant improvement has been achieved in fully autonomous driving and semantic high-definition map (HD) domains, most of the existing 3D point cloud segmentation methods cannot provide high representativeness and remarkable robustness. The principally increasing challenges remain in completely and efficiently extracting high-level 3D point cloud features, specifically in large-scale road environments. This paper provides an end-to-end feature extraction framework for 3D point cloud segmentation by using dynamic point-wise convolutional operations in multiple scales. Compared to existing point cloud segmentation methods that are commonly based on traditional convolutional neural networks (CNNs), our proposed method is less sensitive to data distribution and computational powers. This framework mainly includes four modules. Module I is first designed to construct a revised 3D point-wise convolutional operation. Then, a U-shaped downsampling-upsampling architecture is proposed to leverage both global and local features in multiple scales in Module II. Next, in Module III, high-level local edge features in 3D point neighborhoods are further extracted by using an adaptive graph convolutional neural network based on the K-Nearest Neighbor (KNN) algorithm. Finally, in Module IV, a conditional random field (CRF) algorithm is developed for postprocessing and segmentation result refinement. The proposed method was evaluated on three large-scale LiDAR point cloud datasets in both urban and indoor environments. The experimental results acquired by using different point cloud scenarios indicate our method can achieve state-of-the-art semantic segmentation performance in feature representativeness, segmentation accuracy, and technical robustness.

Recently, the advancement of deep learning (DL) in discriminative feature learning from 3-D LiDAR data has led to rapid development in the field of autonomous driving. However, automated processing uneven, unstructured, noisy, and massive 3-D point clouds are a challenging and tedious task. In this article, we provide a systematic review of existing compelling DL architectures applied in LiDAR point clouds, detailing for specific tasks in autonomous driving, such as segmentation, detection, and classification. Although several published research articles focus on specific topics in computer vision for autonomous vehicles, to date, no general survey on DL applied in LiDAR point clouds for autonomous vehicles exists. Thus, the goal of this article is to narrow the gap in this topic. More than 140 key contributions in the recent five years are summarized in this survey, including the milestone 3-D deep architectures, the remarkable DL applications in 3-D semantic segmentation, object detection, and classification; specific data sets, evaluation metrics, and the state-of-the-art performance. Finally, we conclude the remaining challenges and future researches.

Recent geometric deep learning works define convolution operations in local regions and have enjoyed remarkable success on non-Euclidean data, including graph and point clouds. However, the high-level geometric correlations between the input and its neighboring coordinates or features are not fully exploited, resulting in suboptimal segmentation performance. In this article, we propose a novel graph convolution architecture, which we term as Taylor Gaussian mixture model (GMM) network (TGNet), to efficiently learn expressive and compositional local geometric features from point clouds. The TGNet is composed of basic geometric units, TGConv, that conduct local convolution on irregular point sets and are parametrized by a family of filters. Specifically, these filters are defined as the products of the local point features and the neighboring geometric features extracted from local coordinates. These geometric features are expressed by Gaussian weighted Taylor kernels. Then, a parametric pooling layer aggregates TGConv features to generate new feature vectors for each point. TGNet employs TGConv on multiscale neighborhoods to extract coarse-to-fine semantic deep features while improving its scale invariance. Additionally, a conditional random field (CRF) is adopted within the output layer to further improve the segmentation results. Using three point cloud data sets, qualitative and quantitative experimental results demonstrate that the proposed method achieves 62.2% average accuracy on ScanNet, 57.8% and 68.17% mean intersection over union (mIoU) on Stanford Large-Scale 3D Indoor Spaces (S3DIS) and Paris-Lille-3D data sets, respectively.

This paper presents a segment-based traffic sign detection method using vehicle-borne mobile laser scanning (MLS) data. This method has three steps: road scene segmentation, clustering and traffic sign detection. The non-ground points are firstly segmented from raw MLS data by estimating road ranges based on vehicle trajectory and geometric features of roads (e.g., surface normals and planarity). The ground points are then removed followed by obtaining non-ground points where traffic signs are contained. Secondly, clustering is conducted to detect the traffic sign segments (or candidates) from the non-ground points. Finally, these segments are classified to specified classes. Shape, elevation, intensity, 2D and 3D geometric and structural features of traffic sign patches are learned by the support vector machine (SVM) algorithm to detect traffic signs among segments. The proposed algorithm has been tested on a MLS point cloud dataset acquired by a Leador system in the urban environment. The results demonstrate the applicability of the proposed algorithm for detecting traffic signs in MLS point clouds.

This study recognizes the significance and considerable commercial applications in creating Level of Detail (LoD) building models for 3D city models generation. Accordingly, this paper proposes a novel method to identify and extract window frames on building facades from Mobile Laser Scanning (MLS) point clouds. The proposed method can typically be regarded as a stepwise procedure. Firstly, a voxel-based upward-growing method is applied to distinguish non-ground points from ground points. Next, outliers are filtered out from non-ground points by statistical analysis. Then, all the remaining non-ground points are clustered based on the conditional Euclidean clustering algorithm to segment out building facades. A volumetric box is afterward created to store façade points so that neighbors of each point can be operated. Finally, a manipulator is applied according to the structural characteristics of window frames to extract the potential window points. Quantitative evaluations based on 2D validation and 3D validation were both conducted. In the 2D validation, the lowest F1-measure of the test datasets is 0.740, and the highest can be 0.977. While in the 3D validation, the lowest precision of the test dataset is 79.58%, and the highest can be 97.96%. The results demonstrate the proposed method can successfully extract the rectangular or curved windows in the test datasets with promising accuracies to support the generation of LoD3 building models.