Combining Mixture of Experts (MoE) with Low-Rank Adaptation (LoRA) has shown promising efficiency in multi-task instruction tuning for Large Language Models (LLMs). While existing routing schemes for such MoE systems employ auxiliary functions to ensure both expert selection certainty and workload balance among experts, they are hindered by two critical challenges: (1) Existing methods overlook the evolving cross-expert relationships across layers, leading to inefficient expert utilization. (2) The auxiliary functions fail to incorporate cross-task semantic characteristics during expert assignment, leading to suboptimal task adaptation. To address these challenges, we propose Hybrid routing for a Mixture of LoRA Experts (HotMoE), a novel multi-task instruction tuning framework that adapts hierarchical routing to the distinct characteristics of different LLM layers. First, we design a hybrid routing module. In lower layers, expert-expert attention facilitates cross-task collaboration and generalization. In higher layers, token-expert attention enables precise alignment between task semantics and specialized experts. Second, we introduce a similarity-guided auxiliary loss module to regularize routing decisions by exploiting hidden state similarities. This loss synergistically reinforces expert specialization without sacrificing certainty of expert selection by promoting cohesive activation patterns among semantically related tasks while sharpening distinctions between conflicting ones. Experiments across two multi-task instruction tuning scenarios covering seven NLP benchmarks demonstrate that HotMoE consistently outperforms all baselines, improving Mean Relative Difference by up to 1.68% with only 3.1% of trainable parameters.

LiDAR odometry is a critical component of SLAM in autonomous driving and robotics. Learning-based methods have shown remarkable performance by regressing relative poses in an end-to-end manner. However, when applying these trained models, originally developed on the widely used KITTI dataset, to other scenes, performance often drops significantly. In other words, existing methods struggle to generalize well to new environments. To address this challenge, we propose RCP-LO, a simple yet effective LiDAR odometry framework. We introduce a novel representation for relative poses, reformulating them as relative coordinates, which can then be solved using geometrical verification. This approach avoids overly simplified pose representations and makes better use of scene geometry, thereby improving generalization. Moreover, to capture the inherent uncertainties in relative pose estimation from occluded LiDAR point clouds from dynamic environments, we adapt our framework to learn a denoising diffusion model, allowing for sampling plausible relative coordinates while improving robustness. We also introduce a differentiable geometric weighted singular value decomposition module, enabling efficient pose estimation through a single forward pass. Extensive experiments demonstrate that RCP-LO, trained exclusively on the KITTI dataset, achieves competitive performance compared to SOTA learning-based methods and generalizes effectively to the KITTI-360, Ford, and Oxford datasets.

3D object detection in adverse weather is crucial for autonomous driving, especially in smoke where LiDAR data becomes sparse and noisy. Due to the lack of real smoke data, this paper introduces a physics-based simulation framework to generate realistic LiDAR point clouds of smoke and augment large-scale driving datasets. First, we present a 3D fluid dynamics-based smoke simulation framework in Unity, which models the realistic spatial diffusion and temporal evolution of smoke particles. Coupled with a physically accurate LiDAR perception module, our system captures complex light interactions—such as beam attenuation, scattering, and multi-path effects—to generate high-fidelity, physically consistent smoke point clouds. Second, we propose a range image-based data fusion strategy that seamlessly integrates the simulated smoke point clouds into large-scale real-world LiDAR datasets (e.g., Waymo). This approach accurately emulates LiDAR scanning characteristics and naturally incorporates occlusion effects, enabling realistic smoke integration without compromising spatial consistency. To validate our approach, we collect a real-world LiDAR smoke dataset (LiSmoke) and conduct extensive experiments using state-of-the-art 3D detectors. Results demonstrate that models trained with our augmented synthetic data achieve significant improvements in smoke-affected scenarios, while maintaining competitive performance in clear-weather conditions. Our work provides a cost-effective solution for enhancing perception robustness in safety-critical environments.

Remote sensing imagery often involves complex backgrounds and multi-scale targets, while variations such as rotation and scaling significantly degrade the performance of existing object detection algorithms and hinder effective modeling of spatial relationships between objects. To address these challenges, we propose SFE-CapsNet. First, we fuse scalar features extracted by convolutional neural networks (CNNs) with vector features from capsule networks through structural reorganization to form virtual capsules, approximating the dynamic routing process via a single fully connected layer. This design preserves object pose and texture information while streamlining information flow. Second, we introduce a capsule attention module that generates attention masks to dynamically enhance target-relevant features and suppress background noise, strengthening multi-level feature representations. Integrated with a feature pyramid network (FPN) architecture, our approach achieves precise detection of targets at varying scales. Experimental results demonstrate that multi-level feature fusion and the capsule attention mechanism significantly improve detection accuracy and robustness, achieving 77.65% mean Average Precision (mAP) on the DOTA dataset and 97.63% mAP on HRSC2016, highlighting its effectiveness and efficiency in complex scenes.

Enhancing perception performance via multi-agent collaboration has gained increasing attention in the field of autonomous driving. However, as the number of agents grows, the manual annotation required for training collaborative detectors increases significantly. To tackle this problem, we introduce an unsupervised method that learns to Detect Objects from Multi-Agent LiDAR scans, named DOtA, without using labels from external. DOtA first generates preliminary labels by an initial detector, which is trained by internally shared information of collaborative agents. DOtA then optimizes these preliminary labels by utilizing the physical rule constraints derived from the surrounding area of the object. Building on DOtA, we further propose DOtA++, an enhanced version that improves performance by leveraging composite prior constraints. Beyond physical rule constraints, DOtA++ further uses image data as an auxiliary modality to introduce multi-agent observation consistency constraints, boosting object classification, while also incorporating point cloud geometric distribution constraints to improve structural description. Extensive experiments on widely-used benchmarks demonstrate that DOtA and DOtA++ effectively perceive potential objects in the scene without manual annotations. In particular, DOtA++ shows 10.7% mAP improvement over traditional unsupervised methods on V2X-R dataset.

Efficient fusion of infrared and visible images is of critical importance for real-time applications such as autonomous driving. While deep learning-based fusion methods have demonstrated significant improvements in fusion quality in recent years, current network architectures still exhibit unsatisfactory computational complexity and processing speed. To reduce computational complexity and improve fusion efficiency, mask-based methods or approaches driven by downstream tasks often prioritize key regions. However, such methods tend to overemphasize target objects, potentially overlooking contextually significant elements. To address this limitation and achieve more effective fusion, we propose BEFuse, a decoupled two-stage training strategy with an end-to-end inference framework. BEFuse extracts shallow features and gradient information from images in the first stage via a cross-modal image segmentation subnetwork. In the fusion stage, we use the Hadamard product to map features to an implicit quadratic feature space, combining feature similarity and gradient mask information, allowing automatic adjustment of loss weights and improving fusion accuracy. Experiments on four datasets (MSRS, TNO, RoadScene, and M3FD) show that BEFuse outperforms existing methods in both fusion quality and computational speed.

Cross-view geo-localization (CVGL) aims to match images of the same location captured from different viewpoints, such as those captured by Unmanned Aerial Vehicles (UAVs) and satellite platforms. The task is particularly challenging due to significant variations in scale, viewpoint, and illumination. Most existing methods employ symmetric sampling strategy to construct drone–satellite image pairs for deep metric learning, but neglect the potential of incorporating multi-view drone images to enhance the viewpoint robustness of features. To address this, we propose leveraging multi-view images to learn Domain-Invariant Discriminative Embeddings (DIDE) for CVGL. DIDE introduces an Inter-view Feature Aggregation Module (IFAM), which dynamically integrates multi-view drone information into robust embeddings. These are used in contrastive learning with satellite embeddings within batches to learn view-invariant discriminative features, while representation learning further improves scene discrimination across batches. To reduce the domain gap, DIDE constructs and aligns drone and satellite prototypes for effective cross-domain feature alignment. Furthermore, we adopt a parameter-efficient transfer learning strategy that leverages the capabilities of pre-trained foundation models while fine-tuning only dual adapters, significantly reducing the trainable parameters. DIDE achieves the state-of-the-art on University-1652 and University-160k, competitive results on SUES-200, and demonstrates strong cross-dataset transferability, with fewer training parameters and lower computational cost.

Although 3D Gaussian Splatting (3DGS) has greatly advanced the development of novel view synthesis (NVS) and surface reconstruction tasks, it still faces serious challenges in indoor scenes with only sparse views due to the lack of initialization point cloud. Existing GS-like methods are geometrically initialized using point clouds from Structure from Motion (SfM), and their performance is heavily limited by the point cloud quality. However, in indoor scenes containing large areas of textureless or weakly textured regions, the SfM algorithm struggles to generate point clouds in these areas, especially when only a limited number of input views are available. This situation can cause the GS optimization process to overfit the photometric error, resulting in severe geometric degradation in regions lacking proper initialization. To address the above problem, we propose a novel sparse-view 3DGS method using global scale-aligned monocular depth to initialize Gaussian primitives and optimize them in view space, named VGA-GS. First, we design a global scale-aligned monocular depth initialization strategy, which provides geometric priors in textureless or weakly textured regions while avoiding the local geometric distortions in vanilla alignments. Second, we propose a novel Gaussian primitive representation based on view space and a refinement strategy based on pixel errors, which can constrain each primitive to always be within the field of view where it is observed to be fully optimized. Finally, we extract the surface prior from monocular depth to regularize the rendering depth and normal, and further propagate the constraints of the training view to the neighboring pseudo-views via image warping. Our method achieves state-of-the-art reconstruction performance in both textureless regions and local details for indoor scenes with sparse views. Moreover, it is highly scalable, supporting integration with various 3DGS rasterizers and depth estimation methods. We release the code at https://github.com/XT5un/VGA-GS.

Gait recognition aims to identify individuals based on walking patterns in a long-range, contactless manner. While camera-based methods have advanced significantly, their performance deteriorates under poor lighting conditions. LiDAR offers a promising alternative by capturing accurate 3D gait information regardless of illumination. However, effectively integrating heterogeneous data from diverse sensors, such as LiDAR and cameras, remains a key challenge for cross-modal gait recognition. Existing approaches often minimize modality discrepancy directly, which can lead to class collapse and damage to inter-class discriminability. To overcome these limitations, we propose a Semantic-Guided Cross-modal Gait recognition framework, SG-CrossGait, that introduces text features as the prototype space to bridge camera and LiDAR modalities. We design structured Gait Description Factors (GDF) and leverage multimodal large language models (MLLMs) for automatic factor annotation and text generation, enriching existing datasets with textual descriptions, yielding SUSTech1K-Text and FreeGait-Text. A CLIP-based pipeline aligns multi-grained representations from both modalities to the text prototype space. We further propose the Dual-stream Cross-attention Fusion (DCF) module for fine-grained feature integration and the Semantic-Guided Feature Decoupling (SGFD) module to disentangle shared and modality-specific features. A Multi-task Training (MT) scheme incorporating Gait Attribute Recognition (GAR) further enhances intra-class compactness. Extensive experiments validate the effectiveness of our approach. On SUSTech1K-Text, our method achieves 61% accuracy in LiDAR-to-Camera recognition, outperforming the state-of-the-art method by 8.3%. We also release the Gait-Text benchmark to promote future research at the intersection of gait analysis and vision-language learning. Code and datasets are available at: https://github.com/O-VIGIA/SCCG.git.

Gait recognition provides a crucial biometric identification technique for intelligent security since it is long-range and nonintrusive. Previous methods have generally focused on gait recognition within a single modality, but with the constant advancement of multimedia sensors, there is a growing requirement for the development of multimodal and cross-modal gait recognition. The various modalities are aligned directly in a hard manner by current cross-modal approaches, which could result in class collapse caused by imbalanced learning and make learning discriminative representations difficult for the weaker modality. To overcome these limitations, we propose a unified cross-modal gait recognition framework, UniCrossGait. Specifically, we employ the knowledge distillation paradigm to transfer the knowledge of a fused multimodal teacher network, Exist Methods UniCrossGait(Ours) Figure 1: Illustration of comparison between exist methods and which serves as a shared intermediate representation, to unimodal student networks, which enables them to learn modality-invariant representations. To perform knowledge distillation effectively, we design the Direction-level Feature Imitation loss and Intra&Inter sample Correlation loss. Furthermore, we construct the Warm up operator and Distillation Balancing operator to prevent the network from falling into suboptimal solutions due to pure imitation and imbalanced distillation. Our method achieves state of-the-art (SOTA) results on cross-modal gait recognition datasets, and extensive ablation experiments demonstrate the effectiveness of the proposed paradigm and modules. Our code is available at https://github.com/O-VIGIA/UniCrossGait.

The reputation of universities has drawn increasing attention in recent years, especially with the emergence of various rankings. However, despite advances in big data technologies that facilitate data collection and analysis, accurately defining and balancing factors related to university reputation and educational quality remains complex and tedious. Moreover, current educational assessment methods exhibit notable differences and controversies. In this paper, we present Iva , a human-in-the-loop I ntelligent V isual A ssessment system for higher education quality. This system utilizes large language models to analyze extensive multi-modal educational data, with visualization techniques incorporated to enable multi-scale exploration and interaction. Our extensive evaluations, including a carefully-designed user study and expert interviews, demonstrate the system’s potential value and provide insights for future improvements.

Scene flow, which provides the 3D motion field of the first frame from two consecutive point clouds, is vital for dynamic scene perception. However, contemporary scene flow methods face three major challenges. Firstly, they only consider the context of individual point clouds before flow embedding, leading to embedded points struggling to perceive the consistent semantic relationship of another frame. To address this issue, we propose a novel approach called Dual Cross Attentive (DCA) for the latent fusion and alignment between two frames based on semantic contexts. This is then integrated into Global Fusion Flow Embedding (GF) to initialize flow embedding based on global correlations in both contextual and Euclidean spaces. Secondly, deformations exist in non-rigid objects after the warping layer, which distorts the spatiotemporal relation between the consecutive frames. For a more precise estimation of residual flow at next-level, the Spatial Temporal Re-embedding (STR) module is devised to update the point sequence features at current-level. Lastly, poor generalization is often observed due to the significant domain gap between synthetic and LiDAR-scanned datasets. We leverage novel domain adaptive losses to effectively bridge the gap of motion inference from synthetic to real-world. Experiments demonstrate that our approach achieves state-of-the-art (SOTA) performance across various datasets, with particularly outstanding results in real-world LiDAR-scanned situations.

Visual localization is a fundamental machine learning problem. Absolute Pose Regression (APR) trains a scene-dependent model to efficiently map an input image to the camera pose in a pre-defined scene. However, many applications have continually changing environments, where inference data at novel poses or scene conditions (weather, geometry) appear after deployment. Training APR on a fixed dataset leads to overfitting, making it fail catastrophically on challenging novel data. This work proposes Continual Domain Expansion (ConDo), which continually collects unlabeled inference data to update the deployed APR. Instead of applying standard unsupervised domain adaptation methods which are ineffective for APR, ConDo effectively learns from unlabeled data by distilling knowledge from scene-agnostic localization methods. By sampling data uniformly from historical and newly collected data, ConDo can effectively expand the generalization domain of APR. Large-scale benchmarks with various scene types are constructed to evaluate models under practical (long-term) data changes. ConDo consistently and significantly outperforms baselines across architectures, scene types, and data changes. On challenging scenes (Fig.1), it reduces the localization error by >7x (14.8m vs 1.7m). Analysis shows the robustness of ConDo against compute budgets, replay buffer sizes and teacher prediction noise. Comparing to model re-training, ConDo achieves similar performance up to 25x faster.

LiDAR-based 3D object detection is crucial for autonomous driving. However, due to the quality deterioration of LiDAR point clouds, it suffers from performance degradation in adverse weather conditions. Fusing LiDAR with the weatherrobust 4D radar sensor is expected to solve this problem; however, it faces challenges of significant differences in terms of data quality and the degree of degradation in adverse weather. To address these issues, we introduce L4DR, a weather-robust 3D object detection method that effectively achieves LiDAR and 4D Radar fusion. Our L4DR proposes Multi-Modal Encoding (MME) and Foreground-Aware Denoising (FAD) modules to reconcile sensor gaps, which is the first exploration of the complementarity of early fusion between LiDAR and 4D radar. Additionally, we design an Inter-Modal and IntraModal ({IM}2) parallel feature extraction backbone coupled with a Multi-Scale Gated Fusion (MSGF) module to counteract the varying degrees of sensor degradation under adverse weather conditions. Experimental evaluation on a VoD dataset with simulated fog proves that L4DR is more adaptable to changing weather conditions. It delivers a significant performance increase under different fog levels, improving the 3D mAP by up to 20.0% over the traditional LiDAR-only approach. Moreover, the results on the K-Radar dataset validate the consistent performance improvement of L4DR in realworld adverse weather conditions.

Language-based localization is a crucial task in robotics and computer vision, enabling robots to understand spatial positions through language. Recent methods rely on contrastive learning to establish correspondences between global features of texts and point clouds. However, the inherent ambiguity of textual descriptions makes it difficult to convey geometric information accurately, forcing alignment of them in the feature space may compromise the expressiveness of the point clouds. Unlike previous methods, this paper proposes using language as a filter to distinguish dissimilar locations. To this end, we propose a robust framework of multi-level negative contrastive learning for language-based localization, fully leveraging the descriptive power of language for spatial localization. Our method learns multiple mismatched factors by minimizing the similarity of different locations at different levels, including global-level, instance-level and relationlevel, respectively. Extensive experiments conducted on the KITTI360Pose benchmark demonstrate that our method outperforms better that the state-of-the-art methods. Specifically, we achieve a 56.3% improvement in Top-1 retrieval recall and a 45.9% improvement in 5m localization recall.

While Neural Radiance Fields (NeRFs) have advanced the frontiers of novel view synthesis (NVS) using LiDAR data, they still struggle in dynamic scenes. Due to the low frequency and sparsity characteristics of LiDAR point clouds, it is challenging to spontaneously learn a dynamic and consistent scene representation from posed scans. In this paper, we propose STGC-NeRF, a novel LiDAR NeRF method that combines spatial-temporal geometry consistency to enhance the reconstruction of dynamic scenes. First, we propose a temporal geometry consistency regularization to enhance the regression of time-varying scene geometries from low-frequency LiDAR sequences. By estimating the pointwise correspondences between synthetic (or real) and real frames at different times, we convert them into various forms of temporal supervision. This alleviates the inconsistency caused by moving objects in dynamic scenes. Second, to improve the reconstruction of sparse LiDAR data, we propose spatial geometric consistency constraints. By computing multiple neighborhood feature descriptors incorporating geometric and contextual information, we capture structural geometry information from sparse LiDAR data. This helps encourage consistent direction, smoothness, and detail of the local surface. Extensive experiments on the KITTI-360 and nuScenes datasets demonstrate that STGC-NeRF outperforms state-of-the-art methods in both geometry and intensity accuracy for dynamic LiDAR scene reconstruction.

Autonomous vehicles (AVs) rely on LiDAR sensors for environmental perception and decision-making in driving scenarios. However, ensuring the safety and reliability of AVs in complex environments remains a pressing challenge. To address this issue, we introduce a real-world dataset (ROLiD) comprising LiDAR-scanned point clouds of two random objects: water mist and smoke. In this paper, we introduce a novel adversarial perspective by proposing an attack framework that utilizes water mist and smoke to simulate environmental interference. Specifically, we propose a point cloud sequence generation method using a motion and content decomposition generative adversarial network named PCS-GAN to simulate the distribution of random objects. Furthermore, leveraging the simulated LiDAR scanning characteristics implemented with Range Image, we examine the effects of introducing random object perturbations at various positions on the target vehicle. Extensive experiments demonstrate that adversarial perturbations based on random objects effectively deceive vehicle detection and reduce the recognition rate of 3D object detection models.

LIDAR-based 3D object detection and semantic segmentation are critical tasks in 3D scene understanding. Traditional detection and segmentation methods supervise their models through bounding box labels and semantic mask labels. However, these two independent labels inherently contain significant redundancy. This paper aims to eliminate the redundancy by supervising 3D object detection using only semantic labels. However, the challenge arises due to the incomplete geometry structure and boundary ambiguity of point cloud instances, leading to inaccurate pseudo-labels and poor detection results. To address these challenges, we propose a novel method, named Seg2Box. We first introduce a Multi-Frame Multi-Scale Clustering (MFMS-C) module, which leverages the spatio-temporal consistency of point clouds to generate accurate box-level pseudo-labels. Additionally, the Semantic-Guiding Iterative-Mining Self-Training (SGIM-ST) module is proposed to enhance the performance by progressively refining the pseudo-labels and mining the instances without generating pseudo-labels. Experiments on the Waymo Open Dataset and nuScenes Dataset show that our method significantly outperforms other competitive methods by 23.7% and 10.3% in mAP, respectively. The results demonstrate the great label-efficient potential and advancement of our method.

Recently, Visual Foundation Models (VFMs) have shown a remarkable generalization performance in 3D perception tasks. However, their effectiveness in large-scale outdoor datasets remains constrained by the scarcity of accurate supervision signals, the extensive noise caused by variable outdoor conditions, and the abundance of unknown objects. In this work, we propose a novel label-free learning method, Adaptive Label Correction (AdaCo), for 3D semantic segmentation. AdaCo first introduces the Cross-modal Label Generation Module (CLGM), providing cross-modal supervision with the formidable interpretive capabilities of the VFMs. Subsequently, AdaCo incorporates the Adaptive Noise Corrector (ANC), updating and adjusting the noisy samples within this supervision iteratively during training. Moreover, we develop an Adaptive Robust Loss (ARL) function to modulate each sample's sensitivity to noisy supervision, preventing potential underfitting issues associated with robust loss. Our proposed AdaCo can effectively mitigate the performance limitations of label-free learning networks in 3D semantic segmentation tasks. Extensive experiments on two outdoor benchmark datasets highlight the superior performance of our method.

Scene coordinate regression achieves impressive results in outdoor LiDAR localization but requires days of training. Since training needs to be repeated for each new scene, long training times make these impractical for applications requiring time-sensitive system upgrades, such as autonomous driving, drones, robotics, etc. We identify large coverage areas and vast amounts of data in large-scale outdoor scenes as key challenges that limit fast training. In this paper, we propose LightLoc, the first method capable of efficiently learning localization in a new scene at light speed. Beyond freezing the scene-agnostic feature backbone and training only the scene-specific prediction heads, we introduce two novel techniques to address these challenges. First, we introduce sample classification guidance to assist regression learning, reducing ambiguity from similar samples and improving training efficiency. Second, we propose redundant sample downsampling to remove well-learned frames during training, reducing training time without compromising accuracy. In addition, the fast training and confidence estimation characteristics of sample classification enable its integration into SLAM, effectively eliminating error accumulation. Extensive experiments on large-scale outdoor datasets demonstrate that LightLoc achieves state-of-the-art performance with just 1 hour of training—50× faster than existing methods. Our Code is available at https://github.com/liw95/LightLoc.

Federated learning (FL) is emerging as a promising technique for collaborative learning without local data leaving their devices. However, clients’ data originating from diverse domains may degrade model performance due to domain shifts, preventing the model from learning consistent representation space. In this paper, we propose a novel FL framework, Federated Domain Shift Eraser (FDSE), to improve model performance by differently erasing each client’s domain skew and enhancing their consensus. First, we formulate the model forward passing as an iterative deskewing process that extracts and then deskews features alternatively. This is efficiently achieved by decomposing each original layer in the neural network into a Domain-agnostic Feature Extractor (DFE) and a Domain-specific Skew Eraser (DSE). Then, a regularization term is applied to promise the effectiveness of feature deskewing by pulling local statistics of DSE’s outputs close to the globally consistent ones. Finally, DFE modules are fairly aggregated and broadcast to all the clients to maximize their consensus, and DSE modules are personalized for each client via similarity-aware aggregation to erase their domain skew differently. Comprehensive experiments were conducted on three datasets to confirm the advantages of our method in terms of accuracy, efficiency, and generalizability.

Adversarial examples can mislead deep neural networks with subtle perturbations, causing them to make incorrect predictions. Notably, adversarial examples crafted for one model can also deceive other models, a phenomenon known as the transferability of adversarial examples. To improve transferability, existing studies have designed increasingly complex mechanisms, but the improvements achieved remain relatively limited and are often difficult to adapt to other modalities, further restricting the scalability of these methods. In this work, we observe a mirroring relationship between model generalization and adversarial example transferability. Motivated by this observation, we propose an augmentation-based attack, called OPS (OperatorPerturbation-based Stochastic optimization), which constructs a stochastic optimization problem by input transformation operators and random perturbations, and solves this problem to generate adversarial examples with better transferability. Extensive experiments on both images and 3D point clouds demonstrate that OPS significantly outperforms existing state-of-the-art methods in terms of both performance and cost, showcasing the universality and superiority of our approach. The code is available at https://github.com/the-full/OPS.

LiDAR odometry is a critical module in autonomous driving systems, responsible for accurate localization by estimating the relative pose transformation between consecutive point cloud frames. However, existing studies frequently encounter challenges with unreliable pose estimation, due to the lack of in-depth understanding of scenario and the presence of noise interference. To address this challenge, we propose DiffLO, a semantic-aware LiDAR odometry network with diffusion-based refinement. To mitigate the impact of challenging cases such as dynamic, repetitive patterns, and low textures, we introduce a semantic distillation method that integrates semantic information into the odometry task. This allows the network to gain a semantic understanding of the scene, enabling it to focus more on the objects that are beneficial for pose estimation. Additionally, to enhance the robustness, we propose a diffusion-based refinement method. This method uses pose-related features as conditional constraints for generative diversity, iteratively refining the pose estimation to achieve greater accuracy. Comparative experiments on the KITTI odometry dataset demonstrate that the proposed method achieves state-of-the-art performance among existing learning-based approaches. Furthermore, the proposed DiffLO method outperforms the classic A-LOAM on most evaluation sequences. The code will be released at https://github.com/hytree7/difflo.

Current Vehicle-to-Everything (V2X) systems have significantly enhanced 3D object detection using LiDAR and camera data. However, they face performance degradation in adverse weather. Weather-robust 4D radar, with Doppler velocity and additional geometric information, offers a promising solution to this challenge. To this end, we present V2X-R, the first simulated V2X dataset incorporating LiDAR, camera, and 4D radar modalities. V2X-R contains 12,079 scenarios with 37,727 frames of LiDAR and 4D radar point clouds, 150,908 images, and 170,859 annotated 3D vehicle bounding boxes. Subsequently, we propose a novel cooperative LiDAR-4D radar fusion pipeline for 3D object detection and implement it with multiple fusion strategies. To achieve weather-robust detection, we additionally propose a Multi-modal Denoising Diffusion (MDD) module in our fusion pipeline. MDD utilizes weather-robust 4D radar feature as a condition to guide the diffusion model in denoising noisy LiDAR features. Experiments show that our LiDAR-4D radar fusion pipeline demonstrates superior performance in the V2X-R dataset. Over and above this, our MDD module further improved the foggy/snowy performance of the basic fusion model by up to 5.73%/6.70% and barely disrupting normal performance. The dataset and code will be publicly available at: https://github.com/ylwhxht/V2X-R.

Unsupervised 3D object detection serves as an important solution for offline 3D object annotation. However, due to the data sparsity and limited views, the clustering-based label fitting in unsupervised object detection often generates low-quality pseudo-labels. Multi-agent collaborative dataset, which involves the sharing of complementary observations among agents, holds the potential to break through this bottleneck. In this paper, we introduce a novel unsupervised method that learns to Detect Objects from Multi-Agent LiDAR scans, termed DOtA, without using labels from external. DOtA first uses the internally shared ego-pose and ego-shape of collaborative agents to initialize the detector, leveraging the generalization performance of neural networks to infer preliminary labels. Subsequently, DOtA uses the complementary observations between agents to perform multi-scale encoding on preliminary labels, then decodes high-quality and low-quality labels. These labels are further used as prompts to guide a correct feature learning process, thereby enhancing the performance of the unsupervised object detection task. Extensive experiments on the V2V4Real and OPV2V datasets show that our DOtA outperforms state-of-the-art unsupervised 3D object detection methods. Additionally, we also validate the effectiveness of the DOtA labels under various collaborative perception frameworks. The code is available at https://github.com/xmuqimingxia/DOtA.

Human Motion Recovery (HMR) research mainly focuses on ground-based motions such as running. The study on capturing climbing motion, an off-ground motion, is sparse. This is partly due to the limited availability of climbing motion datasets, especially large-scale and challenging 3D labeled datasets. To address the insufficiency of climbing motion datasets, we collect AscendMotion, a large-scale well-annotated, and challenging climbing motion dataset. It consists of 412k RGB, LiDAR frames, and IMU measurements, including the challenging climbing motions of 22 skilled climbing coaches across 12 different rock walls. Capturing the climbing motions is challenging as it requires precise recovery of not only the complex pose but also the global position of climbers. Although multiple global HMR methods have been proposed, they cannot faithfully capture climbing motions. To address the limitations of HMR methods for climbing, we propose Climbing-Cap, a motion recovery method that reconstructs continuous 3D human climbing motion in a global coordinate system. One key insight is to use the RGB and LiDAR modalities to separately reconstruct motions in camera coordinates and global coordinates and to optimize them jointly. We demonstrate the quality of the AscendMotion dataset and present promising results from ClimbingCap. The AscendMotion dataset and source code release publicly at http://www.lidarhumanmotion.net/climbingcap/

Recently, sparsely-supervised 3D object detection has gained great attention, achieving performance close to fully-supervised 3D detectors while requiring only a few annotated instances. Nevertheless, these methods suffer challenges when accurate labels are extremely absent. In this paper, we propose a boosting strategy, termed SP3D, explicitly utilizing the cross-modal semantic prompts generated from Large Multimodal Models (LMMs) to boost the 3D detector with robust feature discrimination capability under sparse annotation settings. Specifically, we first develop a Confident Points Semantic Transfer (CPST) module that generates accurate cross-modal semantic prompts through boundary-constrained center cluster selection. Based on these accurate semantic prompts, which we treat as seed points, we introduce a Dynamic Cluster Pseudo-label Generation (DCPG) module to yield pseudo-supervision signals from the geometry shape of multi-scale neighbor points. Additionally, we design a Distribution Shape score (DS score) that chooses high-quality supervision signals for the initial training of the 3D detector. Experiments on the KITTI dataset and Waymo Open Dataset (WOD) have validated that SP3D can enhance the performance of sparsely supervised detectors by a large margin under meager labeling conditions. Moreover, we verified SP3D in the zero-shot setting, where its performance exceeded that of the state-of-the-art methods. The code is available at https://github.com/xmuqimingxia/SP3D.

Diffusion models have been widely adopted in image and language generation and are now being applied to reinforcement learning. However, the application of diffusion models in offline cooperative Multi-Agent Reinforcement Learning (MARL) remains limited. Although existing studies explore this direction, they suffer from scalability or poor cooperation issues due to the lack of design principles for diffusion-based MARL. The Individual-Global-Max (IGM) principle is a popular design principle for cooperative MARL. By satisfying this principle, MARL algorithms achieve remarkable performance with good scalability. In this work, we extend the IGM principle to the Individual-Global-identically-Distributed (IGD) principle. This principle stipulates that the generated outcome of a multi-agent diffusion model should be identically distributed as the collective outcomes from multiple individual-agent diffusion models. We propose DoF, a diffusion factorization framework for Offline MARL. It uses noise factorization function to factorize a centralized diffusion model into multiple diffusion models. We theoretically show that the noise factorization functions satisfy the IGD principle. Furthermore, DoF uses data factorization function to model the complex relationship among data generated by multiple diffusion models. Through extensive experiments, we demonstrate the effectiveness of DoF. The source code is available at [https://github.com/xmu-rl-3dv/DoF](https://github.com/xmu-rl-3dv/DoF).

Parameter-sharing (PS) techniques have been widely adopted in cooperative Multi-Agent Reinforcement Learning (MARL). In PS, all the agents share a policy network with identical parameters, which enjoys good sample efficiency. However, PS could lead to homogeneous policies that limit MARL performance. We tackle this problem from the angle of gradient conflict among agents. We find that the existence of futile neurons whose update is canceled out by gradient conflicts among agents leads to poor learning efficiency and diversity. To address this deficiency, we propose GradPS, a gradient-based PS method. It dynamically creates multiple clones for each futile neuron. For each clone, a group of agents with low gradient-conflict shares the neuron's parameters. Our method can enjoy good sample efficiency by sharing the gradients among agents of the same clone neuron. Moreover, it can encourage diverse behaviors through independently updating an exclusive clone neuron. Through extensive experiments, we show that GradPS can learn diverse policies with promising performance. The source code for GradPS is available in \url{https://github.com/xmu-rl-3dv/GradPS}.

Partial point attack approaches focus on leveraging the fewest points to achieve the best attack efficiency for easy implementation in the physical domain. For the first time, this paper proposes that the partial point attack strategy should pay attention to not only the selection and disturbance of points, but also the penetration of current defense methods. By re-examining characteristics of previous partial point attack approaches leading to performance improvement, we discover two fundamental principles: first, the selection of attacked points should consider not only the favourable visual salience but also the proper position concentration, thus to acquire effective structural destruction on the basis of remaining imperceptible; second, the perturbation of target points should form meaningful structures rather than outliers. To achieve this, we first propose a novel distributed-concentrated point selection (DPS) strategy, which is easier to concentrate salient points containing rich local information in a few tiny regions. Additionally, to enhance the penetration efficacy and real-time performance of attack point clouds against defenses, we further design a perturbation network based on the multi-scale penetration loss (L_msp), which can generate adversarial samples with as few outliers as possible only through a single forward propagation. Experimental results demonstrate that the real-time distributed-concentrated penetration attack (RDPA) framework can achieve state-of-the-art (SOTA) success rates by perturbing only 3.5% of points, and have the best penetration for mainstream defense methods such as SRS and SOR.

In recent years, oriented object detection has made significant advancements in computer vision. However, detecting objects with large aspect ratios is still challenging due to the following issues. Firstly, the convolution operation extracts features of object, but accompanied irrelevant background information. This information obscures the expression of useful features and limits accuracy of detection. Secondly, in the label assignment process, traditional method of defining positive and negative samples relies on the Euclidean distance. This method forms a circular equivalent range, which is irrespective of the object’s geometric shape and only adapts to square objects. To address these issues, we propose a geometry-aware framework, a Shape and Ellipse guided oriented Detector (SEDet), to detect large aspect ratios objects in remote sensing images. We made innovations in two aspects: feature extraction and label assignment process. For feature extraction, we propose a Shape-guided Matrix Feature Pyramid Grid (SMFPG). SMFPG uses adaptive sampling strides according to the size and angle of the objects for better feature alignment and extraction. For label assignment, we propose an Ellipse-guided Adaptive Label Assignment Strategy (EALAS). EALAS uses an ellipse distance associated with the shape to suppress sample misclassification in the label assignment process. Extensive experiments on multiple datasets prove the effectiveness of our approach. On DOTA v1.0, HRSC2016, and FAIR1M v1.0 datasets, we improved mAP by 2.43%, 2.19%, and 0.72%, respectively.

Change detection (CD) plays a crucial role in applications such as disaster monitoring, urban planning, and natural resource management. Recent advances in satellite imaging technology have led to the availability of bi-temporal images with very high resolution, making binary CD increasingly popular. Although deep learning methods have outperformed traditional approaches in CD tasks, most existing models rely on symmetric dual-branch encoders with shared parameters. However, these architectures may find it increasingly complex to further effectively capture essential CD features from bi-temporal images. To address this limitation, we propose an innovative CD information encoding structure called the Asymmetric Encoder Network (AENet). AENet comprises two asymmetric encoders and a CD information fusion module. Unlike traditional symmetric architectures, our approach enables different CD information to be selected from each other during the fusion process, capturing key change information that enhances the sensitivity of the network. Additionally, we introduce a feature extraction module for multi-scale spatial aggregation applied to this structure. Experimental results demonstrate that AENet achieves higher F1 scores on three publicly available datasets while maintaining moderate computational complexity and fewer parameters.

Edge detection in point clouds is a fundamental problem in 3D vision. Previous approaches based on point-based neural networks treat edge detection as a binary segmentation task, directly applying semantic segmentation frameworks. However, due to the intrinsic differences between edge detection and semantic segmentation, these methods have limitations compared to traditional methods that rely on hand-crafted features, as well as recent volumetric edge representation methods. To address these challenges, we propose three key constraints: Local Structure Disruption, Encoding Region Offset, and Information Loss in Feature Propagation. Based on these constraints, we introduce a novel Multi-Scale Structure-Aware Network (MSSA-Net). MSSA-Net is designed around local multi-scale neighborhoods, learning intact local structural features at different scales within highly relevant neighborhoods for each point. This strategy effectively avoids Local Structure Disruption and Encoding Region Offset. During feature learning, the MSSA Core module generates adaptive encoding branches for different scales of neighborhoods. The learned multi-scale local structural features are then concatenated to encode the local structural information of each point in the point cloud. MSSA-Net eliminates the need for propagating high-level features to low-level features, thereby preventing Information Loss in Feature Propagation. Extensive experiments demonstrate that our MSSA-Net surpasses existing works by a large margin and achieves state-of-the-art performance on various datasets.

Point cloud regression localization technology has a wide range of applications in the multimedia field. For example, in virtual reality and augmented reality, accurate point cloud localization can significantly enhance the user experience. Recently, point cloud pose regression algorithms based on APR (Absolute Pose Regression) and SCR (Scene Coordinate Regression) have achieved near sub-meter accuracy, requiring multiple repetitive trajectories for training. The key to their success lies in the diversity of viewpoints, temporal changes, and trajectories, which is resource-consuming. However, due to the errors in GPS/INS, the coupling between trajectories is not ideal, and the stability of re-localization is insufficient. Since LiDAR has covered most of the scene, single-shot localization has the potential to approach or even surpass multi-trajectory localization methods through pose enhancement. Specifically, we present Pose Enhancement Localization (PELoc), which feeds one trajectory, proposing SSDA (Single-shot Data Augmentation) and LTI (LiDAR Trajectories-coupled Interpolation) to simulate different driving poses, and we introduce KP-CL (Key Points Contrastive Learning) through feature perturbation to mitigate the differences in viewpoint/temporal phase transformations in similar scenes across different trajectories. Our algorithm has been tested on the Oxford, QE-Oxford, and NCLT datasets, where single-shot localization accuracy can approach near sub-meter level on QE-Oxford and NCLT. The code will be published in https://github.com/Eaton2022/PELoc.

Large Language Models (LLMs) are increasingly being explored across a range of reasoning tasks. However, LLMs sometimes struggle with reasoning tasks under uncertainty that are relatively easy for humans, such as planning actions in stochastic environments. The adoption of LLMs for reasoning is impeded by uncertainty challenges, such as LLM uncertainty and environmental uncertainty. LLM uncertainty arises from the stochastic sampling process inherent to LLMs. Most LLM-based Decision-Making (LDM) approaches address LLM uncertainty through multiple reasoning chains or search trees. However, these approaches overlook environmental uncertainty, which leads to poor performance in environments with stochastic state transitions. Some recent LDM approaches deal with uncertainty by forecasting the probability of unknown variables. However, they are not designed for multi-step reasoning tasks that require interaction with the environment. To address uncertainty in LLM decision-making, we introduce PlanU, an LLM-based planning method that captures uncertainty within Monte Carlo Tree Search (MCTS). PlanU models the return of each node in the MCTS as a quantile distribution, which uses a set of quantiles to represent the return distribution. To balance exploration and exploitation during tree search, PlanU introduces an Upper Confidence Bounds with Curiosity (UCC) score which estimates the uncertainty of MCTS nodes. Through extensive experiments, we demonstrate the effectiveness of PlanU in LLM-based reasoning tasks under uncertainty.

We tackle the challenge of LiDAR-based place recognition, which traditionally depends on costly and time-consuming prior 3D maps. To overcome this, we first construct LiRSI-XA dataset, which encompasses approximately $110,000$ remote sensing submaps and $13,000$ LiDAR point cloud submaps captured in urban scenes, and propose a novel method, L2RSI, for cross-view LiDAR place recognition using high-resolution Remote Sensing Imagery. This approach enables large-scale localization capabilities at a reduced cost by leveraging readily available overhead images as map proxies. L2RSI addresses the dual challenges of cross-view and cross-modal place recognition by learning feature alignment between point cloud submaps and remote sensing submaps in the semantic domain. Additionally, we introduce a novel probability propagation method based on particle estimation to refine position predictions, effectively leveraging temporal and spatial information. This approach enables large-scale retrieval and cross-scene generalization without fine-tuning. Extensive experiments on LiRSI-XA demonstrate that, within a $100km^2$ retrieval range, L2RSI accurately localizes $83.27\%$ of point cloud submaps within a $30m$ radius for top-$1$ retrieved location. Our project page is publicly available at https://shizw695.github.io/L2RSI/.

Prevailing scene coordinate regression methods for LiDAR localization suffer from localization ambiguities, as distinct locations can exhibit similar geometric signatures — a challenge that current geometry-based regression approaches have yet to solve. Recent vision–language models show that textual descriptions can enrich scene understanding, supplying potential localization cues missing from point cloud geometries. In this paper, we propose GTR-Loc, a novel text-assisted LiDAR localization framework that effectively generates and integrates geospatial text regularization to enhance localization accuracy. We propose two novel designs: a Geospatial Text Generator that produces discrete pose-aware text descriptions, and a LiDAR-Anchored Text Embedding Refinement module that dynamically constructs view-specific embeddings conditioned on current LiDAR features. The geospatial text embeddings act as regularization to effectively reduce localization ambiguities. Furthermore, we introduce a Modality Reduction Distillation strategy to transfer textual knowledge. It enables high-performance LiDAR-only localization during inference, without requiring runtime text generation. Extensive experiments on challenging large-scale outdoor datasets, including QEOxford, Oxford Radar RobotCar, and NCLT, demonstrate the effectiveness of GTR-Loc. Our method significantly outperforms state-of-the-art approaches, notably achieving a 9.64%/8.04% improvement in position/orientation accuracy on QEOxford. Our code is available at https://github.com/PSYZ1234/GTR-Loc.

Ubiquitous sensors and mobile devices have spurred the growth of Web-of-Things (WoT) services in smart cities, making accurate spatio-temporal traffic predictions increasingly crucial. Leveraging advances in deep learning, recent Spatio-Temporal Graph Neural Networks (STGNNs) have achieved remarkable results. However, these methods address scenario-specific spatio-temporal heterogeneity by designing model architectures, often overlooking the importance of selecting optimal spatio-temporal knowledge (i.e., model inputs). In this paper, we propose an automated framework for spatio-temporal knowledge optimization to address this challenge. Our framework seamlessly integrates with downstream models, enhancing their performance across various prediction tasks. Specifically, we design a knowledge search space composed of parameters that represent scenario-specific spatio-temporal correlations within data. Additionally, we employ a bandit-based multi-fidelity algorithm for knowledge optimization to solve the constraint of limited resource. Furthermore, we adopt a meta-learner to extract transferable meta-knowledge about optimal knowledge, facilitating efficient exploration of the search space. Extensive experiments on five widely used real-world datasets demonstrate the effectiveness of our proposed framework. To the best of our knowledge, we are the first to automatically optimize spatio-temporal knowledge for spatio-temporal traffic prediction.

In recent years, graph neural networks (GNNs) have been commonly utilized for social recommendation systems. However, real-world scenarios often present challenges related to user privacy and business constraints, inhibiting direct access to valuable social information from other platforms. While many existing methods have tackled matrix factorization-based social recommendations without direct social data access, developing GNN-based federated social recommendation models under similar conditions remains largely unexplored. To address this issue, we propose a novel vertical federated social recommendation method leveraging privacy-preserving two-party graph convolution networks (P4GCN) to enhance recommendation accuracy without requiring direct access to sensitive social information. First, we introduce a Sandwich-Encryption module to ensure comprehensive data privacy during the collaborative computing process. Second, we provide a thorough theoretical analysis of the privacy guarantees, considering the participation of both curious and honest parties. Extensive experiments on four real-world datasets demonstrate that P4GCN outperforms state-of-the-art methods in terms of recommendation accuracy.

Traditional 3D object detectors, whether fully-, semi-, or weakly-supervised, rely heavily on extensive human annotations. In contrast, this paper introduces an unsupervised 3D object detector that automatically discerns object patterns without such annotations. To achieve this, we propose a Commonsense Prototype-based Detector (CPD) for unsupervised 3D object detection. CPD first constructs Commonsense Prototypes (CProto) to represent the geometric center and size of objects. It then generates high-quality pseudo-labels and guides detector convergence using size and geometry priors from CProto. Building on CPD, we further introduce CPD++, an enhanced version that improves performance by leveraging motion cues. CPD++ learns localization from stationary objects and recognition from moving objects, facilitating the mutual transfer of localization and recognition knowledge between these two object types. Both CPD and CPD++ outperform existing state-of-the-art unsupervised 3D detectors. Furthermore, when trained on Waymo Open Dataset (WOD) and tested on KITTI, CPD++ achieves 89.25% 3D Average Precision (AP) on the moderate car class at a 0.5 IoU threshold, reaching 95.3% of the performance attained by fully supervised counterparts. These results underscore the significant advancements brought by our method.

Earth observation satellites can capture optical images under different temporal, climatic conditions, and platforms exhibit substantial differences in color and brightness, leading to poor visual experiences when synthesizing large-area optical satellite images. The related issue of color balancing has attracted considerable attention from researchers, yet challenges such as a lack of research data and sensitivity to model parameters persist. To address these problems, this article publishes a publicly open dataset and presents a semantic segmentation-enhanced color balance network (SECBNet). First, to mitigate the scarcity of research data, we develop a publicly available remote sensing image color balance dataset, Zhu Hai color balance image (ZHCBI), to support related research activities. Second, to improve semantic consistency between the color-balanced images and the target images, we design a dual-branch U-Net architecture guided by segmentation results and propose a novel segmentation feature loss function. Finally, to address issues of seams and unnatural transitions between blocks in segmented processing, we introduce a postprocessing module based on weighted averaging. We conducted comparative experiments and analyses with existing mainstream color balancing algorithms on the ZHCBI dataset. The results demonstrate that our proposed method achieves state-of-the-art color balancing quality, with significant improvement in visual effects and a higher peak signal-to-noise ratio (PSNR) (23.64 dB) compared with other mainstream methods.

Semantic segmentation of large-scale point clouds under weak supervision is challenging due to the limited annotations. Current methods typically rely on the implicit use of annotation information to supervise the network, thereby constraining the capacity to characterize features at annotation points. In this article, we propose a query points component reasoning (QPCR) framework, which enhances the utilization of annotation information by introducing them into the middle layer, thereby refining the features used for semantic segmentation. First, we propose the semantic annotations component code reasoning (SACC Reasoning) module, which uses annotations of query points to supervise the semantic information of gradual query points. The semantic annotations component code (SACC) Reasoning module can guide the learning of feature representations of semantic annotations at multiple scales. Second, we create the semantic primitive component code reasoning (SPCC Reasoning) module, which addresses the issue of supervising the network using existing annotation point information that may suppress active features. The semantic primitive component code (SPCC) Reasoning module infers semantic primitive labels of query points using the decoding layer and uses them to guide the corresponding coding layer in learning semantic primitives. Finally, our QPCR method achieves state-of-the-art (SOTA) results on public benchmark datasets. Specifically, even with only 1% of annotated points, our QPCR method achieves the highest mIoU accuracy, i.e., 65.4% on S3DIS Area 5, 56.4% on SensatUrban and 55.3% mIoU on SemanticKITTI, respectively.

3D object detection is an essential scene perception capability for autonomous vehicles. In intelligent transportation systems, autonomous vehicles require minimal inference latency to sense their surroundings in real-time. However, advanced 3D detection methods often suffer from high inference latency. This limits the real-time deployment of 3D detection models in the real world. To address this problem, this paper proposes a fast and deployable 3D object detection method from the LiDAR point cloud for autonomous driving, namedFade3D. Firstly, we propose a Lightweight Input Encoder (LIE) to extract the most critical features from point clouds. Then, we develop a Spatial Feature Enhancement BEV backbone (SFENet) that efficiently encodes geometry features into compact representations. Additionally, we design an IoU-aware Loss Re-weighting (ILR) that enhances performance by shifting more attention to hard samples. Leveraging LIE and SFENet, our approach is independent of point cloud density and number, achieving significant speed advantages in processing large-scale point clouds and being deployment-friendly. Extensive experiments on KITTI and Waymo Open Dataset (WOD) datasets comparing various baseline detectors demonstrate its universality and superiority. Specifically, our method demonstrates impressive real-time inference capabilities, achieving 51.5 Hz on an RTX3090 GPU and 12.4 Hz on a Jetson Orin embedded development board. Code will be available at https://github.com/wayyeah/Fade3D

Fire events threaten the safety of residents and the health of ecosystems in affected areas, and post-disaster recovery efforts also require a large investment of resources and time. In recent years, the rising frequency of fire events has motivated local governments to strengthen their monitoring and emergency response efforts. However, current fire event identification methods can only identify the presence of a fire, without the ability to distinguish its specific category. In addition, when a fire occurs, the lack of information about the affected areas makes it challenging for emergency management authorities to take timely and effective rescue measures. To address these issues, we propose a two-stage framework for fire event identification and assessment. Specifically, in the first stage, based on multi-band fused remote sensing images and heterogeneous environmental images, the proposed framework not only identifies various fire events but also accurately identifies the boundaries of the fire events. In the second stage, integrating the results of fire event identification with social media data and domain knowledge, we present a real-time assessment agent for fire events based on the large language model. This agent enables timely and accurate analysis of the impact of fires on the affected areas. We evaluate our method on a real-world authority dataset, and results show that our framework identifies fire events with an F1-score of 61.0$\%$and a mAP of 57.7$\%$, which outperforms state-of-the-art baseline methods. In addition, the assessment results of fire events in real cases indicate that the proposed fire event assessment agent can assist emergency responders in obtaining timely and accurate information.

Although 3D point cloud classification neural network models have been widely used, the in-depth interpretation of the activation of the neurons and layers is still a challenge. We propose a novel approach, named Relevance Flow, to interpret the hidden semantics of 3D point cloud classification neural networks. It delivers the class Relevance to the activated neurons in the intermediate layers in a back-propagation manner, and associates the activation of neurons with the input points to visualize the hidden semantics of each layer. Specially, we reveal that the 3D point cloud classification neural network has learned the plane-level and part-level hidden semantics in the intermediate layers, and utilize the normal and IoU to evaluate the consistency of both levels' hidden semantics. Besides, by using the hidden semantics, we generate the adversarial attack samples to attack 3D point cloud classifiers. Experiments show that our proposed method reveals the hidden semantics of the 3D point cloud classification neural network on ModelNet40 and ShapeNet, which can be used for the unsupervised point cloud part segmentation without labels and attacking the 3D point cloud classifiers.

LiDAR-based 3D object detection models inevitably struggle under rainy conditions due to the degraded and noisy scanning signals. Previous research has attempted to address this by simulating the noise from rain to improve the robustness of detection models. However, significant disparities exist between simulated and actual rain-impacted data points. In this work, we propose a novel rain simulation method, termed DRET, that unifies Dynamics and Rainy Environment Theory to provide a cost-effective means of expanding the available realistic rain data for 3D detection training. Furthermore, we present a Sunny-to-Rainy Knowledge Distillation (SRKD) approach to enhance 3D detection under rainy conditions. Extensive experiments on the Waymo-Open-Dataset show that, when combined with the state-of-the-art DSVT model and other classical 3D detectors, our proposed framework demonstrates significant detection accuracy improvements, without losing efficiency. Remarkably, our framework also improves detection capabilities under sunny conditions, therefore offering a robust solution for 3D detection regardless of whether the weather is rainy or sunny.

Point cloud registration, a fundamental task in 3D computer vision, has remained largely unexplored in cross-source point clouds and unstructured scenes. The primary challenges arise from noise, outliers, and variations in scale and density. However, neglected geometric natures of point clouds restricts the performance of current methods. In this paper, we propose a novel method termed SPEAL to leverage skeletal representations for effective learning of intrinsic topologies of point clouds, facilitating robust capture of geometric intricacy. Specifically, we design the Skeleton Extraction Module to extract skeleton points and skeletal features in an unsupervised manner, which is inherently robust to noise and density variances. Then, we propose the Skeleton-Aware GeoTransformer to encode high-level skeleton-aware features. It explicitly captures the topological natures and inter-point-cloud skeletal correlations with the noise-robust and density-invariant skeletal representations. Next, we introduce the Correspondence Dual-Sampler to facilitate correspondences by augmenting the correspondence set with skeletal correspondences. Furthermore, we construct a challenging novel cross-source point cloud dataset named KITTI CrossSource for benchmarking cross-source point cloud registration methods. Extensive quantitative and qualitative experiments are conducted to demonstrate our approach’s superiority and robustness on both cross-source and same-source datasets. To the best of our knowledge, our approach is the first to facilitate point cloud registration with skeletal geometric priors.

3D human pose estimation (3HPE) in large-scale outdoor scenes using commercial LiDAR has attracted significant attention due to its potential for real-life applications. However, existing LiDAR-based methods for 3HPE primarily rely on recovering 3D human poses from individual point clouds, and the coherence cues present in the neighborhood are not sufficiently harnessed. In this work, we explore spatial and contexture coherence cues contained in the neighborhood that lead to great performance improvements in 3HPE. Specifically, firstly, we deeply investigate the 3D neighbor in the background (3BN) which serves as a spatial coherence cue for inferring reliable motion since it provides physical laws to limit motion targets. Secondly, we introduce a novel 3D scanning neighbor (3SN) generated during the data collection and 3SN implies structural edge coherence cues. We use 3SN to overcome the degradation of performance and data quality caused by the sparsity-varying properties of LiDAR point clouds. In order to effectively model the complementation between these distinct cues and build consistent temporal relationships across human motions, we propose a new transformer-based module called the CoherenceFuse module. Extensive experiments were conducted on publicly available datasets, namely LidarHuman26M, CIMI4D, SLOPER4D and Waymo Open Dataset v2.0, showcase the superiority and effectiveness of our proposed method. In particular, when compared with LidarCap on the LidarHuman26M dataset, our method demonstrates a reduction of 7.08mm in the average MPJPE metric, along with a decrease of 16.55mm in the MPJPE metric for distances exceeding 25 meters. The code and models are available at https://github.com/jingyi-zhang/Neighborhood-enhanced-LidarCap.

Absolute pose regression (APR) estimates global pose in an end-to-end manner, achieving impressive results in learn-based LiDAR localization. However, compared to the top-performing methods reliant on 3D-3D correspondence matching, APR's accuracy still has room for improvement. We recognize APR's lack of robust features learning and iterative denoising process leads to suboptimal results. In this paper, we propose DiffLoc, a novel framework that formulates LiDAR localization as a conditional generation of poses. First, we propose to utilize the foundation model and static-object-aware pool to learn robust features. Second, we incorporate the iterative denoising process into APR via a diffusion model conditioned on the learned geometrically robust features. In addition, due to the unique nature of diffusion models, we propose to adapt our models to two additional applications: (1) using multiple inferences to evaluate pose uncertainty, and (2) seamlessly introducing geometric constraints on denoising steps to improve prediction accuracy. Extensive experiments conducted on the Oxford Radar RobotCar and NCLT datasets demonstrate that DiffLoc outperforms better than the state-of-the-art methods. Especially on the NCLT dataset, we achieve 35% and 34.7% improvement on position and orientation accuracy, respectively. Our code is released at https://github.com/liw95/DiffLoc.

It is challenging for Neural Radiance Fields (NeRFs) in the few-shot setting to reconstruct high-quality novel views and depth maps in 360° outward-facing indoor scenes. The captured sparse views for these scenes usually contain large viewpoint variations. This greatly reduces the potential consistency between views, leading NeRFs to degrade a lot in these scenarios. Existing methods usually leverage pre-trained depth prediction models to improve NeRFs. However, these methods cannot guarantee geometry consistency due to the inherent geometry ambiguity in the pretrained models, thus limiting NeRFs' performance. In this work, we present p2 NeRF to capture global and hierarchical geometry consistency priors from pretrained models, thus facilitating few-shot NeRFs in 360° outward-facing indoor scenes. On the one hand, we propose a matching-based geometry warm-up strategy to provide global geometry consistency priors for NeRFs. This effectively avoids the overfitting of early training with sparse inputs. On the other hand, we propose a group depth ranking loss and ray weight mask regularization based on the monocular depth estimation model. This provides hierarchical geometry consistency priors for NeRFs. As a result, our approach can fully leverage the geometry consistency priors from pretrained models and help few-shot NeRFs achieve state-of-the-art performance on two challenging indoor datasets. Our code is released at https://github.com/XT5un/P2NeRF.

The prevalent approaches of unsupervised 3D object de-tection follow cluster-based pseudo-label generation and iterative self-training processes. However, the challenge arises due to the sparsity of LiDAR scans, which leads to pseudo-labels with erroneous size and position, resulting in subpar detection performance. To tackle this problem, this paper introduces a Commonsense Prototype-based Detector, termed CPD, for unsupervised 3D object de-tection. CPD first constructs Commonsense Prototype (CProto) characterized by high-quality bounding box and dense points, based on commonsense intuition. Subse-quently, CPD refines the low-quality pseudo-labels by lever-aging the size prior from CProto. Furthermore, CPD en-hances the detection accuracy of sparsely scanned objects by the geometric knowledge from CProto. CPD outper-forms state-of-the-art unsupervised 3D detectors on Waymo Open Dataset (WOD), PandaSet, and KITTI datasets by a large margin. Besides, by training CPD on WOD and testing on KITTI, CPD attains 90.85% and 81.01% 3D Aver-age Precision on easy and moderate car classes, respectively. These achievements position CPD in close prox-imity to fully supervised detectors, highlighting the sig-nificance of our method. The code will be available at https://github.com/hailanyi/CPD.

Current sparsely-supervised object detection methods largely depend on high threshold settings to derive high-quality pseudo labels from detector predictions. However, hard instances within point clouds frequently display incomplete structures, causing decreased confidence scores in their assigned pseudo-labels. Previous methods inevitably result in inadequate positive supervision for these instances. To address this problem, we propose a novel Hard INsTance Enhanced Detector (HINTED), for sparsely-supervised 3D object detection. Firstly, we design a self-boosting teacher (SBT) model to generate more potential pseudo-labels, enhancing the effectiveness of information transfer. Then, we introduce a mixed-density student (MDS) model to concentrate on hard instances during the training phase, thereby improving detection accuracy. Our extensive experiments on the KITTI dataset validate our method's superior performance. Compared with leading sparsely-supervised methods, HINTED significantly improves the detection performance on hard instances, no-tably outperforming fully-supervised methods in detecting challenging categories like cyclists. HINTED also significantly outperforms the state-of-the-art semi-supervised method on challenging categories. The code is available at https://github.com/xmuqimingxia/HINTED.

Comprehensive capturing of human motions requires both accurate captures of complex poses and precise localization of the human within scenes. Most of the HPE datasets and methods primarily rely on RGB, LiDAR, or IMU data. However, solely using these modalities or a combination of them may not be adequate for HPE, particularly for complex and fast movements. For holistic human motion understanding, we present RELI11D, a high-quality multimodal human motion dataset involves LiDAR, IMU system, RGB camera, and Event camera. It records the motions of 10 actors performing 5 sports in 7 scenes, including 3.32 hours of synchronized LiDAR point clouds, IMU measurement data, RGB videos and Event steams. Through extensive experiments, we demonstrate that the RELI 11 D presents considerable challenges and opportunities as it contains many rapid and complex motions that require precise location. To address the challenge of integrating different modalities, we propose LEIR, a multimodal baseline that effectively utilizes LiDAR Point Cloud, Event stream, and RGB through our cross-attention fusion strategy. We show that LEIR exhibits promising results for rapid motions and daily motions and that utilizing the characteristics of multiple modalities can indeed improve HPE performance. Both the dataset and source code release publicly in http://www.lidarhumanmotion.net/reli11d/, fostering collaboration and enabling further exploration in this field.

LiDAR localization is a fundamental task in robotics and computer vision, which estimates the pose of a Li- DAR point cloud within a global map. Scene Coordinate Regression (SCR) has demonstrated state-of-the-art performance in this task. In SCR, a scene is represented as a neural network, which outputs the world coordinates for each point in the input point cloud. However, SCR treats all points equally during localization, ignoring the fact that not all objects are beneficial for localization. For exam-ple, dynamic objects and repeating structures often negatively impact SCR. To address this problem, we introduce LiSA, the first method that incorporates semantic aware-ness into SCR to boost the localization robustness and accuracy. To avoid extra computation or network parame-ters during inference, we distill the knowledge from a seg-mentation model to the original SCR network. Experi-ments show the superior performance of LiSA on standard LiDAR localization benchmarks compared to state-of-the- art methods. Applying knowledge distillation not only pre-serves high efficiency but also achieves higher localization accuracy than introducing extra semantic segmentation modules. We also analyze the benefit of semantic in-formation for LiDAR localization. Our code is released at https://github.com/Ybchun/LiSA.

3D synthetic-to-real unsupervised domain adaptive seg-mentation is crucial to annotating new domains. Self-training is a competitive approach for this task, but its performance is limited by different sensor sampling patterns (i.e., variations in point density) and incomplete training strate-gies. In this work, we propose a density-guided translator (DGT), which translates point density between domains, and integrates it into a two-stage self-training pipeline named DGT-ST. First, in contrast to existing works that simulta-neously conduct data generation and feature/output align-ment within unstable adversarial training, we employ the non-learnable DGT to bridge the domain gap at the in-put level. Second, to provide a well-initialized model for self-training, we propose a category-level adversarial net-work in stage one that utilizes the prototype to prevent neg-ative transfer. Finally, by leveraging the designs above, a domain-mixed self-training method with source-aware consistency loss is proposed in stage two to narrow the domain gap further. Experiments on two synthetic-to-real segmentation tasks (SynLiDAR → semanticKITTI and SynL- iDAR → semanticPOSS) demonstrate that DGT-ST outper-forms state-of-the-art methods, achieving 9.4% and 4.3% mIoU improvements, respectively. Code is available at https://github.com/yuan-zm/DGT-ST.

Image matching is a fundamental computer vision problem. While learning-based methods achieve state-of-the-art performance on existing benchmarks, they generalize poorly to in-the-wild images. Such methods typically need to train separate models for different scene types (e.g., indoor vs. outdoor) and are impractical when the scene type is unknown in advance. One of the underlying problems is the limited scalability of existing data construction pipelines, which limits the diversity of standard image matching datasets. To address this problem, we propose GIM, a self-training framework for learning a single generalizable model based on any image matching architecture using internet videos, an abundant and diverse data source. Given an architecture, GIM first trains it on standard domain-specific datasets and then combines it with complementary matching methods to create dense labels on nearby frames of novel videos. These labels are filtered by robust fitting, and then enhanced by propagating them to distant frames. The final model is trained on propagated data with strong augmentations. Not relying on complex 3D reconstruction makes GIM much more efficient and less likely to fail than standard SfM-and-MVS based frameworks. We also propose ZEB, the first zero-shot evaluation benchmark for image matching. By mixing data from diverse domains, ZEB can thoroughly assess the cross-domain generalization performance of different methods. Experiments demonstrate the effectiveness and generality of GIM. Applying GIM consistently improves the zero-shot performance of 3 state-of-the-art image matching architectures as the number of downloaded videos increases (Fig. 1 (a)); with 50 hours of YouTube videos, the relative zero-shot performance improves by 6.9% − 18.1%. GIM also enables generalization to extreme cross-domain data such as Bird Eye View (BEV) images of projected 3D point clouds (Fig. 1 (c)). More importantly, our single zero-shot model consistently outperforms domain-specific baselines when evaluated on downstream tasks inherent to their respective domains. The code will be released upon acceptance.

Collaborative fairness stands as an essential element in federated learning to encourage client participation by equitably distributing rewards based on individual contributions. Existing methods primarily focus on adjusting gradient allocations among clients to achieve collaborative fairness. However, they frequently overlook crucial factors such as maintaining consistency across local models and catering to the diverse requirements of high-contributing clients. This oversight inevitably decreases both fairness and model accuracy in practice. To address these issues, we propose FedSAC, a novel Federated learning framework with dynamic Submodel Allocation for Collaborative fairness, backed by a theoretical convergence guarantee. First, we present the concept of "bounded collaborative fairness (BCF)", which ensures fairness by tailoring rewards to individual clients based on their contributions. Second, to implement the BCF, we design a submodel allocation module with a theoretical guarantee of fairness. This module incentivizes high-contributing clients with high-performance submodels containing a diverse range of crucial neurons, thereby preserving consistency across local models. Third, we further develop a dynamic aggregation module to adaptively aggregate submodels, ensuring the equitable treatment of low-frequency neurons and consequently enhancing overall model accuracy. Extensive experiments conducted on three public benchmarks demonstrate that FedSAC outperforms all baseline methods in both fairness and model accuracy. We see this work as a significant step towards incentivizing broader client participation in federated learning. The source code is available at https://github.com/wangzihuixmu/FedSAC.

We introduce HmPEAR, a novel dataset crafted for advancing research in 3D Human Pose Estimation (3D HPE) and Human Action Recognition (HAR), with a primary focus on outdoor environments. This dataset offers a synchronized collection of imagery, LiDAR point clouds, 3D human poses, and action categories. In total, the dataset encompasses over 300,000 frames collected from 10 distinct scenes and 25 diverse subjects. Among these, 250,000 frames of data contain 3D human pose annotations captured using an advanced motion capture system and further optimized for accuracy. Furthermore, the dataset annotates 40 types of daily human actions, resulting in over 6,000 action clips. Through extensive experimentation, we have demonstrated the quality of HmPEAR and highlighted the challenges it presents to current methodologies. Additionally, we propose baselines leveraging sequential images and point clouds for 3D HPE and HAR, which underscore the mutual reinforcement between them, highlighting the potential for cross-task synergies. The dataset is available at http://www.lidarhumanmotion.net/hmpear.

Managing glass facade through unmanned and intel-ligent methods has become a crucial demand in city management. However, traditional methods fail to adequately convey the status of glass facades and lack clear standards for assessment. To address these issues, we propose a novel method that combines digital twin with multi-modal large model for the intelligent and unmanned management of glass facades. More specifically, we first define comprehensive information presentation requirements for glass facades maintenance, including the facade's appearance, IoT device status, and robot positioning, addressed through image-based 3D modeling, cloud integration, and infrared motion capture. Second, we utilize multi-modal large model fine-tuning and prompt learning to assimilate maintenance knowledge, en-hancing decision-making and planning for maintenance tasks, with outputs standardized to integrate seamlessly with the system. Finally, we develop a digital twin manage system which offers robust, reliable management of glass facades with a user-friendly interface. Real-world evaluation confirmed its excellence in information collection and response accuracy, validating its reliability and effectiveness in intelligent facade maintenance.

The advancement of federated learning (FL) has enabled model training in spatio-temporal graphs (STG) while addressing privacy concerns that restrict direct data sharing. However, capturing spatio-temporal dependencies in FL settings that involve subgraphs remains challenging. On one hand, nodes in a subgraph struggle to model spatio-temporal correlations across different subgraphs. On the other hand, temporal data (e.g., traffic flows) often exhibits distinct non-i.i.d. patterns across subgraphs. To address these challenges, we propose a novel framework called Federated Subgraph Learning over Spatio-Temporal Graphs (FedSTG) for effective traffic prediction. FedSTG leverages attention-based models and includes two key modules. First, the Isolated Nodes Connection module integrates and filters the cluster centers of node embeddings from other subgraphs to enhance local model training, thereby capturing spatio-temporal correlations between nodes across subgraphs. Second, the Personalized Model Aggregation module compute the similarities between clients pairwise using their local models and aggregates personalized models based on the similarities as aggregation weights, thereby ensuring the effective model aggregation. Extensive experiments on two attention-based models and three commonly used traffic prediction datasets demonstrate the superiority of FedSTG.

Dynamically deploying mobile base stations is cru-cial for meeting public network demands and optimizing utilization. Traditional dynamic deployment strategies focus on maxi-mizing coverage of Points of Interest (POls), often overlooking the dynamic network demands of these locations. To address this problem, we propose a reinforcement learning-based dynamic deployment method. More specifically, we first design a spatio- temporal modeling-based actor network to comprehensively cap-ture network demand dynamics and make decisions for the next locations. Second, we design an attention-based critic network to efficiently process information and more accurately estimate collaborative decision-making. Third, we formulate a heuristic reward function to align the objectives of maximizing cumulative service utilization and minimizing energy consumption. Finally, we develop a digital twin simulator to validate the proposed method. Experiments using real-world data demonstrate the effectiveness of our approach.

Spatio-temporal traffic prediction (STTP) plays a crucial role in the development of smart cities. Deep learning models have shown superior performance in traffic prediction, but their opacity and complexity of traffic data present challenges for researchers in tuning models. To tune models effectively, we propose a generalized error analysis pipeline and design a corresponding visualization system, STError-Copilot (Spatio-temporal Error Copilot). The pipeline analyzes multi-perspective spatio-temporal features to determine whether prediction errors originate from semantic or modeling levels, and subsequently tunes the model. STErrorCopilot provides a comprehensive data analysis solution, covering the entire workflow from data loading, processing and visualization to final tuning, delivering end-to-end services. We perform error analysis and tuning on two classic models using two real datasets, demonstrating that our method accurately identifies errors and provides appropriate tuning recommendations.

In this work, we study the dormant neuron phenomenon in multi-agent reinforcement learning value factorization, where the mixing network suffers from reduced network expressivity caused by an increasing number of inactive neurons. We demonstrate the presence of the dormant neuron phenomenon across multiple environments and algorithms, and show that this phenomenon negatively affects the learning process. We show that dormant neurons correlates with the existence of over-active neurons, which have large activation scores. To address the dormant neuron issue, we propose ReBorn, a simple but effective method that transfers the weights from over-active neurons to dormant neurons. We theoretically show that this method can ensure the learned action preferences are not forgotten after the weight-transferring procedure, which increases learning effectiveness. Our extensive experiments reveal that ReBorn achieves promising results across various environments and improves the performance of multiple popular value factorization approaches. The source code of ReBorn is available in \url{https://github.com/xmu-rl-3dv/ReBorn}.

Point cloud registration, a fundamental task in 3D vision, has achieved remarkable success with learning-based methods in outdoor environments. Unsupervised outdoor point cloud registration methods have recently emerged to circumvent the need for costly pose annotations. However, they fail to establish reliable optimization objectives for unsupervised training, either relying on overly strong geometric assumptions, or suffering from poor-quality pseudo-labels due to inadequate integration of low-level geometric and high-level contextual information. We have observed that in the feature space, latent new inlier correspondences tend to cluster around respective positive anchors that summarize features of existing inliers. Motivated by this observation, we propose a novel unsupervised registration method termed INTEGER to incorporate high-level contextual information for reliable pseudo-label mining. Specifically, we propose the Feature-Geometry Coherence Mining module to dynamically adapt the teacher for each mini-batch of data during training and discover reliable pseudo-labels by considering both high-level feature representations and low-level geometric cues. Furthermore, we propose Anchor-Based Contrastive Learning to facilitate contrastive learning with anchors for a robust feature space. Lastly, we introduce a Mixed-Density Student to learn density-invariant features, addressing challenges related to density variation and low overlap in the outdoor scenario. Extensive experiments on KITTI and nuScenes datasets demonstrate that our INTEGER achieves competitive performance in terms of accuracy and generalizability.

Cross-domain recommendation (CDR) offers a promising solution to the data sparsity problem by enabling knowledge transfer across source and target domains. However, many recent CDR models overlook crucial issues such as privacy as well as the risk of negative transfer (which negatively impact model performance), especially in multi-domain settings. To address these challenges, we propose FedGCDR, a novel federated graph learning framework that securely and effectively leverages positive knowledge from multiple source domains. First, we design a positive knowledge transfer module that ensures privacy during inter-domain knowledge transmission. This module employs differential privacy-based knowledge extraction combined with a feature mapping mechanism, transforming source domain embeddings from federated graph attention networks into reliable domain knowledge. Second, we design a knowledge activation module to filter out potential harmful or conflicting knowledge from source domains, addressing the issues of negative transfer. This module enhances target domain training by expanding the graph of the target domain to generate reliable domain attentions and fine-tunes the target model for improved negative knowledge filtering and more accurate predictions. We conduct extensive experiments on 16 popular domains of the Amazon dataset, demonstrating that FedGCDR significantly outperforms state-of-the-art methods.

Although transformer models are main network architectures for the delineation of roads from remote sensing imagery, they have critical limitations due to their regular patch mechanism and inefficiency in local information learning. To address these limitations for enhanced road extraction, this letter presents a novel double-branch architecture and random block coding transformer (DBARCT), with the following contributions. First, to improve local spatial details’ learning, we integrate transformer with convolutional neural network (CNN) into a novel dual-branch encoder-decoder architecture, such that the resulting model is efficient at learning both the local edge information and the global context information that are highly complementary for accurate road extraction. Second, to additionally augment the learning of global contextual information, we integrate the regular patching approach in traditional transformer models with a new irregular patching approach, such that it can better capture the global spatial information correlations that might be ignored by the regular patching approach. Third, an array of tests was carried out to meticulously scrutinize the efficacy of the fundamental elements of the suggested model. The empirical findings reveal that the intersection over union (IoU) metric attained by the proposed methodology on the LRSNY dataset stands at 88.53%, thereby corroborating the efficacy and preeminence of our approach in tasks related to road extraction.

Infrared small target detection (IRSTD) aims to identify small and faint targets amidst cluttered background in infrared images, which is vital for applications like maritime surveillance. Traditional methods struggle due to low signal-to-noise ratio (SNR) and contrast. However, recent CNN-based approaches show promise, leveraging deep learning’s strong modeling capabilities. In this letter, we propose a multilevel interactive enhanced network (MIE-Net). In MIE-Net, we use multiple backbones that have progressively decreasing numbers of blocks. Features transfer and information interaction are carried out between different backbones. We designed an attention mechanism-based feature filter (AFF) to reduce background noise interference by filtering the low-level features with high-level features. Furthermore, we proposed a global information enhancement module (GIEM), through which features are enhanced as they are delivered, while further mitigating the problem of small target loss. Experiments on public datasets validate the effectiveness of our method. MIE-Net outperforms the current state-of-the-art (SOTA) methods by approximately 6% in terms of the intersection over union (IoU). There was also about a 2% increase in average area under the curve (AUC).

We introduce HiSC4D, a novel Human-centered interaction and 4D Scene Capture method, aimed at accurately and efficiently creating a dynamic digital world, containing large-scale indoor-outdoor scenes, diverse human motions, rich human-human interactions, and human-environment interactions. By utilizing body-mounted IMUs and a head-mounted LiDAR, HiSC4D can capture egocentric human motions in unconstrained space without the need for external devices and pre-built maps. This affords great flexibility and accessibility for human-centered interaction and 4D scene capturing in various environments. Taking into account that IMUs can capture human spatially unrestricted poses but are prone to drifting for long-period using, and while LiDAR is stable for global localization but rough for local positions and orientations, HiSC4D employs a joint optimization method, harmonizing all sensors and utilizing environment cues, yielding promising results for long-term capture in large scenes. To promote research of egocentric human interaction in large scenes and facilitate downstream tasks, we also present a dataset, containing 8 sequences in 4 large scenes (200 to 5,000 [Formula: see text]), providing 36 k frames of accurate 4D human motions with SMPL annotations and dynamic scenes, 31k frames of cropped human point clouds, and scene mesh of the environment. A variety of scenarios, such as the basketball gym and commercial street, alongside challenging human motions, such as daily greeting, one-on-one basketball playing, and tour guiding, demonstrate the effectiveness and the generalization ability of HiSC4D. The dataset and code will be publicly available for research purposes.

Estimation of urban surface temperature is crucial for urban planning and emergency management. Due to the complexity of intracity structures, it is very challenging to acquire satisfied prediction errors of the land surface temperature (LST) at very high resolution, like 60-by-60 m. Considering this, we propose a low-cost method for generating urban point clouds via readily accessible city data. Then we design an efficient descriptor, geofeature distribution matrix (GFDM) to describe the complex intracity structure. Using GFDM, we introduce a 3-D urban structure guided temperature prediction network (3D-UP Net) to capture the complex relationship between urban structure, upper atmospheric conditions, and surface temperature. The proposed 3D-UP Net is generalizable, capable of predicting future surface temperature for existing cities and even for those that are planned. Experiments conducted in multiple regions of China demonstrate that our method’s error is less than 1.5 K (in most cases) at a high resolution (60-by-60 m).

This paper tackles the problem of requiring a large amount of data annotation in LiDAR point cloud semantic segmentation (PCSS) task by proposing OPOCA, a weakly supervised network that only annotates one point per class in a single LiDAR scan. To compensate for the supervisory losses due to extremely few annotated labels, a large number of pseudo labels is first generated using a Pseudo Label Spreading Module (PLSM), whereas the potential ambiguity and inaccuracy is further addressed by a carefully-designed Spread Distance Loss (SDL) and a Range Image Auxiliary Module (RIAM). Moreover, we propose an iterative Self Training Module (STM) to increase the high-quality pseudo labels for the next round of training. Extensive experiments on various benchmark datasets (SemanticKITTI, Waymo Open dataset, and SemanticPOSS) demonstrate the rationality of each module and the superior performance of the proposed network over the current baseline with below 0.1 ‰ labels.

Object detection for remote sensing images (ODRSI) is an important task in computer vision. Effective algorithms inspired by oriented object detection have been proposed recently. However, a major challenge still remains. Different categories of objects may be similar under different scales, causing cross-scale confusion. Different from natural images, remote sensing images have a consistent scale within the same image, which is generally referred to as Ground Sample Distance (GSD). In this paper, we show, that GSD can be utilized to address the cross-scale confusion problem, and effectively boost the performance of ODRSI. Specifically, we propose GSDDet, which embeds the deep features that represent GSD constraints to decrease the cross-scale confusion between different object categories. In GSDDet, a deep GSD classification network is first designed to extract the GSD deep features from remote sensing images. Then, the GSD deep feature is coupled with an attention framework to detect multiple categories of objects. Due to the simplicity of our framework, GSDDet can be applied to improve both one-stage and two-stage methods. Experiments demonstrate that GSDDet outperforms state-of-the-art methods on challenging benchmarks, including DOTA-v1.0, DOTA-v1.5, and HRSC2016. The source code will be released upon publication.

Urban events, such as hurricanes, floods, and epidemic outbreaks, usually have a significant impact on crowd behaviors that may dramatically change people’s mobility patterns, interaction manners, and etc. Accurately foreseeing the abnormal crowd behaviors triggered by urban events can help authorities to dynamically schedule urban resources and services, such as providing shuttle buses, temporal shelters, and psychological first aid. However, traditional crowd behavior prediction models tend to learn regular trends and patterns of crowd traffic, and usually fail to predict the abnormal traffic fluctuations triggered by urban events. In this work, we propose a context-aware framework to accurately predict event-triggered abnormal crowd traffic by explicitly modeling event contexts and their impact. We start by training a state-of-the-art spatiotemporal multi-graph model (MG) leveraging a multi-graph convolutional network (MGCN) to model city-wide venues and their connections, as well as gated recurrent unit (GRU) to capture the crowd traffic patterns within venues. In order to model the time-occasional impact of sudden influential event context (e.g., weather conditions), we propose a multi-task fusion diagram to co-train the MG model by learning a temporal impact embedding with an attention mechanism and recurrent neural network to obtain the MG-MT model. In order to model the hyper-space impact of irregularly distributed event context (e.g., epicenters and trajectories), we propose a multi-view fusion diagram to fine-tune the above-mentioned model by learning a spatial impact graph with a stimulus-response mechanism to obtain the spatiotemporal crowd traffic model (STORM). Experiments using real-world urban data collected from Xiamen, China show that our approach improves over the state-of-the-art baseline by more than 6% in predicting crowd traffic and by more than 40% in the abnormal crowd traffic part.

Absolute pose regression has shown great potential in LiDAR localization, which learns to regress 6-DoF LiDAR poses through deep networks. However, recent regression methods suffer from scene ambiguities in challenging scenarios, leading to inaccurate and unstable localization. Inspired by neurobiological localization mechanisms, i.e., the firing mechanism of place cells, head-direction cells, and grid cells in mammalian brains, we propose a novel LiDAR localization framework called NIDALoc to achieve more robust and accurate results. First, we propose a Hebbian memory module, motivated by place cells, to preserve historical information, which helps refine local view features to reduce scene ambiguities. Specifically, the memory module stores scene information and then recalls it when revisiting an old place. Second, we propose a novel pose constrained framework, consisting of an orientation classification task and a grid center regression task, to regularize orientation and position estimation, respectively. The framework based on head-direction cells and grid cells constrains the absolute pose regression to reduce wrong predictions. Extensive experiments on two outdoor datasets demonstrate the effectiveness of NIDALoc, which outperforms state-of-the-art localization methods, especially in large-scale challenging scenes. The source code is available on the project website at https://github.com/PSYZ1234/NIDALoc.

Autonomous driving safety based on LiDAR perception is increasingly becoming a hot spot. Specifically, 3D adversarial examples always make the prediction results of deep neural network models unpredictable, which poses a major security risk to autonomous driving systems. However, the vulnerability of 3D neural network models to adversarial examples is less explored. At present, existing adversarial attack methods often obtain 3D adversarial examples by perturbing the entire point cloud, which requires a large number of perturbed points, i.e. requires a large perturbation budget. In this paper, we propose a Salient Region Adversarial attack method (SR-Adv) to generate adversarial point clouds, by perturbing fewer regions and fewer points. To our knowledge, we are the first to propose region-based attacks for 3D point clouds. First, the proposed SR-Adv employs game theory to extract salient regions of point clouds. This mechanism assigns a value to each region to measure its importance to the 3D neural network model prediction results and realizes the vulnerability analysis of the 3D model. Second, we propose a novel optimization-based gradient attack algorithm to achieve adversarial attacks on salient regions. We evaluate the proposed SR-Adv attack method on the synthetic datasets ModelNet40 and ShapeNetPart as well as the real-world dataset KITTI and NuScenes. Experimental results show that the proposed SR-Adv achieves a state-of-the-art attack success rate and better imperceptibility by perturbing fewer points on 3D point clouds.

Recent studies have shifted their focus towards formulating traffic forecasting as a spatio-temporal graph modeling problem. Typically, they constructed a static spatial graph at each time step and then connected each node with itself between adjacent time steps to create a spatio-temporal graph. However, this approach failed to explicitly reflect the correlations between different nodes at different time steps, thus limiting the learning capability of graph neural networks. Additionally, those models overlooked the dynamic spatio-temporal correlations among nodes by using the same adjacency matrix across different time steps. To address these limitations, we propose a novel approach called Spatio-Temporal Joint Graph Convolutional Networks (STJGCN) for accurate traffic forecasting on road networks over multiple future time steps. Specifically, our method encompasses the construction of both pre-defined and adaptive spatio-temporal joint graphs (STJGs) between any two time steps, which represent comprehensive and dynamic spatio-temporal correlations. We further introduce dilated causal spatio-temporal joint graph convolution layers on the STJG to capture spatio-temporal dependencies from distinct perspectives with multiple ranges. To aggregate information from different ranges, we propose a multi-range attention mechanism. Finally, we evaluate our approach on five public traffic datasets and experimental results demonstrate that STJGCN is not only computationally efficient but also outperforms 11 state-of-the-art baseline methods.

While federated learning has shown strong results in opti- mizing a machine learning model without direct access to the original data, its performance may be hindered by in- termittent client availability which slows down the conver- gence and biases the final learned model. There are significant challenges to achieve both stable and bias-free training un- der arbitrary client availability. To address these challenges, we propose a framework named Federated Graph-based Sam- pling (FEDGS), to stabilize the global model update and mitigate the long-term bias given arbitrary client availabil- ity simultaneously. First, we model the data correlations of clients with a Data-Distribution-Dependency Graph (3DG) that helps keep the sampled clients data apart from each other, which is theoretically shown to improve the approximation to the optimal model update. Second, constrained by the far- distance in data distribution of the sampled clients, we fur- ther minimize the variance of the numbers of times that the clients are sampled, to mitigate long-term bias. To validate the effectiveness of FEDGS, we conduct experiments on three datasets under a comprehensive set of seven client availability modes. Our experimental results confirm FEDGS’s advantage in both enabling a fair client-sampling scheme and improving the model performance under arbitrary client availability. Our code is available at https://github.com/WwZzz/FedGS.

3D object detection received increasing attention in autonomous driving recently. Objects in 3D scenes are distributed with diverse orientations. Ordinary detectors do not explicitly model the variations of rotation and reflection transformations. Consequently, large networks and extensive data augmentation are required for robust detection. Recent equivariant networks explicitly model the transformation variations by applying shared networks on multiple transformed point clouds, showing great potential in object geometry modeling. However, it is difficult to apply such networks to 3D object detection in autonomous driving due to its large computation cost and slow reasoning speed. In this work, we present TED, an efficient Transformation-Equivariant 3D Detector to overcome the computation cost and speed issues. TED first applies a sparse convolution backbone to extract multi-channel transformation-equivariant voxel features; and then aligns and aggregates these equivariant features into lightweight and compact representations for high-performance 3D object detection. On the highly competitive KITTI 3D car detection leaderboard, TED ranked 1st among all submissions with competitive efficiency. Code is available at https://github.com/hailanyi/TED.

We present SLOPER4D, a novel scene-aware dataset collected in large urban environments to facilitate the research of global human pose estimation (GHPE) with human-scene interaction in the wild. Employing a head-mounted device integrated with a LiDAR and camera, we record 12 human subjects' activities over 10 diverse urban scenes from an egocentric view. Frame-wise annotations for 2D key points, 3D pose parameters, and global translations are provided, together with reconstructed scene point clouds. To obtain accurate 3D ground truth in such large dynamic scenes, we propose a joint optimization method to fit local SMPL meshes to the scene and fine-tune the camera calibration during dynamic motions frame by frame, resulting in plausible and scene-natural 3D human poses. Even-tually, SLOPER4D consists of 15 sequences of human motions, each of which has a trajectory length of more than 200 meters (up to 1,300 meters) and covers an area of more than 200 m2(up to 30,000 m2), including more than 100k LiDAR frames, 300k video frames, and 500k IMU-based motion frames. With SLOPER4D, we provide a detailed and thorough analysis of two critical tasks, including camera-based 3D HPE and LiDAR-based 3D HPE in urban environments, and benchmark a new task, GHPE. The in-depth analysis demonstrates SLOPER4D poses significant challenges to existing methods and produces great research opportunities. The dataset and code are released at http://www.lidarhumanmotion.net/sloper4d/.

LiDAR-based absolute pose regression estimates the global pose through a deep network in an end-to-end manner, achieving impressive results in learning-based localization. However, the accuracy of existing methods still has room to improve due to the difficulty of effectively encoding the scene geometry and the unsatisfactory quality of the data. In this work, we propose a novel LiDAR localization frame-work, SGLoc, which decouples the pose estimation to point cloud correspondence regression and pose estimation via this correspondence. This decoupling effectively encodes the scene geometry because the decoupled correspondence regression step greatly preserves the scene geometry, leading to significant performance improvement. Apart from this decoupling, we also design a tri-scale spatial feature aggregation module and inter-geometric consistency constraint loss to effectively capture scene geometry. Moreover, we empirically find that the ground truth might be noisy due to GPS/INS measuring errors, greatly reducing the pose estimation performance. Thus, we propose a pose quality evaluation and enhancement method to measure and correct the ground truth pose. Extensive experiments on the Oxford Radar RobotCar and NCLT datasets demonstrate the effectiveness of SGLoc, which outperforms state-of-the-art regression-based localization methods by 68.5% and 67.6% on position accuracy, respectively.

Recently, virtuall pseudo-point-based 3D object detection that seamlessly fuses RGB images and LiDAR data by depth completion has gained great attention. However, virtual points generated from an image are very dense, introducing a huge amount of redundant computation during detection. Meanwhile, noises brought by inaccurate depth completion significantly degrade detection precision. This paper proposes a fast yet effective backbone, termed Vir-ConvNet, based on a new operator VirConv (Virtual Sparse Convolution), for virtual-point-based 3D object detection. VirConv consists of two key designs: (1) StVD (Stochastic Voxel Discard) and (2) NRConv (Noise-Resistant Sub-manifold Convolution). StVD alleviates the computation problem by discarding large amounts of nearby redundant voxels. NRConv tackles the noise problem by encoding voxel features in both 2D image and 3D LiDAR space. By integrating VirConv, we first develop an efficient pipeline VirConv-L based on an early fusion design. Then, we build a high-precision pipeline Vir Conv-T based on a transformed refinement scheme. Finally, we develop a semi-supervised pipeline VirConv-S based on a pseudo-label framework. On the KITTI car 3D detection test leader-board, our VirConv-L achieves 85% AP with a fast running speed of 56ms. Our VirConv-T and VirConv-S attains a high-precision of 86.3% and 87.2% AP, and currently rank 2ndand 1st11On the date of CVPR deadline, i. e., Nov.11, 2022, respectively. The code is available at https://github.com/hailanyi/VirConv.

Motion capture is a long-standing research problem. Although it has been studied for decades, the majority of research focus on ground-based movements such as walking, sitting, dancing, etc. Off- grounded actions such as climbing are largely overlooked. As an important type of action in sports and firefighting field, the climbing movements is challenging to capture because of its complex back poses, intricate human-scene interactions, and difficult global localization. The research community does not have an indepth understanding of the climbing action due to the lack of specific datasets. To address this limitation, we collect CIMI4D, a large rock Climbing Motion dataset from 12 persons climbing 13 different climbing walls. The dataset consists of around 180,000 frames of pose inertial measurements, LiDAR point clouds, RGB videos, high-precision static point cloud scenes, and reconstructed scene meshes. Moreover, we frame-wise annotate touch rock holds to facilitate a detailed exploration of human-scene interaction. The core of this dataset is a blending optimization process, which corrects for the pose as it drifts and is affected by the magnetic conditions. To evaluate the merit of CIMI4D, we perform four tasks which include human pose estimations (with/without scene constraints), pose prediction, and pose generation. The experimental results demonstrate that CIMI4D presents great challenges to existing methods and enables extensive research opportunities. We share the dataset with the research community in http://www.lidarhumanmotion.net/cimi4d/.

Recently, 3D object detection with sparse annotations has received great attention. However, current detectors usually perform poorly under very limited annotations. To address this problem, we propose a novel Contrastive Instance feature mining method, named CoIn. To better identify indistinguishable features learned through limited supervision, we design a Multi-Class contrastive learning module (MCcont) to enhance feature discrimination. Meanwhile, we propose a feature-level pseudo-label mining framework consisting of an instance feature mining module (InF-Mining) and a Labeled-to-Pseudo contrastive learning module (LPcont). These two modules exploit latent instances in feature space to supervise the training of detectors with limited annotations. Extensive experiments with KITTI dataset, Waymo open dataset, and nuScenes dataset show that under limited annotations, our method greatly improves the performance of baseline detectors: CenterPoint, Voxel-RCNN, and CasA. Combining CoIn with an iterative training strategy, we propose a CoIn++ pipeline, which requires only 2% annotations in the KITTI dataset to achieve performance comparable to the fully supervised methods. The code is available at https://github.com/xmuqimingxia/CoIn.

Image-to-Color point cloud registration establishes a connection between two-dimensional image data and three-dimensional point cloud data, and plays a vital role in the intelligent city, autonomous driving, and robotics field. However, it is still challenging to automatically register an image and its surrounding color point clouds together, due to the special application scenario, unknown camera intrinsic parameters, lack of train data, and a large amount of noise. To overcome those issues, we propose an iterative reprojection architecture that automatically acquires the matched 2D-3D keypoints pairs between the image and the color point clouds by graph optimization method and mapping transfer first, then completes registration by Alternating Direction Method of Multipliers (ADMM). Experiments results show that the proposed method is more accurate than the manual way.

Event cameras have emerged as a promising vision sensor in recent years due to their unparalleled temporal resolution and dynamic range. While registration of 2D RGB images to 3D point clouds is a long-standing problem in computer vision, no prior work studies 2D-3D registration for event cameras. To this end, we propose E2PNet, the first learning-based method for event-to-point cloud registration. The core of E2PNet is a novel feature representation network called Event-Points-to-Tensor (EP2T), which encodes event data into a 2D grid-shaped feature tensor. This grid-shaped feature enables matured RGB-based frameworks to be easily used for event-to-point cloud registration, without changing hyper-parameters and the training procedure. EP2T treats the event input as spatio-temporal point clouds. Unlike standard 3D learning architectures that treat all dimensions of point clouds equally, the novel sampling and information aggregation modules in EP2T are designed to handle the inhomogeneity of the spatial and temporal dimensions. Experiments on the MVSEC and VECtor datasets demonstrate the superiority of E2PNet over hand-crafted and other learning-based methods. Compared to RGB-based registration, E2PNet is more robust to extreme illumination or fast motion due to the use of event data. Beyond 2D-3D registration, we also show the potential of EP2T for other vision tasks such as flow estimation, event-to-image reconstruction and object recognition. The source code can be found at: https://github.com/Xmu-qcj/E2PNet.

Multi-agent systems are characterized by environmental uncertainty, varying policies of agents, and partial observability, which result in significant risks. In the context of Multi-Agent Reinforcement Learning (MARL), learning coordinated and decentralized policies that are sensitive to risk is challenging. To formulate the coordination requirements in risk-sensitive MARL, we introduce the Risk-sensitive Individual-Global-Max (RIGM) principle as a generalization of the Individual-Global-Max (IGM) and Distributional IGM (DIGM) principles. This principle requires that the collection of risk-sensitive action selections of each agent should be equivalent to the risk-sensitive action selection of the central policy. Current MARL value factorization methods do not satisfy the RIGM principle for common risk metrics such as the Value at Risk (VaR) metric or distorted risk measurements. Therefore, we propose RiskQ to address this limitation, which models the joint return distribution by modeling quantiles of it as weighted quantile mixtures of per-agent return distribution utilities. RiskQ satisfies the RIGM principle for the VaR and distorted risk metrics. We show that RiskQ can obtain promising performance through extensive experiments. The source code of RiskQ is available in https://github.com/xmu-rl-3dv/RiskQ.

Spatio-temporal kriging is an important problem in web and social applications, such as Web or Internet of Things, where things (e.g., sensors) connected into a web often come with spatial and temporal properties. It aims to infer knowledge for (the things at) unobserved locations using the data from (the things at) observed locations during a given time period of interest. This problem essentially requires inductive learning. Once trained, the model should be able to perform kriging for different locations including newly given ones, without retraining. However, it is challenging to perform accurate kriging results because of the heterogeneous spatial relations and diverse temporal patterns. In this paper, we propose a novel inductive graph representation learning model for spatio-temporal kriging. We first encode heterogeneous spatial relations between the unobserved and observed locations by their spatial proximity, functional similarity, and transition probability. Based on each relation, we accurately aggregate the information of most correlated observed locations to produce inductive representations for the unobserved locations, by jointly modeling their similarities and differences. Then, we design relation-aware gated recurrent unit (GRU) networks to adaptively capture the temporal correlations in the generated sequence representations for each relation. Finally, we propose a multi-relation attention mechanism to dynamically fuse the complex spatio-temporal information at different time steps from multiple relations to compute the kriging output. Experimental results on three real-world datasets show that our proposed model outperforms state-of-the-art methods consistently, and the advantage is more significant when there are fewer observed locations. Our code is available at https://github.com/zhengchuanpan/INCREASE.

Unstructured nature of 3D point clouds in large scenes is a challenging problem to effectively learn local geometric structures for point cloud semantic segmentation. To address this issue, we proposed a Feature Graph Convolution Network with Attentive Fusion (FGC-AFNet) in this letter. Our method takes large point clouds as input and uses the Feature Graph Convoluton (FGC) module to construct a graph of the central point with its neighboring points to extract local features. Then, we reduced the number of points using Random Sampling (RS) to expand the receptive field gradually to obtain multi-level features. The network also employs a dual Attention Fusion (AF) mechanism for efficient feature aggregation. One is at different levels for semantic feature fusion, another is for narrowing the semantic feature gap between the encoder and decoder. Compared to state-of-the-art methods on the S3DIS and Toronto3D datasets, our method obtained competitive results, with an overall accuracy of 88.6% and 96.58%, and a mean intersection over union of 71.2% and 81.92% on S3DIS and Toronto3D, respectively.

Existing point cloud modeling datasets primarily express the modeling precision by pose or trajectory precision rather than the point cloud modeling effect itself. Under this demand, we first independently construct a set of LiDAR system with an optical stage, and then we build a HPMB dataset based on the constructed LiDAR system, a High-Precision, Multi-Beam, real-world dataset. Second, we propose an modeling evaluation method based on HPMB for object-level modeling to overcome this limitation. In addition, the existing point cloud modeling methods tend to generate continuous skeletons of the global environment, hence lacking attention to the shape of complex objects. To tackle this challenge, we propose a novel learning-based joint framework, DSMNet, for high-precision 3D surface modeling from sparse point cloud frames. DSMNet comprises density-aware Point Cloud Registration (PCR) and geometry-aware Point Cloud Sampling (PCS) to effectively learn the implicit structure feature of sparse point clouds. Extensive experiments demonstrate that DSMNet outperforms the state-of-the-art methods in PCS and PCR on Multi-View Partial Point Cloud (MVP) database. Furthermore, the experiments on the open source KITTI and our proposed HPMB datasets show that DSMNet can be generalized as a post-processing of Simultaneous Localization And Mapping (SLAM), thereby improving modeling precision in environments with sparse point clouds.

This paper presents a Blur Resist Generative Adversarial Network (GAN) (BrGAN) with multiple joint dilated residual convolutions for chlorophyll image restoration of the Geostationary Ocean Color Imager (GOCI). First, a publicly available dataset was built to support this study. Second, a multiple attention perception mechanism and a multiple joint dilated residual convolution module was proposed to cope with the challenge of large missing areas in GOCI chlorophyll images. Third, a patch GAN based discrimination module was proposed to avoid the restored areas with generating mosaic and shadows. Our experimental results demonstrate that the BrGAN can reach 37.06 in the peak signal-to-noise ratio (PSNR) and 0.0485 in the Learned Perceptual Image Patch Similarity (LPIPS), respectively. The comparative study shows that the BrGAN achieves the highest effectiveness and advancement among other seven state-of-the-art methods.

Compared to fully supervised 3D large-scale point cloud segmentation methods, which necessitate extensive manual point-wise annotations, weakly supervised segmentation has emerged as a popular approach for significantly reducing labeling costs while maintaining effectiveness. However, the existing methods have exhibited inferior segmentation performance and unsatisfactory generalization capabilities in some scenarios with unique structures (e.g., building facades). In this paper, we propose an effective and generalized weakly supervised semantic segmentation framework, called multi-stage scene-level constraints (MSC), to solve the above problem. To address the issue regarding inadequate labeled data, we use pseudo-labels for unlabeled data and propose an uncertainty-guided adaptive reweighting strategy to reduce the negative impact of erroneous pseudo-labeled data on the model learning process. To address the class imbalance issue, we employ multi-stage scene-level constraints (i.e., encoder, decoder, and classifier stages) to treat each class equally and improve perception ability of the model for each class. Evaluations conducted on multiple large-scale point cloud datasets collected in different scenarios, including building facades, indoor scenes, outdoor scenes, and UAV scenes, show that our MSC achieves a large gain over the existing weakly supervised methods and even surpasses some fully supervised methods.

Semantic segmentation of building facade point clouds has diverse applications. The development of semantic segmentation methods is inextricably linked to datasets. The available building facade datasets suffer from a lack of abundant semantic categories and data completeness. To compensate for these shortcomings, we propose a new building facade dataset characterized by various categories and relatively complete 3-D building facades. In addition, most existing methods focus on fully supervised learning, which relies on manually labeling large-scale point cloud data and results in high time and labor costs. In this article, we propose an effective weakly supervised building facade segmentation approach, called spatial adaptive fusion consistency contrastive constraint (SAF-C3), to solve the above problem. We first design a multirandom point cloud augmentor as an auxiliary supervision branch to enhance the learning ability of the original network branch. Then, we present a spatial adaptive fusion (SAF) module to extract discriminative features for building facade point clouds. Finally, we propose a spatial consistency contrastive constraint to explore the contrastive property in feature space and to ensure the predictive consistency among the augmentation and original branches. The proposed method achieves a significant performance improvement against the state-of-the-art methods on two building facade point cloud datasets through extensive experiments. In particular, the performance of SAF-C3 with 1% labels significantly surpasses the baseline network with 100% labels.

The networks are required to be capable of learning low-level features well when applied to remote sensing image semantic segmentation task. To capture accurate and abundant low-level semantic information, the early feature extractor layer is crucial to the whole network, because all the subsequent features are inferred from that base. To address the low-level feature extraction issue and overcome the shortcomings of traditional convolution like too many parameters or limited receptive field, some novel convolution units have been proposed in the literature. In this paper, we propose two elaborately designed and portable yet effective convolution units, i.e., directional convolution (DC) and large field convolution (LFC), combined as the extractor of low-level semantic features. DC is designed to extract directional features from specific directions, and LFC can achieve a large receptive field with few parameters. Experiment results on two public datasets provide evidence that our convolution units can help deep learning networks improve performance stably and comprehensively compared to the baseline networks.

Unsupervised domain adaptation (UDA) segmentation aims to leverage labeled source data to make accurate predictions on unlabeled target data. The key is to make the segmentation network learn domain-invariant representations. In this work, we propose a prototype-guided multitask adversarial network (PMAN) to achieve this. First, we propose an intensity-aware segmentation network (IAS-Net) that leverages the private intensity information of target data to substantially facilitate feature learning of the target domain. Second, the category-level cross-domain feature alignment strategy is introduced to flee the side effects of global feature alignment. It employs the prototype (class centroid) and includes two essential operations: 1) build an auxiliary nonparametric classifier to evaluate the semantic alignment degree of each point based on the prediction consistency between the main and auxiliary classifiers and 2) introduce two class-conditional point-to-prototype learning objectives for better alignment. One is to explicitly perform category-level feature alignment in a progressive manner, and the other aims to shape the source feature representation to be discriminative. Extensive experiments reveal that our PMAN outperforms state-of-the-art results on two benchmark datasets.

The contours, one of the most significant human perceptual features, have a significant impact on point cloud processing. In urban scenes, contour extraction is quite challenging due to the enormous number of unstructured and irregular points (typically greater than 107points). In this paper, we propose a Large-scale 3D point cloud Contour Extraction Network (LCE-NET) to generate contours consistent with human perception of outdoor scenes. To our knowledge, it is the first time that an end-to-end learning-based framework has been proposed for contour extraction on point cloud at this scale. The proposed LCE-Net is essentially a two-phase system. In the first phase, potential vertexes are detected from the input point cloud by the vertex detection module, then in the second phase, a designed overcomplete line proposal set is generated, and invalid line segments are further suppressed by the line proposal discrimination module. The two phases are jointly trained by a uniform loss function to promote the information interchange, thus leading to extraction results with satisfied accurate and false alarm ratings. Since there is hardly any available dataset with labeled contours for the large-scale outdoor scene, we open sourced SemanticLine, the first dataset for large-scale point clouds with labeled contour information, based on re-annotation of previous mapping level point cloud dataset semantic3D. Experimental results demonstrate that LCE-NET can effectively extract parametric contour lines from large-scale point clouds of urban scenes. Additionally, it outperforms the state-of-the-art approaches. The code will be open source on GitHub soon.

Sequential point clouds acquired by light detection and ranging (LiDAR) technology provide accurate spatial information for environmental sensing. However, semantic segmentation of point cloud sequences relies on many manual point-wise annotations, which are error-prone and expensive. Existing mainstream weakly supervised methods tackle this by reducing the percentage of labeled points, but they are mostly designed for static indoor scenes and are hard to apply practically. From the viewpoint of realistic annotation procedures and the nature of point cloud sequences, this paper proposes a novel semantic segmentation method, SemanticFlow, for LiDAR point cloud sequences using sparse frames with annotations. The proposed method achieves competitive performance compared with fully supervised methods. Specifically, we designed a bidirectional cross-frame pseudo label propagation module that uses scene flow to learn the correlation and propagate pseudo labels across neighboring frames. In addition, a label refinement mechanism is proposed to select reliable pseudo labels for learning. Extensive experiments on SemanticKITTI, SemanticPOSS, and Synthia 4D datasets demonstrate that our sparse frame annotation method is compatible with some fully supervised counterparts.

This paper presents a matching network to establish point correspondence between images. We propose a Multi-Arm Network (MAN) capable of learning region overlap and depth, which can greatly improve keypoint matching robustness while bringing an extra 50% of computational time during the inference stage. By adopting a different design from the state-of-the-art learning based pipeline SuperGlue framework, which requires retraining when a different keypoint detector is adopted, our network can directly work with different keypoint detectors without time-consuming retraining processes. Comprehensive experiments conducted on four public benchmarks involving both outdoor and indoor scenarios demonstrate that our proposed MAN outperforms state-of-the-art methods.

LiDAR localization is of great importance to autonomous vehicles and robotics. Absolute pose regression, directly estimating the mapping from a scene to a 6-DoF pose, has achieved impressive results in learning-based localization. Different from traditional map-based methods, it does not need a pre-built 3D map during inference. However, current regression networks typically suffer from scene ambiguities, especially in challenging traffic environments, leading to large wrong predictions (e.g., outliers) and limited applications. To address this problem, a novel LiDAR localization framework with spatio-temporal constraints is proposed, termed STCLoc, to reduce scene ambiguities and achieve more accurate localization. First, we propose to regularize regression in the spatial dimension with a novel classification task to reduce outliers. Specifically, the classification task categorizes the point cloud in terms of position and orientation and then couples it with the regression task to conduct multi-task learning. Second, to learn discriminative features to reduce scene ambiguities, we propose using attention-based feature aggregation to capture the correlation in LiDAR sequences. We conduct extensive experiments on two benchmark datasets, where the localization takes 97ms on each dataset. Results show that our model outperforms state-of-the-art methods by 43.33%/36.76% (position/orientation) on the Oxford Radar RobotCar dataset, verifying the effectiveness of our method. The source code is available on the project website athttps://github.com/PSYZ1234/STCLoc.

Unsupervised domain adaptation (UDA) is a low-cost way to deal with the lack of annotations in a new domain. For outdoor point clouds in urban transportation scenes, the mismatch of sampling patterns and the transferability difference between classes make cross-domain segmentation extremely difficult. To overcome these challenges, we propose a category-level adversarial framework. Firstly, we propose a multi-scale domain conditioned block that facilitates to extract the critical low-level domain-dependent knowledge and reduce the domain gap caused by distinct LiDAR sampling patterns. Secondly, we make full use of multiple representation forms (i.e., point-based sets and voxel-based cells) and utilize the prediction consistency between the two forms to measure how well each point is semantically aligned. The model then focuses on the poorly-aligned points without affecting the well-aligned points. Experimental results on three autonomous driving point cloud datasets show that the proposed method outperforms existing methods by a large margin, especially on the low-beam to high-beam cross-domain segmentation task.

Recent years have witnessed rapid advances in autonomous driving technologies. Autonomous vehicles are more likely to be accepted if they drive comfortably to avoid potholes, bumps, and other road damage conditions, especially when there are elderly and disabled passengers on board. Traditionally, sensing road damage conditions is either labor-intensive by field investigation and reporting, or inaccurate by surveillance cameras and driving recorders due to limited perspective. In this paper, we propose GoComfort, a crowdsensing-based framework to provide low-cost and fine-grained comfortable navigation for autonomous vehicles with road damage identification leveraging high-precision road sensing data. First, we propose to exploit city-wide autonomous vehicle fleets as crowdsensing participants, and employ an edge-cloud-hybrid computing paradigm to efficiently collect high-precision road damage-related data, including 3D LiDAR point clouds and street view images. Second, we design an accurate road damage identification model fusing spatial structures of point clouds and texture features of street view images, and use an active learning-based method to address the sparse labels issue. Finally, we devise two comfortable navigation scenarios, i.e., fine-grained road damage avoidance and coarse-grained city-wide navigation, and propose a hierarchical road damage assessment diagram for comfortable route planning. Experiments using real-world road sensing data in Xiamen, China show that our approach identifies road damage conditions with an accuracy of 87.5%, and achieves a user acceptance rate of 93.8% in riding comfort evaluation, outperforming the state-of-the-art baselines.

Traffic violations have become one of the major threats to urban transportation systems, undermining human safety and causing economic losses. To alleviate this problem, crowd-based patrol forces including traffic police and voluntary participants have been employed in many cities. To adaptively optimize patrol routes with limited manpower, it is essential to be aware of traffic violation hotspots. Traditionally, traffic violation hotspots are directly inferred from experiences, and existing patrol routes are usually fixed. In this paper, we propose a mobile crowdsensing-based framework to dynamically infer traffic violation hotspots and adaptively schedule crowd patrol routes. Specifically, we first extract traffic violation-prone locations from heterogeneous crowd-sensed data and propose a spatiotemporal context-aware self-adaptive learning model (CSTA) to infer traffic violation hotspots. Then, we propose a tensor-based integer linear problem modeling method (TILP) to adaptively find optimal patrol routes under human labor constraints. Experiments on real-world data from two Chinese cities (Xiamen and Chengdu) show that our approach accurately infers traffic violation hotspots with F1-scores above 90% in both cities, and generates patrol routes with relative coverage ratios above 85%, significantly outperforming baseline methods.

Bike-sharing systems have been deployed globally. One of the key issues for high-quality bike-sharing systems is to rebalance city-wide stations to maintain bike availability. Traditional strategies, such as repositioning bikes by trucks and volunteers based on historical riding records, usually operate in fixed paths and limited capacities, lacking the flexibility to cope with the highly dynamic and context dependent riding demands, and usually suffer from high costs and long delays. In this work, we propose RedPacketBike, an incentive-driven, crowd-based station rebalancing framework to effectively recruit participants from hybrid fleets (e.g., volunteer riders and hired trucks) based on the accurate forecast of bike demand leveraging deep learning techniques. First, we propose a spatiotemporal clustering method to extract bike demand hotspots from fluctuating bike usage data. Then, we build a context-aware deep neural network named BikeNet to forecast the trends of bike demand hotspots, simultaneously modeling the spatial correlations by graph convolution networks (GCN), the temporal dependencies by long short-term memory networks (RNN), and the contextual factors by autoencoders (AE). Finally, we propose a reinforcement-learning-based method to find optimal station rebalancing schemes by generating station rebalancing tasks with an integer linear programming (ILP) algorithm and allocating tasks to participants from hybrid fleets with dynamic incentive designs and reward expectations. Experiments using real-world bike-sharing system data collected from Citi Bike in New York City and Mobike in Xiamen City validate the performance of our framework, achieving a demand forecast error below 4.171 measured in MAE, and a 17.2% improvement of station availability by simulations with real-world parameter settings, outperforming the state-of-the-art baselines.

Road markings extraction (RME) from 3D point clouds acquired by mobile LiDAR systems has been widely used for road safety and autonomous driving. However, due to the increasing awareness of personal data protection and national informat ion security regulations, most autonomous driving companies are not willing to share their private point clouds data with the community. Therefore, such restriction of centralized training might inevitably inhibit the effectiveness of RME procedure. Federated learning (FL) is a distributed machine learning architecture that could address the aforementioned privacy-accuracy dilemma to collaboratively learn a global RME model from multiple clients without sharing raw data. In this paper, we propose a novel FedRME, a federated road markings extraction system to collaboratively learn a global RME model with multiple privacy-preserved local models from 3D mobile LiDAR point clouds. FedRME adopt the classical FedAvg model to construct a generalizable global feature embedding model without accessing local data. Moreover, to tackle data heterogeneity problem that local models vary in point clouds volumes and categories, we design a dynamic weighting mechanism to optimize the cooperative training effectiveness before server aggregation. Experimental results on three real-world mobil e LiDAR point clouds datasets with federated learning settings demonstrate that FedRME not only achieves superior performance but also reduces computation by up to 25%.The source code is available at https://github.com/WwZzz/easyFL#FedRME.

We propose Human-centered 4D Scene Capture (HSC4D) to accurately and efficiently create a dynamic digital world, containing large-scale indoor-outdoor scenes, diverse human motions, and rich interactions between humans and environments. Using only body-mounted IMUs and LiDAR, HSC4D is space-free without any external devices' constraints and map-free without pre-built maps. Considering that IMUs can capture human poses but always drift for long-period use, while LiDAR is stable for global localization but rough for local positions and orientations, HSC4D makes both sensors complement each other by a joint optimization and achieves promising results for long-term capture. Relationships between humans and environments are also explored to make their interaction more realistic. To facilitate many down-stream tasks, like AR, VR, robots, autonomous driving, etc., we propose a dataset containing three large scenes (1k-5k m2) with accurate dynamic human motions and locations. Diverse scenarios (climbing gym, multi-story building, slope, etc.) and challenging human activities (exercising, walking up/down stairs, climbing, etc.) demonstrate the effectiveness and the generalization ability of HSC4D. The dataset and code is available at lidarhumanmotion.net/hsc4d.

Existing motion capture datasets are largely short-range and cannot yet fit the need of long-range applications. We propose LiDARHuman26M, a new human motion capture dataset captured by LiDAR at a much longer range to overcome this limitation. Our dataset also includes the ground truth human motions acquired by the IMU system and the synchronous RGB images. We further present a strong base-line method, LiDARCap, for LiDAR point cloud human motion capture. Specifically, we first utilize$PointNet++$to encode features of points and then employ the inverse kinematics solver and SMPL optimizer to regress the pose through aggregating the temporally encoded features hierarchically. Quantitative and qualitative experiments show that our method outperforms the techniques based only on RGB images. Ablation experiments demonstrate that our dataset is challenging and worthy of further research. Finally, the experiments on the KITTI Dataset and the Waymo Open Dataset show that our method can be generalized to different LiDAR sensor settings.

The Centralized Training with Decentralized Execution paradigm (CTDE), which trains policies centrally with additional information, is important for Multi-Agent Reinforcement Learning (MARL). For CTDE, value function factorization methods make use of state during training and factorize the value function into multiple local value functions for decentralized execution. These approaches do not fully consider the relational information among agents, resulting in sub-optimal models for complex tasks. To remedy this issue, we propose QRelation which is a graph neural network approach for value function factorization. It considers both the static relations (e.g., agent types) and dynamic relations (e.g., close-by). We show that QRelation can obtain better results than state-of-the-art methods on challenging StarCraft II benchmarks.

Recently, laser scanning system is widely used to accurately predict forest inventory attributes. In this work, we propose an efficient framework including Feature Extractor Module and Conditional Adversarial Module for object classification in 3D heterogeneous point clouds. The Feature Extractor Module can aggregate features of objects in point clouds. For domain adaptation task, the Conditional Adversarial Module is proposed to minimize the discrepancy of source and target domains. Since there is no common evaluation benchmark for object classification in 3D point cloud, we build a bench-mark of six tasks from three 3D objects datasets. Evaluations on six tasks with four categories have demonstrated the effectiveness of our proposed framework of object classification. The framework can be applied potentially in forest management.

In this paper, we propose a novel LiDAR-based mapping framework for geometry-featureless scenarios, combining learning-based mismatch detection and intensity-assisted registration optimization. The mismatch detection method hybridizes point cloud features and temporal features of pose to detect the mismatch position, and the optimization method uses the intensity of captured road markers by LiDAR to optimize mismatch pose. Experiments on underground garages data demonstrate that the proposed method performs better localization and mapping accuracy than the LOAM.

Modern autonomous vehicles perceive their surroundings with the help of 3D computer vision technologies (e.g., object detection, vehicle recognition). The 3D point cloud obtained by vehicle-mounted mobile LiDAR is the primary data for 3D visual tasks in the driving system. Due to partial observations, a complete point cloud of the surrounding vehicle cannot be obtained. In this paper, we proposed Point Voxel Completion Network (PVCNet), an end-to-end learning-based model for 3D vehicle completion. Unlike existing point cloud completion methods, PVCNet only predicts the missing parts of the input and preserves the original spatial structure of the point cloud. PVCNet consists of two branches, the voxel branch and the point cloud branch. The voxel branch extracts the local features of the input and predicts the position of the missing point cloud in voxel form. The point cloud branch extracts the global features of the input point cloud. Combined with features from two branches, PVCNet can generate the missing points. The proposed PVCNet is validated on our hybrid dataset and KITTI dataset and performed competitive results in the task of urban vehicle point cloud completion.

Point clouds are receiving increasing attention in the field of computer vision. Hitherto, segmentation tasks for point clouds have dealt with single-frame data without considering the temporal information. In this paper, we propose a point cloud segmentation network based on a neuron-like model that can exploit the time information of point cloud data to improve network performance. The network's encoder adopts the structure of the U-Net network and sparse convolution. The features extracted by the encoder and the last moment are used as input to the time integration module. The time integration module is a timing-based neuron model which can fuse features from two moments and use them to enhance the features of the current moment. Experiments on the public autopilot dataset NuScenes validate that our proposed point cloud segmentation network can achieve a 97.7 % Intersection over Union (IoU) for segmenting static scenes and dynamic objects. The experiments further validate that fusing temporal information improves the performance of point cloud foreground-background segmentation compared to considering only data from a single frame point cloud data.

Point cloud segmentation plays an important role in AI applications such as autonomous driving, AR, and VR. However, previous point cloud segmentation neural networks rarely pay attention to the topological correctness of the segmentation results. In this paper, focusing on the perspective of topology awareness. First, to optimize the distribution of segmented predictions from the perspective of topology, we introduce the persistent homology theory in topology into a 3D point cloud deep learning framework. Second, we propose a topology-aware 3D point cloud segmentation module, TopoSeg. Specifically, we design a topological loss function embedded in TopoSeg module, which imposes topological constraints on the segmentation of 3D point clouds. Experiments show that our proposed TopoSeg module can be easily embedded into the point cloud segmentation network and improve the segmentation performance. In addition, based on the constructed topology loss function, we propose a topology-aware point cloud edge extraction algorithm, which is demonstrated that has strong robustness.

Learning the embeddings for urban regions from human mobility data can reveal the functionality of regions, and then enables the correlated but distinct tasks such as crime prediction. Human mobility data contains rich but abundant information, which yields to the comprehensive region embeddings for cross domain tasks. In this paper, we propose multi-graph fusion networks (MGFN) to enable the cross domain prediction tasks. First, we integrate the graphs with spatio-temporal similarity as mobility patterns through a mobility graph fusion module. Then, in the mobility pattern joint learning module, we design the multi-level cross-attention mechanism to learn the comprehensive embeddings from multiple mobility patterns based on intra-pattern and inter-pattern messages. Finally, we conduct extensive experiments on real-world urban datasets. Experimental results demonstrate that the proposed MGFN outperforms the state-of-the-art methods by up to 12.35% improvement. https://github.com/wushangbin/MGFN

We propose a novel normal estimation method called HSurf-Net, which can accurately predict normals from point clouds with noise and density variations. Previous methods focus on learning point weights to fit neighborhoods into a geometric surface approximated by a polynomial function with a predefined order, based on which normals are estimated. However, fitting surfaces explicitly from raw point clouds suffers from overfitting or underfitting issues caused by inappropriate polynomial orders and outliers, which significantly limits the performance of existing methods. To address these issues, we introduce hyper surface fitting to implicitly learn hyper surfaces, which are represented by multi-layer perceptron (MLP) layers that take point features as input and output surface patterns in a high dimensional feature space. We introduce a novel space transformation module, which consists of a sequence of local aggregation layers and global shift layers, to learn an optimal feature space, and a relative position encoding module to effectively convert point clouds into the learned feature space. Our model learns hyper surfaces from the noise-less features and directly predicts normal vectors. We jointly optimize the MLP weights and module parameters in a data-driven manner to make the model adaptively find the most suitable surface pattern for various points. Experimental results show that our HSurf-Net achieves the state-of-the-art performance on the synthetic shape dataset, the real-world indoor and outdoor scene datasets. The code, data and pretrained models are publicly available.

The factorization of state-action value functions for Multi-Agent Reinforcement Learning (MARL) is important. Existing studies are limited by their representation capability, sample efficiency, and approximation error. To address these challenges, we propose, ResQ, a MARL value function factorization method, which can find the optimal joint policy for any state-action value function through residual functions. ResQ masks some state-action value pairs from a joint state-action value function, which is transformed as the sum of a main function and a residual function. ResQ can be used with mean-value and stochastic-value RL. We theoretically show that ResQ can satisfy both the individual global max (IGM) and the distributional IGM principle without representation limitations. Through experiments on matrix games, the predator-prey, and StarCraft benchmarks, we show that ResQ can obtain better results than multiple expected/stochastic value factorization methods.

The key challenge in processing point clouds lies in the inherent lack of ordering and irregularity of the 3D points. By relying on perpoint multi-layer perceptions (MLPs), most existing point-based approaches only address the first issue yet ignore the second one. Directly convolving kernels with irregular points will result in loss of shape information. This paper introduces a novel point-based bidirectional learning network (BLNet) to analyze irregular 3D points. BLNet optimizes the learning of 3D points through two iterative operations: feature-guided point shifting and feature learning from shifted points, so as to minimise intra-class variances, leading to a more regular distribution. On the other hand, explicitly modeling point positions leads to a new feature encoding with increased structure-awareness. Then, an attention pooling unit selectively combines important features. This bidirectional learning alternately regularizes the point cloud and learns its geometric features, with these two procedures iteratively promoting each other for more effective feature learning. Experiments show that BLNet is able to learn deep point features robustly and efficiently, and outperforms the prior state-of-the-art on multiple challenging tasks.

Abstract Event camera is a new vision sensor that produces independent asynchronous responses to each pixel's change of illumination intensity. The unique principle of event camera has many advantages over traditional cameras, such as low latency, high temporal resolution, and high dynamic range (HDR). These advantages make event camera ideal for dealing with high speed, HDR visual tasks, especially in automatic driving scenes. In this study, we propose an image generation network named Event‐based attention generative adversarial networks (EAGAN), which simultaneously deals with optical flow and depth estimation of monocular event camera data. In addition to the innovative network architecture and loss function suitable for depth estimation, we are also the first to process incomplete training data to obtain more dense and uniform prediction results. Experiments on the multi‐vehicle stereo event camera dataset show that our EAGAN is competitive on the depth estimation task and achieves the state‐of‐the‐art effect in the optical flow estimation task.

Deep learning for 3-D point cloud perception has been a very active research topic in recent years. A current trend is toward the combination of the semantically strong and the fine-grained information from different scales of intermediate representations to boost network generalization power and robustness against scale variation. One prominent challenge is how to effectively conduct the allocation of multiple scales of information. In this letter, we propose a module, named adaptive pyramid context fusion (APCF), to adaptively capture scales of contextual information from a multiscale feature pyramid for the point cloud. The APCF module reweights and aggregates the features from different levels in the feature pyramid via a softmax attention strategy. The allocation of information is adaptively conducted level by level from bottom to up first and then from top to bottom. To ensure both effectiveness and efficiency, we propose a multiscale context-aware network APCF-Net through applying our proposed APCF to the PointConv architecture. Experiments demonstrate that APCF-Net surpasses its vanilla counterpart by a large margin both in effectiveness and efficiency. Especially, APCF-Net outperforms state-of-the-art approaches on 3-D object classification and semantic segmentation task, with the overall accuracy of 93.3% on ModelNet40 and mIoU of 63.1% on ScanNet V2 online test.

We present a registration strategy based on a planar primitive group for indoor environments between a large-scale point cloud and a small-scale point cloud, providing a localization solution that is fully independent of prior information about the initial positions of the two point cloud coordinate systems. The algorithm first divides the point cloud into planes by region growing and then refines the plane boundaries by local$k$-means clustering. The planes are grouped to obtain planar primitive groups that are used to infer potential matching regions for an effective coarse registration and then fine-tuning. The algorithm is applied for autonomous vehicle localization in underground garages. Evaluation on four datasets shows that the algorithm can provide decimeter-level localization accuracy in seconds.

Automatic building extraction from remote sensing imagery is crucial to urban construction and management. To address the main challenges of diverse building scale and appearance, this letter proposes an automatic building instance extraction method based on an improved hybrid task cascade (HTC). Our method consists of three components by obtaining high-resolution representation, defining guided anchor, and forming focal loss to boost the adaptability of automatic building instance extraction. Comprehensive experimental results on WHU aerial building data set demonstrated that compared with the mainstream Mask R-CNN method, our method increased AP and AR in bounding box branch and mask branch by 9.8%–6.5% and 10.7%–8.0% respectively, especially AP$_{S}$and AP$_{L}$in the two branches by 10.1%–6.9% and 3.4%–2.4%, respectively. We evaluated the effectiveness and complexity of these components separately and discussed the universality and practicability of deep learning method in automatic building extraction.

Nowadays, researchers have developed various deep neural networks for processing point clouds effectively. Due to the enormous parameters in deep learning-based models, a lot of manual efforts have to be invested into annotating sufficient training samples. To mitigate such manual efforts of annotating samples for a new scanning device, this letter focuses on proposing a new neural network to achieve domain adaptation in 3D object classification. Specifically, to minimize the data discrepancy of intra-class objects in different domains, an Asymmetrical Siamese module is designed to align the intra-class features. To preserve the discriminative information for distinguishing inter-class objects in different domains, a Conditional Adversarial module is leveraged to consider the classification information conveyed from the classifier. To verify the effectiveness of the proposed method on object classification in heterogeneous point clouds, evaluations are conducted on three point cloud datasets, which are collected in different scenarios by different laser scanning devices. Furthermore, the comparative experiments also demonstrate the superior performance of the proposed method on the classification accuracy.

Large-scale semantic segmentation point cloud is an ongoing research topic for on-land environments. However, there is a rare deep learning research study for the sub-surface environment. Although, PointNet and its successor PointNet++ have become the cornerstone of point cloud segmentation. However, these techniques handle a relatively small number of points. This poses a natural difficulty in a large spatial scene with millions of possible points. In particular, for shallow water of coastal zone, the small number of points where the seabed and water surface meet, close points may belong to different classes. In our work, we present the semantic segmentation on a large-scale airborne Lidar bathymetry (ALB) point cloud containing millions of sample points into two classes of water surface and seabed with the voxel sampling pre-processing (VSP) approach. The proposed approach will allow us to capture the complicated outdoor natural scene components of water surface and seabed more accurately and more realistic through nonuniform voxelization in the mixture of dense and sparse points of the ALB point cloud. The performance of validation results show improvement in a per-point accuracy of 72.45% compared with other state-of-the-art deep learning-based methods.

Rapid and accurate enhancement of road boundaries from terrestrial laser scanning (TLS) 3-D point clouds has been a challenging task in road infrastructure inventory. To address the challenge with a lack of ability to enhance object boundaries when the supervoxel number is less, this letter proposes a novel supervoxel segmentation algorithm framework for enhancing road boundaries from 3-D point clouds. First, we utilize radius$k$nearest-neighbor search method to obtain the neighborhood information after partitioning points on octrees with seed points. Second, the iterative weighted least square algorithm and spatial structure judgment are used to segment point clouds based on seed points. Finally, an update method to adjust the supervoxel centroids is applied with surrounding information in the first part. To verify the excellent performance, we tested the proposed method on two publicly large-scale point clouds benchmarks—IQmulus and TerraMobilita (IQTM) and Semantic 3-D. The experimental results demonstrate that our approach achieved approximately 48.98% and 68.41% boundary recall higher than two existing classical methods in the street scene, and our running time is feasible and effective.

Automatic building facade point cloud semantic segmentation is an important step in 3-D urban building reconstruction. How to correctly segment the components (e.g., windows, walls, and columns) from the building facade is still a challenging task. According to the characteristics of building facade point clouds, we introduce local fusion attention network (LFA-Net), an efficient neural network that learns LFA features from building facade point clouds, for better capturing the local neighborhood structure information of each point. The core of LFA-Net is the LFA module, which consists of three neural units: local graph attention (LGA), local aggregation attention (LAA), and fusion attention (FA). The LFA-Net is the standard encoder-decoder architecture. Experiments demonstrate that our LFA-Net outperforms the state-of-the-art methods on the large-scale building facade point cloud dataset.

3D human trajectory reconstruction in an indoor scene is critical in various applications, such as indoor navigation and human activity recognition. This task is challenging due to occlusion and clutters of indoor scenes, flexible human body joints, and severe lack of relevant datasets. Although several methods have been proposed to reconstruct a 3D human trajectory, they either can recover only 2D positions or require human initiative cooperation. In this paper, we propose a novel framework for 3D human trajectory reconstruction in an indoor scene using monocular surveillance videos and static point clouds without any initiative cooperation. The proposed framework consists of three modules: 3D pose estimation, depth regression, and trajectory reconstruction. We first estimate 3D pose from videos. Especially, we reconstruct a half-body 3D pose to deal with the occlusion problem. Then, we propose a depth regression approach to iteratively regress the depth of a 3D pose. Unlike data-driven approaches, our depth regression approach does not require training data and can be integrated into any 3D pose model. Finally, we exploit the geometric constraints from the point cloud to optimize the 3D trajectory. We evaluated the 3D pose estimation and depth regression modules on the H3.6M datasets. Due to the lack of evaluation datasets, we also built a trajectory dataset to evaluate the trajectory reconstruction performance. Empirical evaluation shows that our framework achieves accurate trajectory reconstruction results on real-world videos.

Point cloud completion aims to reconstruct complete point clouds from partial point clouds, which is widely used in various fields such as autonomous driving and robotics. Most existing methods are sparse point cloud completion, where the number of point clouds after completion is relatively small and the details are insufficient. This article proposes a novel end-to-end generative adversarial network-based dense point cloud completion architecture (DPCG-Net). We design two generative adversarial network (GAN)-based modules that translate point cloud completion into mapping between global feature distributions obtained by encoding partial point clouds and ground truth, respectively. The first designed generator module proposes skip connections to fully connected layer-based network for regenerating global feature and changing the global feature distribution derived from the encoder module to approximate the ground truth global feature distribution. The second proposed discriminator module divides high-dimensional global feature vectors into several smaller batches for judgment to guarantee the similarity between the regenerated global feature and the ground truth. We perform quantitative and qualitative experiments on the ShapeNet and KITTI datasets. Experiments on ShapeNet demonstrate that our model outperforms other models in cases where the lack of a large proportion of point clouds results in a large loss of spatial structure, especially when 80% of point clouds are missing. Moreover, KITTI experiments reveal that it is also valid for realistic situations. In addition, application in classification shows that the classification accuracy of point clouds completed with DPCG-Net is as high as 86.5% under the condition of 80% missing point clouds.

Extracting the power lines and pylons automatically and accurately from airborne LiDAR data is a critical step in inspecting the routine power line, especially in the remote mountainous areas. However, challenges arise in using existing methods to extract the targets from large scenarios of remote mountainous areas since the terrain is undulating, and the features are difficult to distinguish. In this article, to overcome these challenges, we propose a graph convolutional network (GCN)-based method to extract power lines and pylons from Airborne LiDAR point clouds. First, data augmentation and near-ground filtering methods are developed to overcome the problems of insufficient and imbalanced samples in the LiDAR data. Then, a GCN-based framework is proposed to extract the power lines and pylons, which consist of two main modules, i.e., the neighborhood dimension information (NDI) module and the neighborhood geometry information aggregation (NGIA) module. These two modules are designed to strengthen the model’s ability to portray local geometric details. Besides, an attention fusion module is investigated to further improve the NDI and NGIA features. Finally, a line structure constraint algorithm is proposed to identify individual power lines, where the power corridor is reconstructed using a polynomial-based algorithm. Numerical experiments are conducted based on two different power line scenarios acquired in mountainous areas. The results demonstrate the superior performances of the proposed method over several existing algorithms, where the$F_{1}$score and quality of the power line are 99.3% and 98.6%, and the results of the pylon are 96% and 92.4%, respectively. The identification rate of power line identification is above 98%.

Individual tree detection is critical for forest investigation and monitoring. Several existing methods have difficulties to detect trees in complex forest environments due to insufficiently mining descriptive features. This study proposes a deep learning (DL) framework based on a designed multichannel information complementarity representation for detecting trees in complex forest using UAV laser scanning point clouds. The proposed method consists of two main stages: ground filtering and tree detection. In the first stage, a modified graph convolution network with a local topological information layer is designed to separate the ground points. Unlike most existing parametric methods, our ground filtering method avoids the optimal parameters selection to adapt to different kinds of environments. For tree detection, a top-down slice (TDS) module is first designed to mine the vertical structure information in a top-down way. Then, a special multichannel representation (MCR) is developed to preserve different distribution patterns of points from complementary perspectives. Finally, a multibranch network (MBNet) is proposed for individual tree detection by fusing multichannel features, which can provide discriminative information for MBNet to detect trees more accurately. MBNet was evaluated on seven forest areas [UAV light detection and ranging (LiDAR) data with the mean size of 14$000~\text {m}^{2}$and point density of 250 points/$\text {m}^{2}$]. Experimental results showed that the proposed framework achieves excellent performance. Our method obtains promising performance with a mean recall of 89.23% and a mean F1-score of 87.04%.

Unsupervised domain adaptation (UDA) significantly reduces the gap between the source domain and the target domain in machine learning and computer vision tasks. Most UDA approaches are applied to images and videos, and only a few methods implement domain adaptation on 3-D computer vision problems. The existing UDA approaches operating on point clouds try to extract domain-invariant features in different domains for feature alignment. However, higher commonality brings less diversity and results a loss of detailed information. In this article, we propose a novel dual-branch feature alignment network (DFAN) architecture for domain adaptation on point cloud visual tasks to better exploit the respective characteristics of local and global features. Our approach specializes in the extraction and alignment of global and local features with different strategies in each branch to complement each other. We also introduce a hierarchical alignment strategy for local feature alignment and a distribution alignment strategy for global feature alignment. Experiments on the PointDA-10 and PointSegDA datasets show that our approach achieves state-of-the-art performance on the UDA of point cloud classification and segmentation tasks. The ablation study demonstrates the effectiveness of the dual-branch design and the feature alignment strategies.

3D object detection from LiDAR point clouds has gained great attention in recent years due to its wide applications in smart cities and autonomous driving. Cascade framework shows its advancement in 2D object detection but is less investigated in 3D space. Conventional cascade structures use multipleseparatesub-networks to sequentially refine region proposals. Such methods, however, have limited ability to measure proposal quality in all stages, and hard to achieve a desirable performance improvement in 3D space. This paper proposes a new cascade framework, termed CasA, for 3D object detection from LiDAR point clouds. CasA consists of a Region Proposal Network (RPN) and a Cascade Refinement Network (CRN). In CRN, we designed a new Cascade Attention Module that uses multiple sub-networks and attention modules to aggregate the object features from different stages and progressively refine region proposals. CasA can be integrated into various two-stage 3D detectors and improve their performance. Extensive experiments on KITTI and Waymo datasets with various baseline detectors demonstrate the universality and superiority of our CasA. In particular, based on one variant of Voxel-RCNN, we achieve state-of-the-art results on the KITTI dataset. On the KITTI online 3D object detection leaderboard, we achieve a high detection performance of 83.06%, 47.09%, and 73.47% Average Precision (AP) in the moderate Car, Pedestrian, and Cyclist classes, respectively. Code is available at https://github.com/hailanyi/CasA.

Estimating urban surface temperature at high resolution is crucial for effective urban planning for climate-driven risks. This high-resolution surface temperature over broader scales can usually be obtained via satellite remote sensing for historical period. However, it can be hard for future predictions. This paper presents a Physics Informed Hierarchical Perception (PIHP) network, a novel approach for accurate, high-resolution and generalizable urban surface temperature estimation. The key to our approach is leveraging the implied temperature-related physics information of the land surface structure from high-resolution multi-spectral satellite images, thus achieving precise estimation or prediction for high spatial resolution urban surface temperature. Specifically, a semantic category histogram is first designed to describe the land surface structures. Based on this, a hierarchical urban surface perception network is proposed to capture the complex relationship between the underlying land surface features, upper atmosphere conditions and the intracity temperature. The proposed PIHP-Net makes it possible to generate models that can generalize across different cities, thus to estimating or predicting high-resolution urban surface temperature when the satellite land surface temperature (LST) observation is not available. Experiments over various cities in different climate regions in China show, for the first time, errors less than 2 Kelvin (for most of the cases) at the high resolution (60-by-60 meters grids), thus making it possible to predict futureintracity temperaturefrom forcing meteorology and multi-spectral satellite imagery.

Mobile Laser Scanning (MLS) system can provide high-density and accurate 3D point clouds that enable rapid pavement crack detection for road maintenance tasks. Supervised learning-based algorithms have been proved pretty effective for handling such a large amount of inhomogeneous and unstructured point clouds. However, these algorithms often rely on a lot of annotated data, which is labor-intensive and time-consuming. This paper presents a semi-supervised point-level approach to overcome this challenge. We propose a graph-widen module to construct a reasonable graph structure for point clouds, increasing the detection performance of graph convolutional networks (GCN). The constructed graph characterizes the local features from a small amount of annotated data, avoiding information loss and dramatically reduces the dependence on annotated data. The MLS point clouds acquired by a commercial RIEGL VMX-450 system are used in this study. The experimental results demonstrate that our method outperforms the state-of-the-art point-level methods in terms of recall, F1 score, and efficiency while achieving comparable accuracy.

Mobile Laser Scanning (MLS) systems provide highly dense 3D point clouds that enable the acquisition of accurate traffic facilities information for intelligent transportation system. Road guardrails with safety features that can separate traffic and define moving spaces for pedestrians and vehicles face challenges such as diverse guardrail types and continuous slopes in point clouds data. This paper proposes a novel approach for rapidly extracting urban road guardrails from MLS point clouds, combining a proposed multi-level filtering with a modified Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering, and adapting for most types of guardrails and rough slope roads. We develop a multi-level filter to detect the road surface and remove the undesirable points. Through a proposed modified DBSCAN clustering, the guardrails are extracted after a four-step screening, which includes the limits based on the number of points, the fitting error, the bounding box size and the average reflection intensity for each cluster. The proposed method achieves high precisions of 97.2% and 96.4% respectively for the lane-separating guardrails and the anti-fall guardrails on the dataset. Extensive experiments with test dataset captured by a RIEGL VMX-450 MLS, show that our method outperforms the state-of-the-art method to extract 3D guardrails from point clouds.

Traffic violations have become one of the major threats to urban transportation systems, undermining road safety and causing economic losses. Although various methods have been proposed by road authorities and researchers to find out the possible causes of traffic violations, existing methods often fail to diagnose traffic violations from drivers’ perspectives and contexts or consider their visual and comprehension loads while driving. In this work, we propose a driver-centered simulation platform to inspect drivers’ loads in traffic violation hotspots. Specifically, we first build a driving simulator based on the 3D point clouds of real-world traffic violation hotspots. We then recruit drivers to simulate driving in designated traffic scenes. Indicators for drivers’ visual and comprehension loads are derived based on drivers’ feedback. Upon this basis, we build an explainable model to automatically indicate drivers’ visual and comprehension loads under various crowd-sensed traffic scenes. Experiments using real-world data from a Chinese City (Xiamen) and case studies show that our approach successfully derives a set of prominent indicators to effectively diagnose drivers’ visual and comprehension loads in real-world traffic violation hotspots.

This paper proposes a new 3D multi-object tracker to more robustly track objects that are temporarily missed by detectors. Our tracker can better leverage object features for 3D Multi-Object Tracking (MOT) in point clouds. The proposed tracker is based on a novel data association scheme guided by prediction confidence, and it consists of two key parts. First, we design a new predictor that employs a constant acceleration (CA) motion model to estimate future positions, and outputs a prediction confidence to guide data association through increased awareness of detection quality. Second, we introduce a new aggregated pairwise cost to exploit features of objects in point clouds for faster and more accurate data association. The proposed cost consists of geometry, appearance and motion components. Specifically, we formulate the geometry cost using resolutions (lengths, widths and heights), centroids, and orientations of 3D bounding boxes (BBs), the appearance cost using appearance features from the deep learning-based detector backbone network, and the motion cost by associating different motion vectors. Extensive multi-object tracking experiments on the KITTI tracking benchmark demonstrated that our method outperforms, by a large margin, the state-of-the-art methods in both tracking accuracy and speed.

Nowadays, a large number of sensors are equipped on mobile or stationary platforms, which continuously generate geo-tagged and time-stamped readings (i.e., geo-sensory data) that contain rich information about the surrounding environment. These data have irregular space and time coordinates. To represent geo-sensory data, there have been extensive research efforts using time sequences, grid-like images, and graph signals. However, there still lacks a proper representation that can describe both the mobile and stationary geo-sensory data without the information-losing discretization in spatial and temporal dimensions. In this paper, we propose to represent massive geo-sensory data asspatio-temporal point clouds(STPC), and presentSTPC-Net, a novel deep neural network for processing STPC. STPC leverages the original irregular space-time coordinates, and STPC-Net captures intra-sensor and inter-sensor correlations from STPC. In this way, STPC-Net learns the key information of STPC, and overcomes challenges in data irregularity. Experiments using real-world datasets show that STPC-Net achieves state-of-the-art performance in different tasks on both mobile and stationary geo-sensory data. The source code is available athttps://github.com/zhengchuanpan/STPC-Net.

The coronavirus disease 2019 (COVID-19) break-out in late December 2019 has spread rapidly worldwide. Existing studies have shown that there is a significant correlation between large-scale human movements and the spread of the epidemic. However, there is a lack of quantification of these correlations, and it is still challenging to predict the spread of the epidemic at early stage. In this paper, we address this issue by conducting a statistical analysis on the spatio-temporal relationship between human mobility and the epidemic spread. Specifically, we proposed an improved SEIR model to adapt to the COVID-19 epidemic, so that we can predict the spread of the epidemic at the early stage using human mobility data and the early confirmed cases. We evaluated our model in various provinces and cities in China, and the results are superior to various baselines, verifying the effectiveness of the method.

The communication among agents is important for Multi-Agent Reinforcement Learning (MARL). In this work, we propose GraphComm, a method makes use of the relation-ships among agents for MARL communication. GraphComm takes the explicit relations (e.g., agent types), which can be provided through some knowledge background, into account to better model the relationships among agents. Besides explicit relations, GraphComm considers implicit relations, which are formed by agent interactions. GraphComm use Graph Neural Networks (GNNs) to model the relational information, and use GNNs to assist the learning of agent communication. We show that GraphComm can obtain better results than state-of-the-art methods on the challenging StarCraft II unit micromanagement tasks through extensive experimental evaluation.

Similarity measure of cross-domain descriptors (2D descriptors and 3D descriptors) between 2D image patches and 3D point cloud volumes provides stable retrieval performance and establishes the spatial relationship between 2D and 3D space, which plays the potential applications in geospatial space, such as 2D and 3D interaction of remote sensing, Augmented Reality (AR) and robot navigation. However, the mature handcrafted descriptors of 2D image patches and 3D point cloud volumes are extremely different, resulting in the huge challenge for 2D image patch and 3D point cloud volume matching. In this paper, we propose a novel network which combines both unified descriptor training and descriptor comparison function training for 2D image patch and 3D point cloud volume matching. First, two feature extraction networks are applied for jointly learning the local descriptors for 2D image patches and 3D point cloud volumes, respectively. Second, a fully connected network is introduced to compute the similarity between 2D descriptors and 3D descriptors. Motivated by the successful indicator system on evaluating 2D patch feature representation, we use the false positive rate at 95% recall (FPR95) and precision based on cross-domain descriptors as the measured metric. The experimental results show that our proposed network achieve state-of-the-art performance in the matching of 2D image patches and 3D point cloud volumes.

As a foundational preprocessing step for a lot of downstream tasks, ground filtering from airborne LiDAR data is designed to separate the ground points and preserve the off-ground points with complete shape information. However, because of the undulating terrain, it is still a challenge work to filter the ground under complex mountain regions. In this paper, we provide a deep learning based model to improve the ground filtering performance in abrupt slope using airborne LiDAR point clouds. Specifically, we first design a local topological information mining module to extract the local features. Then a modified graph convolutional networks (GCNs) is developed to fusion the local features and global features. Compared with most existing methods, our model not only enjoys the parameter-free advantage, which means it can be applied easily in various areas, but also obtains better ground filtering performance and can preserve more complete information contained in off-ground points. Experiments was implemented on seven forest areas. The proposed method obtains promising ground filtering results with mean total error of 6.46% and the mean kappa coefficient of 86.01%.

Fairness has emerged as a critical problem in federated learning (FL). In this work, we identify a cause of unfairness in FL -- conflicting gradients with large differences in the magnitudes. To address this issue, we propose the federated fair averaging (FedFV) algorithm to mitigate potential conflicts among clients before averaging their gradients. We first use the cosine similarity to detect gradient conflicts, and then iteratively eliminate such conflicts by modifying both the direction and the magnitude of the gradients. We further show the theoretical foundation of FedFV to mitigate the issue conflicting gradients and converge to Pareto stationary solutions. Extensive experiments on a suite of federated datasets confirm that FedFV compares favorably against state-of-the-art methods in terms of fairness, accuracy and efficiency. The source code is available at https://github.com/WwZzz/easyFL.

This paper proposes the first tracklet proposal network, named PC-TCNN, for Multi-Object Tracking (MOT) on point clouds. Our pipeline first generates tracklet proposals, then refines these tracklets and associates them to generate long trajectories. Specifically, object proposal generation and motion regression are first performed on a point cloud sequence to generate tracklet candidates. Then, spatial-temporal features of each tracklet are exploited and their consistency is used to refine the tracklet proposal. Finally, the refined tracklets across multiple frames are associated to perform MOT on the point cloud sequence. The PC-TCNN significantly improves the MOT performance by introducing the tracklet proposal design. On the KITTI tracking benchmark, it attains an MOTA of 91.75%, outperforming all submitted results on the online leaderboard.

Scene understanding in 3-D point clouds requires to annotate points manually at the model training stage. To reduce manual efforts in labeling points, this letter focuses on proposing an efficient method to implement semiautomatic segmentation of 3-D objects in 3-D point clouds. Specifically, to handle point clouds with high point density, supervoxels are treated as basic operating units during the object segmentation procedure. To obtain the valuable boundaries for guiding 3-D object segmentation, we propose to filter meaningless boundaries obtained by a traditional boundary detection method. Once valuable boundaries are obtained, we propose a boundary-aware Markov random field (MRF) model to consider the object-boundary constraint into generating the boundary-preserved segmentation results. Extensive experiments on two data sets show the effectiveness of our proposed framework on segmenting 3-D objects from point cloud scenes.

Presented in this paper is a novel method for the mapping and semantic modeling of an underground parking lot using 3D point clouds collected by a low-cost Backpack Laser Scanning (BLS) or LiDAR system. Our method consists of two parts: a Simultaneous Localization and Mapping (SLAM) algorithm based on Sparse Point Clouds (SPC) and a semantic modeling algorithm based on a modified PointNet model. The main contributions of this paper are as follows: (1) a probability frontend framework for the alignment of point clouds using the local point cloud surface variance as the weight of registration, which modifies registration failure caused by the lack of features in sparse point clouds, (2) a robust submap-based strategy for loop closure detection and back-end optimization under sparse point clouds, and (3) a modified PointNet model for classifying the point clouds of underground parking lots into four categories: ceiling, floor, wall, others. Experimental results show that our SPC-SLAM algorithm achieves centimeter-level accuracy (0.09% trajectory error rate) after closed loop processing in a Global Navigation Satellite System (GNSS)-denied underground parking lot, and precision of 84.8% in semantic segmentation.

High-Accuracy and high-efficiency 3-D sensing and associated data processing techniques are urgently needed for today’s roadway inventory, infrastructure health monitoring, autonomous driving, connected vehicles, urban modeling, and smart cities. 3D geospatial data acquired by digital photogrammetry or laser scanning or LiDAR systems have become one of the most critical data sources to support the above-mentioned applications. While progress has been made to applying 3D sensory data to those applications related to intelligent transportation systems (ITS), such as road network extraction, platform localization, obstacle avoidance, high-definition map generation, and transportation infrastructure inventory, many essential questions remain regarding the processing and understanding such massive 3D datasets in ITS-related applications. The authors have selected four articles for review in this Special issue. A summary of these articles is outlined below.

The surging traffic volumes and dynamic user mobility patterns pose great challenges for cellular network operators to reduce operational costs and ensure service quality. Cloud-radio access network (C-RAN) aims to address these issues by handling traffic and mobility in a centralized manner, separating baseband units (BBUs) from base stations (RRHs) and sharing BBUs in a pool. The key problem in C-RAN optimization is to dynamically allocate BBUs and map them to RRHs under cost and quality constraints, since real-world traffic and mobility are difficult to predict, and there are enormous numbers of candidate RRH-BBU mapping schemes. In this work, we propose a data-driven framework for C-RAN optimization. First, we propose a deep-learning-based Multivariate long short term memory (MuLSTM) model to capture the spatiotemporal patterns of traffic and mobility for accurate prediction. Second, we formulate RRH-BBU mapping with cost and quality objectives as a set partitioning problem, and propose a resource-constrained label-propagation (RCLP) algorithm to solve it. We show that the greedy RCLP algorithm is monotone suboptimal with worst-case approximation guarantee to optimal. Evaluations with real-world datasets from Ivory Coast and Senegal show that our framework achieves a BBU utilization above 85.2 percent, with over 82.3 percent of mobility events handled with high quality, outperforming the traditional and the state-of-the-art baselines.

Fully exploring correlation among points in point clouds is essential for their feature modeling. This paper presents a novel end-to-end graph model, named Point2Node, to represent a given point cloud. Point2Node can dynamically explore correlation among all graph nodes from different levels, and adaptively aggregate the learned features. Specifically, first, to fully explore the spatial correlation among points for enhanced feature description, in a high-dimensional node graph, we dynamically integrate the node's correlation with self, local, and non-local nodes. Second, to more effectively integrate learned features, we design a data-aware gate mechanism to self-adaptively aggregate features at the channel level. Extensive experiments on various point cloud benchmarks demonstrate that our method outperforms the state-of-the-art.

Long-term traffic prediction is highly challenging due to the complexity of traffic systems and the constantly changing nature of many impacting factors. In this paper, we focus on the spatio-temporal factors, and propose a graph multi-attention network (GMAN) to predict traffic conditions for time steps ahead at different locations on a road network graph. GMAN adapts an encoder-decoder architecture, where both the encoder and the decoder consist of multiple spatio-temporal attention blocks to model the impact of the spatio-temporal factors on traffic conditions. The encoder encodes the input traffic features and the decoder predicts the output sequence. Between the encoder and the decoder, a transform attention layer is applied to convert the encoded traffic features to generate the sequence representations of future time steps as the input of the decoder. The transform attention mechanism models the direct relationships between historical and future time steps that helps to alleviate the error propagation problem among prediction time steps. Experimental results on two real-world traffic prediction tasks (i.e., traffic volume prediction and traffic speed prediction) demonstrate the superiority of GMAN. In particular, in the 1 hour ahead prediction, GMAN outperforms state-of-the-art methods by up to 4% improvement in MAE measure. The source code is available at https://github.com/zhengchuanpan/GMAN.

The routine power line inspection is critical to maintain the reliability, availability, and sustainability of electricity supply. As a key part of inspection, power lines and pylons extraction is essential for resource management and power corridor safety, especially in the mountain regions. In this paper, we proposed a deep learning based method to extract power lines and pylons using ALS point clouds. First, a structure information preserved module is designed to mine the relationship of local neighborhood points. Then, a graph convolutional network (GCN) is used as basic module to extract point features. Finally, three categories, power lines, pylons and other objects are segmented from input point clouds. In addition, we provide an effective data enhancement strategy to generate enough samples to train the proposed model. We evaluated our method using a dataset acquired by our ALS scanning system. Experimental results demonstrate that our method is superior to the state-of-the-art methods on descriptiveness and efficiency. The overall accuracy and mean time are 99.1% and 9.3 seconds, respectively.

This paper presents a boundary-enhanced supervoxel method to solve over-segmentation problems in supervoxel generation of Voxel Cloud Connectivity Segmentation (VCCS). First, we use different searching methods to obtain the neighborhood of each point. Second, three variants of neighbor points are clustered by the local k-means clustering method on points directly instead of on voxels. Finally, a scale metric is used to measure the difference between two points that considers underlying 3D spatial structure of the points. Our proposed is tested on two publicly available benchmark point cloud datasets acquired by mobile laser scanning (MLS) and terrestrial laser scanning (TLS) systems, respectively. Results of the experiments show that the boundary recall approximately 7 and 4 times higher than VCCS for the best results, which our proposed methods are effective, and the cost time is feasible and effective.

Different domain image sensors or imaging mechanisms provide cross-domain images when sensing the same scene. There is a domain shift between cross-domain images so that the image gap between different domains is the major challenge for measuring the similarity of the feature descriptors extracted from different domain images. Specifically, matching ground camera images and unmanned aerial vehicle (UAV) 3-D model-rendered images, which are two kinds of extremely challenging cross-domain images, is a way to establish indirectly the spatial relationship between 2-D and 3-D spaces. This provides a solution for the virtual-real registration of augmented reality (AR) in outdoor environments. However, during matching, handcrafted descriptors and existing learning-based feature descriptors limit the rendered images. In this letter, first, to learn robust and invariant 128-D local feature descriptors for ground camera and rendered images, we present a novel network structure, SiamAM-Net, which embeds the autoencoders with an attention mechanism into the Siamese network. Then, to narrow the gap between the cross-domain images during the optimizing of SiamAM-Net, we design an adaptive margin for the loss function. Finally, we match the ground camera-rendered images by using the learned local feature descriptors and explore the outdoor AR virtual-real registration. Experiments show that the local feature descriptors, learned by SiamAM-Net, are robust and achieve state-of-the-art retrieval performance on the cross-domain image data set of ground camera and rendered images. In addition, several outdoor AR applications also demonstrate the usefulness of the proposed outdoor AR virtual-real registration.

Efficient 3-D mapping provides useful and detailed 3-D data for many applications. In this letter, we present a multisensor calibration and mapping method, to provide highly efficient and relatively accurate colored mapping for GPS-/global navigation satellite system-denied environments. The sensor data include 3-D laser scanning point clouds and camera images. A simultaneous localization and mapping (SLAM)-assisted calibration method is first proposed for multiple multibeam light detection and ranging (LiDAR) and multiple camera calibration. An improved SLAM method with loop closure is proposed for 3-D mapping. With the proposed calibration and mapping methods, centimeter-level colored point clouds can be obtained efficiently. The proposed method was tested with both backpacked and car-mounted systems on indoor and outdoor scenes. Experimental results show the effectiveness and efficiency of the proposed calibration and mapping methods.

Effective object extraction plays an important role in many point cloud-based applications. This letter proposes a 3-D feature matching framework for point cloud object extraction. To determine the optimal affine transformation parameters for each template feature point, a convex dissimilarity function and the locally affine-invariant geometric constraints are designed to construct the overall objective function. The 3-D feature matching framework is integrated into a point cloud object extraction workflow. Extraction results on six test data sets show that average completeness, correctness, quality, and F1-measure of 0.96, 0.97, 0.93, and 0.96, respectively, are obtained in extracting light poles, vehicles, and palm trees. Comparative studies also confirm that the proposed method performs effectively and robustly, and exhibits superior or compatible performance over the other compared methods.

Periodical road manhole cover measurement is extremely important to ensure road safety and reduce traffic disasters. This letter proposes an effective method for delineating road manhole covers from mobile laser scanning point cloud data. To improve processing efficiency, first, road surface points are segmented and rasterized into georeferenced intensity images. Then, object-oriented patches are generated through superpixel segmentation and further fed to a convolutional capsule network classifier for manhole cover detection. Finally, manhole covers are accurately delineated through a marked point process of disks. Quantitative evaluations on three data sets show that an average completeness, correctness, quality, and F1-measure of 0.965, 0.961, 0.929, and 0.963, respectively, are obtained. Comparative studies with three existing methods confirm that the proposed method performs superiorly in delineating manhole covers of varying conditions and on complex road surface environments.

The Landsat 1-5 multispectral scanner system (MSS) collected records of land surface mainly during 1972-1992. Investigations on MSS have been relatively limited compared with the numerous investigations on its successors, such as Thematic Mapper (TM) and Enhanced TM Plus (ETM+). The benefits of the Landsat program are not fully accomplished without the inclusion of MSS archives. Investigations on the Landsat 1-5 MSS channel reflectance characteristics wereperformed followed by derived vegetation spectral indices and the Tasseled Cap (TC) transformed features mainly using a collection of synthesized records. On average, the Landsat 4 MSS is generally comparable to the Landsat 5 MSS. The Landsat 1-3 MSSs show disagreement in channel reflectance compared with the Landsat 5 MSS, especially for the red channel (600-700 nm) and the near-infrared channel (700-800 nm). Meanwhile, the relative differences for vegetation spectral indices of the Landsat 3 MSS are mainly from -16% to -5% with the median about -11.5%, while those of the Landsat 2 MSS are mainly from -15% to -7%. Cross-validation tests and two case applications suggested that between-sensor consistency was improved generally through the transformation models generated by ordinary least-squares regression. To improve the consistency of the vegetation indices and the TC greenness, direct strategy employing respective transformation models was more effective than calculations based on the transformed channel reflectance. Considering the shortages of the Landsat MSS archives, further efforts are needed to improve its comparability with observations by other successive Landsat sensors.

Road extraction from remote sensing images is an attractive but difficult task. Gray-value distribution and structure feature information are both crucial for road extraction task. However, existing methods mainly focus on structure feature information which contains morphological shape features and machine learning features, suffering from lots of false positives which are generated at positions having similar structure features but different gray-value distribution with roads. To effectively fuse the two complementary gray-value distribution and structure feature information, we propose a coarse-to-fine road extraction algorithm from remote sensing images. First, at the coarse level, we introduce a local Dirichlet mixture models (LDMM) which utilizing gray-value distribution information to pre-segment images into potential roads and backgrounds. Thus, most backgrounds having different gray-value distribution with roads can be removed firstly. Compared with original Dirichlet mixture models, the LDMM is much faster and more accurate. Next, at the fine level, we introduce a multiscale-high-order deep learning strategy based on ResNet model which can learn robust structure context features for final road extraction step. Based on the results of LDMM, the multiscale-high-order strategy can further remove false positives which have different structure features with roads. Compared with a single scanning size ResNet, our multiscale-high-order strategy can learn higher-order context information, leading to better performances. We test our algorithm on Shaoshan dataset. Experiments illustrate our better performance compared with other six state-of-the-art methods.

The point clouds acquired by a vehicle-borne mobile laser scanning (MLS) system have shown great potential for many applications such as intelligent transportation systems, road infrastructure inventories, and high-definition (HD) maps to support the advanced driver-assistance systems (ADAS) and autonomous vehicles (AVs). This paper presents a novel two-step approach to automated detection and reconstruction of three-dimensional (3D) highway curves from MLS point clouds. However, when dealing with noisy, unstructured, dense point clouds, we often face some challenges, most notably in handling of the outliers introduced during road marking detection and in recognition of curve types during 3D curve reconstruction. Our approach is formed by two main algorithms: a detector based on intensity variance and a robust model fitting estimator. The experimental results obtained using both a virtual scan dataset and a real MLS dataset demonstrated that our approach is very promising in handling of the outliers and reconstruction of 3D road curves. Specifically, a relative accuracy of 0.6% has been achieved in estimation of circle radii based on the virtual scan dataset. A comparative study also showed that our road marking detection approach is more effective and more stable than state-of-the-art approaches.

Deep learning techniques have been widely applied to traffic flow prediction, considering underlying routine patterns, and multiple context factors (e.g., time and weather). However, the complex spatio-temporal dependencies between inherent traffic patterns and multiple disturbances have not been fully addressed. In this paper, we propose a two-phase end-to-end deep learning framework, namely DeepSTD to uncover the spatio-temporal disturbances (STD) to predict the citywide traffic flow. In the STD Modeling phase, we propose an STD modeling method to model both the different regional disturbances caused by various region functions and the spatio-temporal propagating effects. In the Prediction phase, we eliminate the STD from the historical traffic flow to enhance the leaning of inherent traffic patterns and combine the STD at the prediction time interval to consider the future disturbances. The experimental results on two real-world datasets demonstrate that DeepSTD outperforms the state-of-the-art methods.

Mobile laser scanning (MLS) systems mainly comprise laser scanners and mobile mapping platforms. Typical MLS systems are able to acquire three-dimensional point clouds with 1-10 centimeter point spacing at a normal driving or walking speed in the street or indoor environments. The MLS' advantages of efficiency and stability make it a quite practical tool for three-dimensional urban modeling. This paper reviews the latest advances in 3D modeling of the LiDAR-based mobile mapping system (MMS) point cloud, including LiDAR Simultaneous Localization and Mapping (SLAM), point cloud registration, feature extraction, object extraction, semantic segmentation, and deep learning processing. Then typical urban modeling applications based on MMS are also discussed.

This paper deals with the geometric multi-model fitting from noisy, unstructured point set data (e.g., laser scanned point clouds). We formulate multi-model fitting problem as a sequential decision making process. We then use a deep reinforcement learning algorithm to learn the optimal decisions towards the best fitting result. In this paper, we have compared our method against the state-of-the-art on simulated data. The results demonstrated that our approach significantly reduced the number of fitting iterations.

We present a novel deep convolutional network pipeline, LO-Net, for real-time lidar odometry estimation. Unlike most existing lidar odometry (LO) estimations that go through individually designed feature selection, feature matching, and pose estimation pipeline, LO-Net can be trained in an end-to-end manner. With a new mask-weighted geometric constraint loss, LO-Net can effectively learn feature representation for LO estimation, and can implicitly exploit the sequential dependencies and dynamics in the data. We also design a scan-to-map module, which uses the geometric and semantic information learned in LO-Net, to improve the estimation accuracy. Experiments on benchmark datasets demonstrate that LO-Net outperforms existing learning based approaches and has similar accuracy with the state-of-the-art geometry-based approach, LOAM.

This paper proposes a new end-to-end trainable matching network based on receptive field, RF-Net, to compute sparse correspondence between images. Building end-to-end trainable matching framework is desirable and challenging. The very recent approach, LF-Net, successfully embeds the entire feature extraction pipeline into a jointly trainable pipeline, and produces the state-of-the-art matching results. This paper introduces two modifications to the structure of LF-Net. First, we propose to construct receptive feature maps, which lead to more effective keypoint detection. Second, we introduce a general loss function term, neighbor mask, to facilitate training patch selection. This results in improved stability in descriptor training. We trained RF-Net on the open dataset HPatches, and compared it with other methods on multiple benchmark datasets. Experiments show that RF-Net outperforms existing state-of-the-art methods.

3D objects detected from real-world data are usually incomplete in different degrees. Objects with different degrees of incompleteness should be treated and processed separately. This paper proposes a framework for partial 3D object retrieval and completeness evaluation in an urban street scene based on mobile laser scanning (MLS) point cloud data. The framework consists of three parts. A deep learning method is first used to detect objects from 3D point cloud data. Then, for each detected object, the most similar object in the reference dataset, which contains complete objects, is obtained by a partial 3D shape retrieval method. Last, a completeness evaluation of the detected object is conducted by calculating the completeness index that reflects the integrity of the detected object, and a missing part prediction is given to guide further completion. The proposed framework is validated on the public dataset KITTI and our own point cloud dataset. The experiment includes 3D detection, the partial 3D shape retrieval, and the completeness evaluation. Results show the good performance of the object detection and partial shape retrieval, also a reasonable evaluation of objects completeness.

This paper proposes a novel approach for indoor point clouds quality evaluation, which works well without reference point clouds. In this paper, we mainly evaluate indoor point clouds quality in two aspects: the smoothness of the walls and the degree of occlusion of the walls and the floor. Our approach involves three steps. Firstly, with the S3DIS dataset, we use a deep learning method to train a detector to label walls and floor from indoor scenes. Next, we calculate the normal vector of the wall and the normal vector of each point on the wall. The degree of smoothness of the wall is judged according to the angle between the normal vector of the wall and the normal vectors of the points. Finally, according to the cause of occlusion (objects on the floor or in front of the wall), the occlusion degree of the wall and the floor is obtained. The experimental results demonstrate that the proposed method is suitable for non-reference indoor point cloud quality evaluation.

This paper presents a novel method for reconstruction of three-dimensional (3D) zebra crossings from mobile laser scanning (MLS) point clouds. Firstly, we extract the zebra crossings from the 3D point cloud data in data preprocessing. Secondly, the fitting model is generated by seven parameters to determine one plane commonly and then calculating similarity for fitting the zebra crossings point clouds. Finally, the cuckoo search algorithm is used to adjust the parameters and optimize the results to obtain the optimal model and the geometric information of the zebra crossings can be acquired simultaneously. The proposed algorithm is tested on a set of point-clouds acquired by a RIEGL VMX-450 LiDAR system. The experimental results show the feasibility and stability of our method and the zebra crossings of urban road can be automatically and effectively reconstructed.

Vehicle detection and recognition in remote sensing images are challenging, especially when only limited training data are available to accommodate various target categories. In this paper, we introduce a novel coarse-to-fine framework, which decomposes vehicle detection into segmentation-based vehicle localization and generalized zero-shot vehicle classification. Particularly, the proposed framework can well handle the problem of generalized zero-shot vehicle detection, which is challenging due to the requirement of recognizing vehicles that are even unseen during training. Specifically, a hierarchical DeepLab v3 model is proposed in the framework, which fully exploits fine-grained features to locate the target on a pixel-wise level, then recognizes vehicles in a coarse-grained manner. Additionally, the hierarchical DeepLab v3 model is beneficially compatible to combine the generalized zero-shot recognition. To the best of our knowledge, there is no publically available dataset to test comparative methods, we therefore construct a new dataset to fill this gap of evaluation. The experimental results show that the proposed framework yields promising results on the imperative yet difficult task of zero-shot vehicle detection and recognition.

This paper presents a novel virtual-real registration approach for augmented reality (AR) in large-scale outdoor environments. Essentially, it is a pose estimation for the mobile camera images (ground camera images) in 3D model recovered by Unmanned Aerial Vehicle (UAV) image sequence via Structure-From-Motion (SFM) technology. The approach considers to indirectly establish the spatial relationship between 2D and 3D space by inferring the transformation relationship between the ground camera images and the UAV 3D model rendered images. Specifically, the proposed approach can overcome the positioning errors, which are deterioration and drift in the GPS, and deviation of orientation. The experimental results demonstrate the possibility of the proposed virtual-real registration approach, and show that the approach is robust, efficient and intuitive for AR in large-scale outdoor environments.

Object classification model used in point clouds requires to be retrained when applying on heterogeneous point clouds that are collected from different kinds of laser scanning systems. The model training procedure needs supervised information from the corresponding laser scanning systems. However, the acquisition of supervised information is labor-intensive and time-consuming. This letter proposes a new framework to exploit objects from point clouds with supervised information (source domain) to directly classify objects from heterogeneous point clouds with no supervised information (target domain). More specifically, to alleviate occlusions and point density variations, intraclass variations in object classification, the proposed framework integrates a bag-of-words model to effectively describe 3-D objects of point clouds. To alleviate the difference of point clouds collected from various devices, a joint distribution adaption model is exploited to build an effective feature transformation to accomplish the adaption in the source and target domains. The proposed framework is validated on data sets, which contain three kinds of point clouds. Extensive experiments show the effectiveness of the proposed framework on classifying 3-D objects in heterogeneous point clouds.

This paper presents a framework to build a joint 2-D-3-D traffic sign landmark data set for geo-localization using mobile laser scanning (MLS) data. The MLS data include 3-D point clouds and corresponding multi-view images. First, an integrated method, based on a deep learning network and the retro-reflective properties of traffic signs, is developed to accurately extract traffic signs from MLS point clouds. Next, the semantic and spatial properties of the traffic signs (type, location, position, and geometric characteristics) are obtained. Then, a joint 2-D-3-D traffic sign landmark data set is built, and a semantic-spatial organization graph is used to organize the traffic sign data set. Last, based on the traffic sign landmark data set, a geo-localization method for a driving car is proposed to estimate the driving trajectory. It can be used for auxiliary positioning of autonomous vehicles. Experimental results demonstrate the reliability of our proposed method for traffic sign detection and the potential of building 2-D-3-D traffic sign landmark data set for driving trajectory estimation from MLS data.

Learning autonomous-driving policies is one of the most challenging but promising tasks for computer vision. Most researchers believe that future research and applications should combine cameras, video recorders and laser scanners to obtain comprehensive semantic understanding of real traffic. However, current approaches only learn from large-scale videos, due to the lack of benchmarks that consist of precise laser-scanner data. In this paper, we are the first to propose a LiDAR-Video dataset, which provides large-scale high-quality point clouds scanned by a Velodyne laser, videos recorded by a dashboard camera and standard drivers' behaviors. Extensive experiments demonstrate that extra depth information help networks to determine driving policies indeed.

Single image denoising and super-resolution are sitting in the core of various image processing and pattern recognition applications. Typically, these two tasks are handled separately, without regarding to joint reinforcement and learning. The former deals with equal-size pixel-to-pixel translation, while the latter deals with scaling up amount of input pixels. In this paper, we propose a Generative Adversarial Network(GAN) towards joint learning of single image denoising and super-resolution. In principle, our design allows both tasks to share several common building blocks, with the linking between both outputs to reinforce each other. Such a reinforcement is accomplished via designing a novel generative network through optimizing a novel loss function to achieve both denoising and super-resolution. Quantitatively comparing to a set of alternative approaches and baselines, the experiment demonstrated superior performance our method in denoising and super-resolution with high upscaling factors.

Given the difficulty in labeling sufficient amount of instances across different resolutions and imaging environment of satellite images, weakly supervised vehicle detection is with great importance for satellite images analysis and processing. To prevent such cumbersome and meticulous manual annotation, naturally we have introduced the weakly supervised detection that has recently explosively prevalent in ordinary viewing angle images. Our program merely stands in need of region-level group annotation, i.e., whether this district convers vehicle(s) without plainly pointing out the coordinates of vehicles. There are two major problems are often encountered for Weakly Supervised Object Detection. One is that it is often chooses only a most expressive instance contains multiple target objects which often have a bigger probability when selecting a target block. For this problem, the number of vehicles can be estimated based on the object counting, a combinatorial selection algorithm can be used to select patch which contains at most one vehicle instance. Another problem is that precise object positioning becomes more difficult due to the lack of instance-level supervision. This problem can be optimized by a progressive learning strategy. Experiments was carried on wide-ranging remote sensing dataset and achieved better results compared to the state-of-the-art weakly supervised vehicle detection schemes.

Existing ridesharing services have focused on on-demand trip matching, which resembles traditional taxi dispatching. This may encourage more private vehicles on the road, which aggravate traffic congestions in peak hours rather than alleviating them. We propose CommuteShare, a novel ridesharing service for daily commuters that encourages long-term ridesharing among commuters with similar commuting patterns, to increase the traffic efficiency in peak hours. We first identify commuting private vehicles (CPVs) from traffic records and model their commuting patterns. We then design a dynamic model to formulate the intention level of a CPV driver to offer a ride based on the spatio-temporal convenience and dynamic traffic conditions. Based on the commuting patterns of the CPVs and the dynamic model of the CPV drivers, we propose a ridesharing algorithm to compute ridesharing matches among CPVs. We perform extensive experiments on three real-world cross-domain urban big datasets from a major city of China. Experimental results show that, using the proposed CommuteShare service, over 5,300 private vehicles can be reduced daily on average during morning peak hours, with a reduction of 7-minute average waiting time for the riders.

In recent years, the compact polarimetric SAR (CP SAR) imaging mode has received much attention because of its advantages in swath width compared to the quad-polarization mode and in information of scattering targets compared to the linear dual-polarization mode, respectively. For object detection (e.g., ship, sea ice, and drilling platform) in maritime monitoring which has been widely discussed, it is still a challenge to eliminate the false alarms caused by ocean clutter. Accordingly, a modified feature-based framework for ship detection using CP SAR data was proposed in this paper. In particular, a Guard-filter was used after feature extraction to reduce the effect of ocean clutter. Several simulated CP SAR imageries based on Gaofen-3 quad-polarization SAR image were selected for further investigations on ship detection. A SVM classifier was included in the framework to do a pixel-wise classification. The result shows that the proposed method has a better performance compared to traditional methods (e.g., constant false alarm rate and polarimetric whitening filter) and the feature-based method without filtering. Overall false alarm rates are decreased by 35% with the proposed method.

With the technology enhanced, 3D mobile light detection and ranging (LiDAR) can produce more accurate 3D information for the objects. Meanwhile, hyperspectral remote sensing has more number of wavelengths and provides a higher resolution spectrum of objects. This paper proposes a multi-sensor platform to provide these two data for health detection at the individual tree level in urban environments. We firstly locate and segment the suspected tree objects by ground removal and Euclidean distance clustering. Then we take use of spectrum to remove non-tree objects, e.g., buildings, light poles. After that, we use LiDAR data to compute the geometric parameters of each tree and hyperspectral data to analyze its health situation.

It is challenging to direct extract the feature descriptors of the object in the point cloud, although deep learning has been widely used with the classification and detection in the point cloud, those methods hidden feature presentation in the network. Since the point cloud scanned by the Laser Scanner usually have different point density, unordered and even the different occlusion, which go beyond the reach of handcrafted descriptors, e.g. FPH, FPFH, VFH, ROPS. In this paper, we aim to direct extract the feature descriptors of the point cloud object through the raw point cloud. Inspired by the recent success of the Siamese networks[6], PointNet[7] and PointNet++[8], we propose a novel network to direct extract the feature descriptors of the whole point cloud object. We train our network with the Euclidean distance as the loss function which reflects feature descriptors similarity. The experiment object datasets were acquired by Mobile Laser Scanning (MLS) system which contains 6 categories. Experiment result shows that our network has a robust generalization, which can well direct extract the feature descriptors of the whole point cloud object.

Road network matching has played an important role in road network extraction and update, yet has got extensive researching during the recent decades. Differ from previous road matching methods focus mainly on the city areas, which have accurate and regular road networks, this paper aim to address the matching between incomplete ground survey road network and extracted road network from remote sensing images. Specifically, we propose an extending line based matching scheme to calculate the road primitive similarity by taking into account the surrounding connections and contextual information. The experimental results show that the proposed method is able to provide high quality matching results, even the ground survey data are very different from the extracted road network of the satellite. Thus makes it possible to implement the road network update for the wide rural regions without interested ground survey road network data.

In forest monitoring and management, any rational decision needs to be based on forest parameters. The diameter at breast height (DBH) of a tree is considered to be the most significant parameter among them. This paper presents a novel method for extracting tree stems and estimating DBH of trees in a forest environment from 3D point clouds data acquired by a terrestrial laser scanning (TLS) system. In the proposed method, a downward-growing algorithm is used to extract individual tree stems and DBH of trees are estimated by the circle fitting algorithm. This proposed method can avoid errors caused from tilted trees by estimating a plane perpendicular to the tree stem. With this method, 17 trees were extracted from single-scan point cloud data consisting of 21 trees. The estimated DBH had a bias of 0.38 cm and a root mean squared error of 1.76 cm, These experiment results show the feasibility of the proposed method.

The prediction of traffic flow is of great significance for the prevention of accidents, the avoidance of congestion and the dispatch of command center. Considering the complexity of traffic data in reality, it is an extraordinarily challenging task to forecast accurately from historical patterns. In this paper, we propose a method based on the cascaded artificial neural network (CANN) to predict traffic flow at positions. In order to express the spatial correlation of traffic data, the actual road network distance is introduced in our model. The realworld data derived from video surveillance cameras in Xiamen is used in the experiment which is compared with five baselines. To the best of our knowledge, this is the first time that CANN is applied to forecast traffic flow. The experimental results demonstrate that the CANN method has superior performance. In addition, We also discuss the impact of some external factors such as temperature, weather and holidays on the prediction results.

Describing the same scene with different imaging style or rendering image from its 3D model gives us different domain images. Different domain images tend to have a gap and different local appearances, which raise the main challenge on the cross-domain image patch matching. In this paper, we propose to incorporate AutoEncoder into the Siamese network, named as H-Net, of which the structural shape resembles the letter H. The H-Net achieves state-of-the-art performance on the cross-domain image patch matching. Furthermore, we improved H-Net to H-Net++. The H-Net++ extracts invariant feature descriptors in cross-domain image patches and achieves state-of-the-art performance by feature retrieval in Euclidean space. As there is no benchmark dataset including cross-domain images, we made a cross-domain image dataset which consists of camera images, rendering images from UAV 3D model, and images generated by CycleGAN algorithm. Experiments show that the proposed H-Net and H-Net++ outperform the existing algorithms. Our code and cross-domain image dataset are available at https://github.com/Xylon-Sean/H-Net.

This letter presents a line structure-based method for integration of centimeter-level indoor backpacked scanning point clouds and millimeter-level outdoor terrestrial laser scanning point clouds. Using 3-D lines for registration, instead of matching points directly, can improve the robustness of the method and adapt to multisource point cloud data of different qualities. Considering the limited overlapping between indoor and outdoor scenes, line structures are extracted from overlapped wall areas that may be included in interior and exterior data. Here, a patch-based method labels a point cloud into wall, ceiling, floor categories, as well as assigning the candidate overlapping walls. Then, lines structures are extracted from the wall plane point cloud. Potential door and window line structures are detected and refined for point cloud registration. Last, an iterative closest point-based method is used to fine tune the registration results. Our results show that the proposed method effectively integrates a promising map of indoor and outdoor scenes.

Unsupervised learning of a convolutional neural network (CNN) is a feasible method to represent and classify remote sensing images, where labeling the observed data to prepare training samples is a highly expensive and time-consuming task. In this letter, we propose an unsupervised convolutional feature fusion network to formulate an easy-to-train but effective CNN representation of remote sensing images. The efficiency and effectiveness are derived from the following two aspects. First, the proposed method trains a deep CNN through unsupervised learning of each CNN layer in a greedy layer-wise manner, which makes the training relatively easy and efficient. Second, the feature fusion strategy in the proposed network can effectively use both the information from individual layers and the important interactions between different layers. As a result, the proposed network requires only several layers to obtain comparable or even better results than very deep networks. The experiments on unsupervised deep representations and the classification of remote sensing images demonstrate the efficiency and effectiveness of the proposed method.

Using mobile Light Detection and Ranging point clouds to accomplish road scene labeling tasks shows promise for a variety of applications. Most existing methods for semantic labeling of point clouds require a huge number of fully supervised point cloud scenes, where each point needs to be manually annotated with a specific category. Manually annotating each point in point cloud scenes is labor intensive and hinders practical usage of those methods. To alleviate such a huge burden of manual annotation, in this paper, we introduce an active learning method that avoids annotating the whole point cloud scenes by iteratively annotating a small portion of unlabeled supervoxels and creating a minimal manually annotated training set. In order to avoid the biased sampling existing in traditional active learning methods, a neighbor-consistency prior is exploited to select the potentially misclassified samples into the training set to improve the accuracy of the statistical model. Furthermore, lots of methods only consider short-range contextual information to conduct semantic labeling tasks, but ignore the long-range contexts among local variables. In this paper, we use a higher order Markov random field model to take into account more contexts for refining the labeling results, despite of lacking fully supervised scenes. Evaluations on three data sets show that our proposed framework achieves a high accuracy in labeling point clouds although only a small portion of labels is provided. Moreover, comparative experiments demonstrate that our proposed framework is superior to traditional sampling methods and exhibits comparable performance to those fully supervised models.

Effective extraction of road boundaries plays a significant role in intelligent transportation applications, including autonomous driving, vehicle navigation, and mapping. This paper presents a new method to automatically extract 3-D road boundaries from mobile laser scanning (MLS) data. The proposed method includes two main stages: supervoxel generation and 3-D road boundary extraction. Supervoxels are generated by selecting smooth points as seeds and assigning points into facets centered on these seeds using several attributes (e.g., geometric, intensity, and spatial distance). 3-D road boundaries are then extracted using the α-shape algorithm and the graph cuts-based energy minimization algorithm. The proposed method was tested on two data sets acquired by a RIEGL VMX-450 MLS system. Experimental results show that road boundaries can be robustly extracted with an average completeness over 95%, an average correctness over 98%, and an average quality over 94% on two data sets. The effectiveness and superiority of the proposed method over the state-of-the-art methods is demonstrated.

Automatic annotation of 3D objects in cluttered scenes shows its great importance to a variety of applications. Nowadays, 3D point clouds, a new 3D representation of real-world objects, can be easily and rapidly collected by mobile LiDAR systems, e.g. RIEGL VMX-450 system. Moreover, the mobile LiDAR system can also provide a series of consecutive multi-view images which are calibrated with 3D point clouds. This paper proposes to automatically annotate 3D objects of interest in point clouds of road scenes by exploiting a multitude of annotated images in image databases, such as LabelMe and ImageNet. In the proposed method, an object detector trained on the annotated images is used to locate the object regions in acquired multi-view images. Then, based on the correspondences between multi-view images and 3D point clouds, a probabilistic graphical model is used to model the temporal, spatial and geometric constraints to extract the 3D objects automatically. A new dataset was built for evaluation and the experimental results demonstrate a satisfied performance on 3D object extraction.

Unlicensed taxis are widely considered as major obstacles to city traffic regulation and public safety. Thus, many governments have issued restrictions for car-hailing services and alleged that the use of unlicensed vehicles was illegal. However, it is very challenging that traffic administrative enforcements face limited manpower to prohibit unlicensed taxis, due to costly and time-consuming procedure of on-site evidence collection. In this paper, we propose an effective service to incorporate human mobility mechanism into unlicensed taxis detection from massive city-wide vehicles. We first extract 276 spatio-temporal features, which are grouped into two categories, including daily behaviors and sustainable behaviors to capture the mobility characteristics of unlicensed taxis. Second, we investigate the detection accuracy of three machine learning techniques, viz. support vector machines, decision tree, and logical regression. We illustrate our approach using real-world vehicle license plate recognition dataset in Xiamen, China, which contains 336 million passing records for 6.2 million vehicles filmed by 439 devices in August 2016. Experimental results reveal that LR outperforms SVM and DT in prediction accuracy and F-score measurement, while SVM is capable of identifying the largest number of unlicensed taxis.

Extreme climate events are projected to increase under the context of global warming associated with the increase in greenhouse gas emissions. In particular, coastal regions which are also characterized with significant urbanization will be vulnerable to the extreme events, such as severe typhoon. Timely assessment and accurate information on the extent and severity of the damage caused by extreme event is necessary to better facilitate decision-making for disaster alleviation and post-recovery. Satellite remote sensing has an important advantage in assessing the impacts caused by extreme events on natural environment and socio-economic dimension at a variety of spatial and temporal scales. Surface greenness disturbance mainly over Xiamen, China, caused by Typhoon Meranti (2016) was investigated using a pair of observations by Landsat 7 ETM+ and Landsat 8 OLI. Significant decreases in greenness were detected, which mainly located along the Maluan Bay and the settlements in suburban and rural areas surrounding the track of typhoon.

This paper presents a standardized quality criteria to evaluate the 3D point cloud model of the indoor building which is based on point cloud's data accuracy, the prior characteristics of the building and the coincidence errors of the point cloud model. Our assessment framework involves three steps: the point cloud data acquisition, model generation and quality evaluation. In model generation progress, incapacity of scanning the whole building information one time since its multi-storied spatial structure, the building model need to be registered and merged. In evaluation step, taking into account the need of mapping, indoor location and navigation, the establishment of an interior spatial building model requires accurate measurement. Therefore, we adopt data noise analysis to give a judgement. Then, since the geometric characteristics of the building model are varying, the geometric analysis is proposed to evaluate acquisition errors and registration error. Comparative experiments demonstrate our method give integrate, realistic and reliable quality framework for the indoor building point cloud model.

Airborne acquisition and ground-view 3D point cloud provide complementary 3D information at city scale. A complete but lacks ground-view details, while the latter is incomplete for higher floors and severe occlusion. In this paper, First, a volumetric fusion method based on graph cuts were applied for fusing of airborne and terrestrial 3D LiDAR data. Second, we propose a method of constraints based on the local centroid of point cloud to eliminate the gap of fusion boundary. Finally, the experiments show that the improved fusion algorithm implement blending effectively.

In this paper, we propose a novel approach for road width measurement from high resolution satellite or aerial images. The proposed approach has three main steps. First, we extract line segments and road center lines on the given remote sensing images. Second, we could obtain many pairs of parallel lines with width information by computing the positional relationship between each other. Then K-means is performed to cluster these parallel lines into several clusters by the width information of them. Finally, an energy function is introduced to assign the width range of a cluster to each pixel on road center lines, the width range is viewed as the width of the corresponding road segment. Attribute to parallel lines extraction, parallel lines clustering and our energy function, the proposed road width measurement method is able to provide high quality results on road width measurement.

This paper proposes a novel approach for traffic sign detection and rapid damage inspection in natural scenes based on mobile laser scanning (MLS) data, including images and point clouds. The inspection results assist traffic management departments to take immediate measures to update and maintain traffic signs after natural disasters leading to many damaged traffic signs. Our approach involves four steps: Firstly, we use a deep learning network, Fast regions with convolutional neural network (Fast R-CNN), to train a traffic sign detector in an open benchmark, where the images are more variable and have a higher resolution. Then, traffic signs in images are detected by using the trained detector. Next, the area of the traffic sign, based on the sign area in the image, is roughly detected in MLS point clouds. Then, an accurate traffic sign is detected. Finally, some placement parameters of the traffic sign are measured for damage inspection and further inventory. Our proposed approach is validated on a set of point-clouds acquired by a RIEGL VMX-450 MLS system. Experimental results demonstrate that the rapidity and reliability of our proposed approach in traffic sign detection and damage inspection are robust.

Recently, the classification of tree species using 3-D point clouds has drawn wide attention in surveys and forestry investigations. This letter proposes a new voxel-based deep learning method to classify tree species in 3-D point clouds collected from complex forest scenes. The proposed method includes three steps: 1) individual tree extraction based on the density of the point clouds; 2) low-level feature representation through voxel-based rasterization; and 3) classification of tree species by a deep learning model. Two data sets of 3-D forest point clouds acquired by terrestrial laser scanning systems are used to evaluate the proposed method. The method achieves an average classification accuracy of 93.1% and 95.6% on the two data sets. Furthermore, in comparative experiments, the proposed method exhibits performance superior to that of the other 3-D tree species classification methods.

As one of the most common features in the man-made environments, straight lines play an important role in many applications. In this paper, we present a new framework to extract line segments from large-scale point clouds. The proposed method is fast to produce results, easy for implementation and understanding, and suitable for various point cloud data. The key idea is to segment the input point cloud into a collection of facets efficiently. These facets provide sufficient information for determining linear features in the local planar region and make line segment extraction become relatively convenient. Moreover, we introduce the concept “number of false alarms” into 3-D point cloud context to filter the false positive line segment detections. We test our approach on various types of point clouds acquired from different ways. We also compared the proposed method with several other methods and provide both quantitative and visual comparison results. The experimental results show that our algorithm is efficient and effective, and produce more accurate and complete line segments than the comparative methods. To further verify the accuracy of the line segments extracted by the proposed method, we also present a line-based registration framework, which employs these line segments on point clouds registration.

In this paper, we propose a task-oriented enhancing technique for extracting road networks from satellite images. By exploiting an approximate estimation of the potential road edges for guidance, we developed a joint enhancing filtering framework to generate a version of the input image that facilitates road network extraction. First, an adaptive smoothing scheme is designed to suppress the interference of noise or heavy textures, such as residential areas or terrain boundaries. By combining this scheme with the proposed novel anisotropic shock filter, the edges of the potential road regions can be kept sharp and clear. Through abundant experimental comparisons with state-of-the-art filtering techniques and quantitative evaluations using data from various satellite sensors, the performance of the proposed approach is comprehensively evaluated. The experimental results demonstrate that our system can address heavy high contrast textures and provide a meaningful improvement in the feature detection for road extraction.

For survey and maintenance of traffic signs, this paper presents a novel traffic sign occlusion detection method using 3-D point clouds and trajectory data acquired by a mobile laser scanning system. To produce a maintenance guide, our method aims to obtain the degree of occlusion by analyzing the spatial relationship between traffic signs, surroundings, and drivers on the road. First, a detection method considering both reflectance and geometric features is developed to capture traffic signs. Next, to simulate the driver's view, a trajectory-based method is proposed to determine driver's observation location and the corresponding observed traffic sign. Finally, to determine whether a traffic sign is in occlusion, a hidden point removal algorithm is adopted and carried out. Furthermore, we develop two indices to evaluate the degree of occlusion. The proposed method is tested using two point cloud data sets collected by an RIEGL VMX-450 system along a 23.68-km-long urban road. The obtained results illustrate the feasibility of the proposed occlusion detection method.

This paper presents a supervoxel-based approach for automated localization and extraction of street light poles in point clouds acquired by a mobile LiDAR system. The method consists of five steps: preprocessing, localization, segmentation, feature extraction, and classification. First, the raw point clouds are divided into segments along the trajectory, the ground points are removed, and the remaining points are segmented into supervoxels. Then, a robust localization method is proposed to accurately identify the pole-like objects. Next, a localization-guided segmentation method is proposed to obtain pole-like objects. Subsequently, the pole features are classified using the support vector machine and random forests. The proposed approach was evaluated on three datasets with 1,055 street light poles and 701 million points. Experimental results show that our localization method achieved an average recall value of 98.8%. A comparative study proved that our method is more robust and efficient than other existing methods for localization and extraction of street light poles.

Vehicle detection in satellite image has attracted extensive research attentions with various emerging applications.However, the detector performance has been significantly degenerated due to the low resolutions of satellite images, as well as the limited training data.In this paper, a robust domain-adaptive vehicle detection framework is proposed to bypass both problems.Our innovation is to transfer the detector learning to the high-resolution aerial image domain,where rich supervision exists and robust detectors can be trained.To this end, we first propose a super-resolution algorithm using coupled dictionary learning to ``augment'' the satellite image region being tested into the aerial domain.Notably, linear detection loss is embedded into the dictionary learning, which enforces the augmented region to be sensitive to the subsequent detector training.Second, to cope with the domain changes, we propose an instance-wised detection using Exemplar Support Vector Machines (E-SVMs), which well handles the intra-class and imaging variations like scales, rotations, and occlusions.With comprehensive experiments on large-scale satellite image collections, we demonstrate that the proposed framework can significantly boost the detection accuracy over several state-of-the-arts.

In this paper, we propose a learning-based road network extraction scheme from high resolution satellite. First, the convolutional neural network (CNN), which is able to capture large context of local structures, are applied to predict the probability of a pixel belonging to road regions, and assign labels to each pixel to describe whether it is road. Then, a line integral convolution based algorithm is developed to smooth the rough map to connect small gaps. Finally, by combining with some common image processing operators, road centerlines are able to be acquired. Attribute to the learning capacity of CNN, and the line integral convolution based connection scheme, the proposed road extraction method is able to provide high quality results comparing to current state-of-art road extraction methods.

Coastline extraction in Synthetic aperture radar (SAR) images is a fundamental and challenging task due to the speckle noise. In this paper, we propose a new method for automatic coastline extraction in SAR images. In our method, we combine K-means and speckle noise removal methods together to increase the dissimilarity between sea and land. To enhance the robustness to speckle noise, and preserve the targets boundaries, we treat superpixels as basic regions instead of pixels in traditional pixel-based methods. Finally, an adaptive threshold is applied to classify these regions into sea or land. Based on the classifications, a canny detector is employed to detect the coastline. We evaluate our proposed method on SAR images and the improved coastline extraction method superpixel-based is verified on remote sensing images with RGB channels. The experimental results demonstrate its superior performance on coastline extraction.

With rapid development of light detection and ranging (LiDAR) technologies, three dimensional point clouds increasingly become a new approach to sense the world. In our previous work, light poles were detected from mobile LiDAR point clouds without using their locations. In this paper, we improve our previous work by considering location information between two neighboring light poles to reduce false alarm. In the proposed method, the potential light poles are first detected by the extended Hough Forest Framework. Then, a gaussian distribution is exploited to model the distance between two light poles by using locations of those detected light poles. Finally, inaccurately detected light poles are removed by considering the distance between two adjacent objects. We evaluate our proposed method on mobile LiDAR point clouds acquired by RIEGL VMX-450 system. On the basis of the experimental test instances, we demonstrate improved accuracy on light pole detection.

LIDAR (Light Detection and Ranging) is an especially effective tool for acquiring geo-referenced point clouds of urban site. Accurate extraction of elevated features such as building rooftops is vitally important in various applications. However, it is still challenging to determine an accurate rooftop contour from the irregularly distributed LIDAR point clouds. In this paper an efficient LIDAR segmentation method is presented in order to achieve automated rooftop extraction. First, we apply a voxel-based upward growing algorithm that filters out the ground points from the raw point cloud scenes. Second, we employ a Euclidean based clustering method on non-ground points by making use of nearest neighbors. Then we introduce RANSAC (RANdom SAmple Consensus) technique to estimate primitive planes for fitting rooftop facets. Finally, we use concave hull and L0regularization to determine the rooftop contour. Accurate experimental results demonstrate the validity of our segmentation method for rooftop extraction.

This paper presents a new algorithm to directly extract 3D road boundaries from mobile laser scanning (MLS) point clouds. The algorithm includes two stages: 1) non-ground point removal by a voxel-based elevation filter, and 2) 3D road surface extraction by curb-line detection based on energy minimization and graph cuts. The proposed algorithm was tested on a dataset acquired by a RIEGL VMX-450 MLS system. The results fully demonstrate the effectiveness and superiority of the proposed algorithm.

This letter presents a backpack mapping system for creating indoor 2-D and 3-D maps of building interiors. For many applications, indoor mobile mapping provides a 3-D structure via an indoor map. Because there are significant roll and pitch motions of the indoor mobile mapping system, the need arises for a moving mobile system with 6 degrees of freedom (DOFs) (x, y, and z positions and roll, yaw, and pitch angles). First, we present a 6-DOF pose estimation algorithm by fusing 2-D laser scanner data with inertial sensor data using an extended Kalman filter-based method. The estimated 6-DOF pose is used as the initialized transformation for consecutive map alignment in 3-D map building. The 6-DOF pose gives a full 3-D estimation of the system pose and is used to accelerate the map alignment process and also align the two maps directly when there are few or no overlapping areas between the maps. Our results show that the proposed system effectively builds a consistent 2-D grid map and a 3-D point cloud map of an indoor environment.

This paper presents a study of vehicle detection from high-resolution aerial images. In this paper, a superpixel segmentation method designed for aerial images is proposed to control the segmentation with a low breakage rate. To make the training and detection more efficient, we extract meaningful patches based on the centers of the segmented superpixels. After the segmentation, through a training sample selection iteration strategy that is based on the sparse representation, we obtain a complete and small training subset from the original entire training set. With the selected training subset, we obtain a dictionary with high discrimination ability for vehicle detection. During training and detection, the grids of histogram of oriented gradient descriptor are used for feature extraction. To further improve the training and detection efficiency, a method is proposed for the defined main direction estimation of each patch. By rotating each patch to its main direction, we give the patches consistent directions. Comprehensive analyses and comparisons on two data sets illustrate the satisfactory performance of the proposed algorithm.

This paper presents an automated algorithm for rapidly and effectively detecting cars directly from large-volume 3-D point clouds. Rather than using low-order descriptors, a multilayer feature generation model is created to obtain high-order feature representations for 3-D local patches through deep learning techniques. To handle cars with different levels of incompleteness caused by data acquisition ways and occlusions, a hierarchical visibility estimation model is developed to augment Hough voting. Considering scale and orientation variations in the azimuth direction, a set of multiscale Hough forests is constructed to rotationally cast votes to estimate cars' centroids. Quantitative assessments show that the proposed algorithm achieves average completeness, correctness, quality, and F1-measure of 0.94, 0.96, 0.90, and 0.95, respectively, in detecting 3-D cars. Comparative studies also demonstrate that the proposed algorithm outperforms the other four existing algorithms in accurately and completely detecting 3-D cars from large-scale 3-D point clouds.

In this paper, we present a novel aperiodic directional structure measurement (ADSM) toward road network extraction. Based on the observations from Cognitive Psychology regarding the aperiodicity and local directionality, ADSM can well characterize roadlike structures independent of the spectral character and contrast. By exploiting such measurement as guidance, we construct a mask to denote potential road regions. Then, by combining with some common morphology operators, our approach is able to provide robust road centerlines efficiently. We evaluate our approach with data from various satellite sensors and make comprehensive comparisons with previous state-of-the-art methods. Experimental results demonstrate the merit using our ADSM as a metric to identify potential road structures, as well as the effectiveness and efficiency of our road network extraction system.

This paper presents an algorithm for vehicle detection in high-resolution aerial images through a fast sparse representation classification method and a multiorder feature descriptor that contains information of texture, color, and high-order context. To speed up computation of sparse representation, a set of small dictionaries, instead of a large dictionary containing all training items, is used for classification. To extract the context information of a patch, we proposed a high-order context information extraction method based on the proposed fast sparse representation classification method. To effectively extract the color information, the RGB color space is transformed into color name space. Then, the color name information is embedded into the grids of histogram of oriented gradient feature to represent the low-order feature of vehicles. By combining low- and high-order features together, a multiorder feature is used to describe vehicles. We also proposed a sample selection strategy based on our fast sparse representation classification method to construct a complete training subset. Finally, a set of dictionaries, which are trained by the multiorder features of the selected training subset, is used to detect vehicles based on superpixel segmentation results of aerial images. Experimental results illustrate the satisfactory performance of our algorithm.

Semantic labeling of road scenes using colorized mobile LiDAR point clouds is of great significance in a variety of applications, particularly intelligent transportation systems. However, many challenges, such as incompleteness of objects caused by occlusion, overlapping between neighboring objects, interclass local similarities, and computational burden brought by a huge number of points, make it an ongoing open research area. In this paper, we propose a novel patch-based framework for labeling road scenes of colorized mobile LiDAR point clouds. In the proposed framework, first, three-dimensional (3-D) patches extracted from point clouds are used to construct a 3-D patch-based match graph structure (3D-PMG), which transfers category labels from labeled to unlabeled point cloud road scenes efficiently. Then, to rectify the transferring errors caused by local patch similarities in different categories, contextual information among 3-D patches is exploited by combining 3D-PMG with Markov random fields. In the experiments, the proposed framework is validated on colorized mobile LiDAR point clouds acquired by the RIEGL VMX-450 mobile LiDAR system. Comparative experiments show the superior performance of the proposed framework for accurate semantic labeling of road scenes.

This paper presents a spatial-related traffic sign inspection process for sign type, position, and placement using mobile laser scanning (MLS) data acquired by a RIEGL VMX-450 system and presents its potential for traffic sign inventory applications. First, the paper describes an algorithm for traffic sign detection in complicated road scenes based on the retroreflectivity properties of traffic signs in MLS point clouds. Then, a point cloud-to-image registration process is proposed to project the traffic sign point clouds onto a 2-D image plane. Third, based on the extracted traffic sign points, we propose a traffic sign position and placement inspection process by creating geospatial relations between the traffic signs and road environment. For further inventory applications, we acquire several spatial-related inventory measurements. Finally, a traffic sign recognition process is conducted to assign sign type. With the acquired sign type, position, and placement data, a spatial-associated sign network is built. Experimental results indicate satisfactory performance of the proposed detection, recognition, position, and placement inspection algorithms. The experimental results also prove the potential of MLS data for automatic traffic sign inventory applications.

This paper proposes a novel algorithm for detecting road scene objects (e.g., light poles, traffic signposts, and cars) from 3-D mobile-laser-scanning point cloud data for transportation-related applications. To describe local abstract features of point cloud objects, a contextual visual vocabulary is generated by integrating spatial contextual information of feature regions. Objects of interest are detected based on the similarity measures of the bag of contextual-visual words between the query object and the segmented semantic objects. Quantitative evaluations on two selected data sets show that the proposed algorithm achieves an average recall, precision, quality, and F-score of 0.949, 0.970, 0.922, and 0.959, respectively, in detecting light poles, traffic signposts, and cars. Comparative studies demonstrate the superior performance of the proposed algorithm over other existing methods.

This paper proposes a method for automated extraction of street trees in a typical urban environment from 3D point cloud data acquired by the mobile laser scanning system. First, the algorithm utilizes the voxel-based method to remove the ground points from the scene. Second, the Euclidean distance clustering is adopted to cluster points into individual objects. The eigenvalues of neighborhood covariance matrix and the corresponding normalized centroid distance are computed for each point to obtain the subdivided dimensional features. Finally, the statistical component features and horizontal information are calculated for object detection. The experiment results show the feasibility of the proposed algorithm.

This paper presents a novel method for traffic sign detection and visibility evaluation from mobile Light Detection and Ranging (LiDAR) point clouds and the corresponding images. Our algorithm involves two steps. Firstly, a detection algorithm based on high retro-reflectivity of the traffic sign from the MLS point clouds is designed for sign detection in complicated road scenes. To solve the spatial features of traffic signs, we also create geo-referenced relations between traffic signs and roads according to the normal of ground. Secondly, we propose a visibility estimation method to evaluate the visibility level of the traffic sign based on a combination of visual appearance and spatial-related features. The proposed algorithm is validated on a set of transportation-related point-clouds acquired by a RIEGL VMX-450 LiDAR system. The experiment results demonstrate that the efficiency and reliability of the proposed algorithm in detection traffic signs are robust, and also prove the potential of using mobile LiDAR data for traffic sign visibility evaluation.

This paper presents a novel approach for extracting street lighting poles directly from MLS point clouds. The approach includes four stages: 1) elevation filtering to remove ground points, 2) Euclidean distance clustering to cluster points, 3) voxel-based normalized cut (Ncut) segmentation to separate overlapping objects, and 4) statistical analysis of geometric properties to extract 3D street lighting poles. A Dataset acquired by a RIEGL VMX-450 MLS system are tested with the proposed approach. The results demonstrate the efficiency and reliability of the proposed approach to extract 3D street lighting poles.

We propose a new approach for hidden or potential object detection behind the vegetation based on the multi-echo small-footprint of light detection and ranging(LiDAR) point cloud. According to the specific characteristic that laser beam is able to penetrate foliage gaps, which offers the opportunity to perceive and detect the object invisible to the naked eye. First, the waveform sample data of the small footprint multi-echo liDAR uses a Gaussian fitting tool to curve fitting. Second, the peak detection of the waveform can be classified in statistical results of its wave numbers. Decomposition and correction are the processing for classifying based on the statistic data. After selecting the point clouds which are contained in the multi-peaks echoes, we obtain the tree and the embedded target behind it as well as ground elimination. The range of the waveform component is used to separate the penetrations material and the target by distance discriminant function. Experiments are implemented on the waveforms acquired by small-footprint LiDAR system VZ-1000 Sensor. The results indicate that the algorithm could provide an optimal solution for LiDAR waveform hidden target detection.

Super‐resolution (SR) is the problem of generating a high‐resolution (HR) image from one or more low‐resolution (LR) images. This study presents a new approach to single‐image super‐resolution based on group sparse representation. Two dictionaries are constructed corresponding to the LR and HR image patches, respectively. The sparse coefficients of an input LR image patch in terms of the LR dictionary are used to recover the HR patch from the HR dictionary. When constructing the dictionaries, the three colour channels in a training image patch are considered a group composed of three atoms. The whole group is selected simultaneously when representing an image patch so that the correlations between the colour channels can be retained. A dictionary training method is also designed in which the two dictionaries are trained jointly to ensure that the corresponding LR and HR patches have the same sparse coefficients. Experimental results demonstrate the effectiveness of the proposed method and its robustness to noise.

Occluded boundary detection in a 3-D point cloud is an indispensable preprocessing step for many applications, such as point cloud completion. Meanwhile, existing methods do not have the ability of distinguishing occluded and complete surface borders, such as the border of a sign board. To solve this problem, this letter presents an occluded boundary detection method for small-footprint LIDAR point clouds. The main novelty of this letter is using the last-echo information for occluded boundary detection. Seed boundary (SB) points are subsequently detected using this last-echo information. Finally, the SB points are grown into neighboring points using an occluded boundary growth algorithm. To the best of our knowledge, this method is the first method that uses the last-echo information to detect occluded boundaries. Experimental results with comparisons indicate that the proposed method can accurately and efficiently detect an occluded boundary without contamination from a complete surface border. These advantages allow the proposed method to benefit further applications such as point cloud completion, as demonstrated in the application section.

The accurate extraction of roads is a prerequisite for the automatic extraction of other road features. This letter describes a method for detecting road boundaries from mobile laser scanning (MLS) point clouds in an urban environment. The key idea of our method is directly constructing a saliency map on 3-D unorganized point clouds to extract road boundaries. The method consists of four major steps, i.e., road partition with the assistance of the vehicle trajectory, salient map construction and salient points extraction, curb detection and curb lowest points extraction, and road boundaries fitting. The performance of the proposed method is evaluated on the point clouds of an urban scene collected by a RIEGL VMX-450 MLS system. The completeness, correctness, and quality of the extracted road boundaries are 95.41%, 99.35%, and 94.81%, respectively. Experimental results demonstrate that our method is feasible for detecting road boundaries in MLS point clouds.

This letter presents a 3-D object matching framework to support information extraction directly from 3-D point clouds. The problem of 3-D object matching is to match a template, represented by a group of 3-D points, to a point cloud scene containing an instance of that object. A locally affine-invariant geometric constraint is proposed to effectively handle affine transformations, occlusions, incompleteness, and scales in 3-D point clouds. The 3-D object matching framework is integrated into 3-D correspondence computation, 3-D object detection, and point cloud object classification in mobile laser scanning (MLS) point clouds. Experimental results obtained using the 3-D point clouds acquired by a RIEGL VMX-450 system showed that completeness, correctness, and quality of over 0.96, 0.94, and 0.91 are achieved, respectively, with the proposed framework in 3-D object detection. Comparative studies demonstrate that the proposed method outperforms the two existing methods for detecting 3-D objects directly from large-volume MLS point clouds.

The assessment of pavement cracks is one of the essential tasks for road maintenance. This paper presents a novel framework, called ITVCrack, for automated crack extraction based on iterative tensor voting (ITV), from high-density point clouds collected by a mobile laser scanning system. The proposed ITVCrack comprises the following: 1) the preprocessing involving the separation of road points from nonroad points using vehicle trajectory data; 2) the generation of the georeferenced feature (GRF) image from the road points; and 3) the ITV-based crack extraction from the noisy GRF image, followed by an accurate delineation of the curvilinear cracks. Qualitatively, the method is applicable for pavement cracks with low contrast, low signal-to-noise ratio, and bad continuity. Besides the application to GRF images, the proposed framework demonstrates much better crack extraction performance when quantitatively compared to existing methods on synthetic data and pavement images.

This paper proposes a novel algorithm for extracting street light poles from vehicleborne mobile light detection and ranging (LiDAR) point-clouds. First, the algorithm rapidly detects curb-lines and segments a point-cloud into road and nonroad surface points based on trajectory data recorded by the integrated position and orientation system onboard the vehicle. Second, the algorithm accurately extracts street light poles from the segmented nonroad surface points using a novel pairwise 3-D shape context. The proposed algorithm is tested on a set of point-clouds acquired by a RIEGL VMX-450 mobile LiDAR system. The results show that road surfaces are correctly segmented, and street light poles are robustly extracted with a completeness exceeding 99%, a correctness exceeding 97%, and a quality exceeding 96%, thereby demonstrating the efficiency and feasibility of the proposed algorithm to segment road surfaces and extract street light poles from huge volumes of mobile LiDAR point-clouds.

This paper presents a survey of literature about road feature extraction, giving a detailed description of a Mobile Laser Scanning (MLS) system (RIEGL VMX-450) for transportation-related applications. This paper describes the development of automated algorithms for extracting road features (road surfaces, road markings, and pavement cracks) from MLS point cloud data. The proposed road surface extraction algorithm detects road curbs from a set of profiles that are sliced along vehicle trajectory data. Based on segmented road surface points, we create Geo-Referenced Feature (GRF) images and develop two algorithms, respectively, for extracting the following: 1) road markings with high retroreflectivity and 2) cracks containing low contrast with their surroundings, low signal-to-noise ratio, and poor continuity. A comprehensive comparison illustrates satisfactory performance of the proposed algorithms and concludes that MLS is a reliable and cost-effective alternative for rapid road inspection.

Updating road markings is one of the routine tasks of transportation agencies. Compared with traditional road inventory mapping techniques, vehicle-borne mobile light detection and ranging (LiDAR) systems can undertake the job safely and efficiently. However, current hurdles include software and computing challenges when handling huge volumes of highly dense and irregularly distributed 3-D mobile LiDAR point clouds. This paper presents the development and implementation aspects of an automated object extraction strategy for rapid and accurate road marking inventory. The proposed road marking extraction method is based on 2-D georeferenced feature (GRF) images, which are interpolated from 3-D road surface points through a modified inverse distance weighted (IDW) interpolation. Weighted neighboring difference histogram (WNDH)-based dynamic thresholding and multiscale tensor voting (MSTV) are proposed to segment and extract road markings from the noisy corrupted GRF images. The results obtained using 3-D point clouds acquired by a RIEGL VMX-450 mobile LiDAR system in a subtropical urban environment are encouraging.

This paper proposes a novel, automated algorithm for rapidly extracting urban road facilities, including street light poles, traffic signposts, and bus stations, for transportation-related applications. A detailed description and implementation of the proposed algorithm is provided using mobile laser scanning data collected by a state-of-the-art RIEGL VMX-450 system. First, to reduce the quantity of data to be handled, a fast voxel-based upward growing method is developed to remove ground points. Then, off-ground points are clustered and segmented into individual objects via Euclidean distance clustering and voxel-based normalized cut segmentation, respectively. Finally, a 3-D object matching framework, benefiting from a locally affine-invariant geometric constraint, is developed to achieve the extraction of 3-D objects. Quantitative evaluations show that the proposed algorithm attains an average completeness, correctness, quality, and F1-measure of 0.949, 0.971, 0.922, and 0.960, respectively, in extracting 3-D light poles, traffic signposts, and bus stations. Comparative studies demonstrate the efficiency and feasibility of the proposed algorithm for automated and rapid extraction of urban road facilities.

In this paper, a rapid oil spill detection approach which still maintains high detection accuracy is presented. The major contribution of the approach is using a superpixel segmentation method to subdivide the target SAR image into many approximate uniform scale pieces and preserves the boundaries well. Furthermore, a novel approach combine space distance, intensity deviation and size information together (SIS) is presented to eliminate the potential false positive, which is convenient and effective meanwhile. The proposed approach performs well and fast in both the synthetic data and RAD ARS AT-1 ScanSAR data which contain verified oil spills. The processing time is about 6s for a 512×512 image.

This paper presents a stepwise algorithm for extracting power-lines from mobile laser scanning (MLS) data. This algorithm first extracts non-road points from MLS data by estimating road ranges with regard to scanning mechanism and applying elevation-difference and slope criteria to the road ranges scan-line by scan-line. Then, three filters, in terms of height, spatial density, and size-and-shape, are proposed to extract power-line points in the identified non-road points, followed by Hough transform and Euclidean distance clustering. Finally, a 3D power line is modelled as a horizontal line in X-Y plane and a vertical catenary curve defined by a hyperbolic cosine function in X-Z plane. The proposed algorithm has been tested on a sample of point clouds acquired by a RIEGL VMX-450 MLS system. The results demonstrate the applicability of the proposed algorithm in extracting power transmission lines.

This paper presents a novel method for estimating earthwork volumes in asphalt pavement reconstruction using a mobile laser scanning (MLS) system. First, based on the static targets, this method registers two point cloud datasets into the same coordinate system, which respectively are acquired in the reconstructing road before and after asphalting. Next, road surface points are detected from each point cloud using a curb-based method, and further divided into a set of blocks. Afterwards, the blocks are perpendicularly partitioned into grids, where two surface features are extracted using the RANSAC. Finally, the volume of each grid is calculated according to these two surface features. The proposed algorithm has been tested on two sets of point clouds acquired by a RIEGL VMX-450 MLS system in the reconstructing road before and after asphalting. The results demonstrate the accuracy and efficiency of the proposed algorithm in estimating earthwork volumes.

This paper presents a novel algorithm for extracting 3D crack skeletons from 3D point clouds acquired by a mobile Light Detection and Ranging (LiDAR) system. This algorithm uses intensity information of cloud clouds to identify pavement cracks that usually exhibit lower intensities compared to their surroundings. First, crack candidates are extracted by applying the Otsu thresholding algorithm. Then, a spatial density filter is used to remove outliers. Next, crack points are grouped into crack-lines using a Euclidean distance clustering method. Finally, crack skeletons are extracted based on an L1-medial skeleton extraction method. The proposed algorithm has been tested on a set of mobile LiDAR point clouds acquired by a state-of-the-art RIEGL VMX-450 mobile LiDAR system. The results demonstrate the efficiency and reliability of the proposed algorithm in extracting 3D crack skeletons.

Sparse representation has been used to fuse high-resolution panchromatic (HRP) and low-resolution multispectral (LRM) images. However, the approach faces the difficulty that the dictionary is generated from the high-resolution multispectral (HRM) images, which are unknown. In this letter, a two-step method is proposed to train the dictionary from the HRP and LRM images. In the first step, coarse HRM images are obtained by additive wavelet fusion method. The initial dictionary is composed of randomly sampled patches from the coarse HRM images. In the second step, a linear constraint K-SVD method is designed to train the dictionary to improve its representation ability. Experimental results using QuickBird and IKONOS data indicate that the trained dictionary yields comparable fusion products with raw-patch-dictionary sampled from HRM images.

The classification of hyperspectral imagery is a challenging problem because few labeled pixels are available. In this letter, we propose a new semisupervised learning algorithm to combine both cluster and manifold assumptions to increase classification reliability and accuracy. The new method uses a concave-convex procedure and sequential minimization optimization technologies for transductive multiple-kernel learning (TMKL). Then, a one-against-all strategy is adopted to generalize the binary TMKL classifiers to solve the multiclass problem of remote sensing images. Experimental results on two real data sets indicate that the proposed method exhibits both high accuracy and good computational performance.

This letter presents a novel rotation-invariant method for object detection from terrestrial 3-D laser scanning point clouds acquired in complex urban environments. We utilize the Implicit Shape Model to describe object categories, and extend the Hough Forest framework for object detection in 3-D point clouds. A 3-D local patch is described by structure and reflectance features and then mapped to the probabilistic vote about the possible location of the object center. Objects are detected at the peak points in the 3-D Hough voting space. To deal with the arbitrary azimuths of objects in real world, circular voting strategy is introduced by rotating the offset vector. To deal with the interference of adjacent objects, distance weighted voting is proposed. Large-scale real-world point cloud data collected by terrestrial mobile laser scanning systems are used to evaluate the performance. Experimental results demonstrate that the proposed method outperforms the state-of-the-art 3-D object detection methods.

Three-dimensional mobile mapping in indoor environment, mostly global navigation satellite system-denied space, is to consecutively align the frames to build a global 3-D map of an indoor environment. One of the major difficulties of the current solutions is the failure at the insufficient overlapping between the frames, which is the reality of a lack of correspondences between the frames. To overcome this problem, a 3-D indoor mobile mapping system that integrates a 2-D laser scanner, and an RGB-Depth camera is presented in this letter. In this system, a fusion-iterative closest point (ICP) method, which combines the 2-D mobile platform pose from a Rao-Blackwellized particle filter estimation, an ICP, and a generalized-ICP method, is proposed for the consecutive frame alignment. Fusion-ICP achieves effective frame alignment, particularly in solving the insufficient overlapping frame alignment problem. Comparative experiments were conducted to evaluate the mapping system. The experimental results demonstrate the effectiveness and efficiency of our system for 3-D indoor mobile mapping.

A novel object detection algorithm is developed for automatically detecting road manhole and sewer well covers from mobile light detection and ranging point clouds. This algorithm takes advantage of a marked point process of disks and rectangles to model the locations of manhole and sewer well covers and their geometric dimensions. A reversible jump Markov chain Monte Carlo algorithm is implemented for simulating the posterior distribution obtained using a Bayesian paradigm. The detection results obtained from the road surface point clouds acquired by a RIEGL VMX-450 system show that the manhole and sewer well covers can be detected automatically and accurately. The performance achieved using the proposed algorithm is much more accurate and effective than those of the other three existing algorithms.

A novel pairwise 3-D shape context for partial object matching and retrieval is developed for extracting 3-D light poles and trees from mobile laser scanning (MLS) point clouds in a typical urban street scene. Unlike the single-point shape context describing only the local topology of a shape, the pairwise 3-D shape context can simultaneously model the local and global geometric structures of a shape in manifold space. By using histogram descriptors, the pairwise 3-D shape context has such characteristics as invariance to scale, invariance to orientation, and partial insensitivity to topological changes. Our results show that 3-D light poles and individual trees can be extracted from the RIEGL VMX-450 MLS point clouds and the performance achieved using our algorithm is much more accurate and effective than those of the other two existing algorithms.

With the introduction of manifold assumption, Laplacian Support Vector Machine (LapSVM) has advantages over the traditional SVM classifiers. However the dual solution of LapSVM is still a major barrier on the further application of LapSVM. Primal optimization is a promising solution to this problem. In this paper, we introduce a novel primal Laplacian Support Vector Machine with Precondition Conjugate Gradient method (PCG) to the problem of hyperspectral images classification which is one type of primal optimization solution. To prove the effectiveness of the proposed method, we apply it into the hyperspectral image data set Indian Pine. The experiment results show higher accuracy and better generalization ability than dual strategy.

This letter presents a novel semisupervised method for addressing a domain adaptation problem in the classification of hyperspectral data. To overcome the influence of distribution bias between the source and target domains, we introduce the domain transfer multiple-kernel learning to simultaneously minimize the maximum mean discrepancy criterion and the structural risk functional of support vector machines. Then, the pairwise binary classifiers are merged as the multiclass classifier for solving the classification problem in hyperspectral data. Both bias and nonbias sampling strategies are introduced to evaluate the robustness of the proposed method against the spectral distribution bias. The results obtained from real data sets show that the proposed method can achieve higher classification accuracy even with cross-domain distribution bias and provide robust solutions with different labeled and unlabeled data sizes.