Multimodal 3D object detection for autonomous driving, a task for real-world applications, poses substantial challenges in maintaining robust performance under various perturbations and complex environmental conditions. However, most existing approaches primarily focus on performance optimization under relatively ideal scenarios or focus on one or few disturbing conditions (or adverse conditions), lacking systematic exploration of robustness against real-world factors, including high class imbalance, adverse weather conditions, sensor jitter and failures, and significant scene variations. To address this issue, we propose a robust multimodal 3D detector, termed RobusTor3D, which integrates robustness at both the structural and supervisory levels by blending the knowledge from Vision-Language Models (VLMs). Structurally, textual descriptions are incorporated to enhance the semantic richness and diversity of rare classes. This novel semantic injection operation compensates for the inherent class imbalance and modality weakness in conventional visual features. Furthermore, semantic alignment capability and robust representation by Vision-Language Knowledge Extraction (V-LKE) serve as semantic priors to complement modality-specific representations, significantly improving model adaptability. At the supervisory level, we propose a Scene-level Multimodal Consistency Learning (SMCL) strategy, which jointly enforces global semantic constraints across modalities, encouraging the learning of stable and abundant semantic representations. This special design specifically reduces the impact of spatial alignment, while notably enabling semantic compensation under modality-loss conditions. Extensive robustness experiments conducted on KITTI, KITTI-C, and CADC benchmarks evaluate five robustness aspects, including long-tail problem, adverse weather (rain, snow, fog, strong sunlight), sensor spatial misalignment and motion blur, modality loss, and cross-domain scenarios. The results show that RobusTor3D demonstrates superior robustness across all five evaluated aspects. It consistently outperforms the state-of-the-art methods under various challenging conditions.

Unsupervised domain adaptation (UDA) is critical for cross-modal 3D semantic segmentation in real scenarios, as it significantly reduces the annotation costs in new domains. Existing methods primarily focus on various alignment across domains or modalities, however, domain shift and modal differences are inbuilt, and learning under the guidance of source semantic knowledge instead of forced alignment may be more favorable for UDA tasks. Motivated by this insight, we propose cross Domain Label-Guided learning (xDLG) for cross-modal UDA 3D semantic segmentation, which utilizes multimodal pseudo-labels generated online as a bridge for cross-domain label knowledge transfer. Specifically, we propose two types of label-guided learning: Label-Guided Semantic Prototype Learning (LG-SPL) and Label-Guided Pseudo-Label Learning (LG-PLL). First, in LG-SPL, we utilize massive class instances from the source domain to correct the semantic prototypes in the target domain, embedding genuine semantic similarity to tackle the issue of class confusion. Next in LG-PLL, we conceptualize the refinement of pseudo-labels as a reverse diffusion process, leveraging label information from the source domain as a guiding signal to adjust the semantic distribution of pseudo-labels on a global scale. Furthermore, we incorporate domain-specific cues from target 2D images as supplementary conditions throughout the denoising stages, ensuring semantic consistency at finer details. We evaluate our method on 3D semantic segmentation tasks. The results demonstrate that the proposed method achieves state-of-the-art performance across multiple cross-modal UDA scenarios.

Abstract Hyperspectral images (HSIs) contain hundreds of continuous spectral bands and are rich in spectral‐spatial information. In terms of HSIs’ classification, traditional convolutional neural networks (CNNs) extract features based on HSI's spectral‐spatial information through 2D convolution. However, 2D convolution extracts features in 2D plane without considering the relationships between spectral bands, which inevitably leads to insufficient feature extraction. 3D convolutional neural networks (3DCNNs) take account of the correlations among spectral bands and outperform 2D convolutional networks in feature extraction, but the computational cost is rather expensive. To address the above problem, a light‐weight three‐layer 3D convolutional network Module (3D‐M) for HSIs’ spectral‐spatial feature extraction is proposed. Another challenge is that neither 2D convolution nor 3D convolution utilizes the structural information inherent in the data. Graph convolution networks (GCNs) can model and utilize such information through the similarity matrix, also known as adjacency matrix. However, traditional GCNs cannot handle large‐scale data because they construct adjacency matrix on all data, which results in high computational complexity and large storage requirement. To conquer this challenge, this article proposes a batch‐graph strategy on which a scalable GCN is developed. Finally, a hybrid network model (HNM) based on the proposed light‐weight 3D‐M and scalable GCN is presented. HNM extracts spectral‐spatial features of HSIs with low computational complexity through the light‐weight 3D convolution network and leverages the structural information in data via the scalable GCN. The experimental results on three public datasets with different sizes demonstrate that the proposed HNM produces better classification results than other state‐of‐the‐art hyperspectral images classification models in terms of overall accuracy (OA), average accuracy (AA) and kappa coefficient (Kappa).

How to acquire effective information from multispectral remote sensing images is a challenging task in semantic segmentation of remote sensing images. In this paper, an end-to-end semantic segmentation network (BLASeNet) is proposed. The model adopts an encoder-decoder structure. In the encoder phase, to exploit the band correlation of multispectral remote sensing images, we propose an effective 3D Residual block to encode the spectral-spatial features of images. In order to extract more discriminative features from multispectral images, a band-location adaptive selection mechanism is proposed to adaptively learn the weights of different bands and different spatial locations within a single band, enhancing the expression of features. In the decoder phase, we introduce two trainable parameter matrices$\mathrm{W}_{\alpha}$and Wβ in the skip connections, adaptively adjusting the fusion ratio of low-level detail features and high-level semantic features by network learning, improving the image segmentation accuracy. In addition, we extend the channel attention to 3D data, further refining the fused feature maps. Experimental results on ISPRS Potsdam and Qinghai datasets demonstrate the effectiveness of BLASeNet.