Denoising Diffusion Probabilistic Models (DDPMs) have shown success in robust 3D object detection tasks. Existing methods often rely on the score matching from 3D boxes or pre-trained diffusion priors. However, they typically require multi-step iterations in inference, which limits efficiency. To address this, we propose a Robust single-stage fully Sparse 3D object Detection Network with a Detachable Latent Framework (DLF) of DDPMs, named RSDNet. Specifically, RSDNet learns the denoising process in latent feature spaces through lightweight denoising networks like multi-level denoising autoencoders (DAEs). This enables RSDNet to effectively understand scene distributions under multi-level perturbations, achieving robust and reliable detection. Meanwhile, we reformulate the noising and denoising mechanisms of DDPMs, enabling DLF to construct multi-type and multi-level noise samples and targets, enhancing RSDNet robustness to multiple perturbations. Furthermore, a semantic-geometric conditional guidance is introduced to perceive the object boundaries and shapes, alleviating the center feature missing problem in sparse representations, enabling RSDNet to perform in a fully sparse detection pipeline. Moreover, the detachable denoising network design of DLF enables RSDNet to perform single-step detection in inference, further enhancing detection efficiency. Extensive experiments on public benchmarks show that RSDNet can outperform existing methods, achieving state-of-the-art detection.

Existing conditional Denoising Diffusion Probabilistic Models (DDPMs) with a Noise-Conditional Framework (NCF) remain challenging for 3D scene understanding tasks, as the complex geometric details in scenes increase the difficulty of fitting the gradients of the data distribution (the scores) from semantic labels. This also results in longer training and inference time for DDPMs compared to non-DDPMs. From a different perspective, we delve deeply into the model paradigm dominated by the Conditional Network. In this paper, we propose an end-to-end robust semantic Segmentation Network based on a Conditional-Noise Framework (CNF) of DDPMs, named CDSegNet. Specifically, CDSegNet models the Noise Network (NN) as a learnable noise-feature generator. This enables the Conditional Network (CN) to understand 3D scene semantics under multi-level feature perturbations, enhancing the generalization in unseen scenes. Meanwhile, benefiting from the noise system of DDPMs, CDSegNet exhibits strong robustness for data noise and sparsity in experiments. Moreover, thanks to CNF, CDSegNet can generate the semantic labels in a single-step inference like non-DDPMs, due to avoiding directly fitting the scores from semantic labels in the dominant network of CDSegNet. On public indoor and outdoor benchmarks, CDSegNet significantly outperforms existing methods, achieving state-of-the-art performance.

Recent advancements indicate that the novel Mamba framework, with its linear reasoning capabilities and performance on par with the Transformer framework, has become a popular alternative to Transformers. However, there is still a lack of exploration in remote sensing image captioning (RSIC). One important challenge is that the 1-D selective state-space model (SSM) is not suitable for 2-D remote sensing images (RSIs). To mitigate this challenge, this article proposes an SSM with genetic operations for RSIC (RSIC-GMamba) that integrates the search capabilities of heuristic genetic algorithm and the global modeling capabilities of Transformer into the Mamba framework. To comprehensively capture the multiscale global context of RSIs, we design a genetic SSM by incorporating a dilated convolution, genetic operations (crossover and mutation), and self-attention mechanism into the selective SSM. Specifically, dilated convolutions are employed to extract multiscale visual features through varying dilation rates. Crossover operation expands the spatial arrangement of image regions to thoroughly capture contextual information and mutation operation introduces randomness to enhance the model’s robustness. The self-attention mechanism is integrated to model relationships among SSM hidden states, thereby enhancing visual context. In addition, to fully utilize high- and low-level visual-semantic information, we propose a vision and scene text aggregation (ViSTA) module based on gating mechanisms. Experimental results on four RSIC datasets demonstrate the effectiveness of the proposed RSIC-GMamba. The code will be publicly available athttps://github.com/One-paper-luck/RSIC-GMamba.

Recent progress has shown that integrating multiscale visual features with advanced Transformer architectures is a promising approach for remote sensing image captioning (RSIC). However, the lack of local modeling ability in self-attention may potentially lead to inaccurate contextual information. Moreover, the scarcity of trainable image-caption pairs poses challenges in effectively harnessing the semantic alignment between images and texts. To mitigate these issues, we propose a Multiscale Grouping Transformer with Contrastive Language-Image Pre-training (CLIP) latents (MG-Transformer) for RSIC. First of all, a CLIP image embedding and a set of region features are extracted within a Multi-level Feature Extraction module. To achieve a comprehensive image representation, a Semantic Correlation module is designed to integrate the image embedding and region features with an attention gate. Subsequently, the integrated image features are fed into a Transformer model. The Transformer encoder utilizes dilated convolutions with different dilation rates to obtain multiscale visual features. To enhance the local modeling ability of the self-attention mechanism in the encoder, we introduce a Global Grouping Attention mechanism. This mechanism incorporates a grouping operation into self-attention, allowing each attention head to focus on different contextual information. The Transformer decoder then adopts the Meshed Cross-Attention mechanism to establish relationships between various scales of visual features and text features. This facilitates the generation of captions for images by the decoder. Experimental results on three RSIC datasets demonstrate the superiority of the proposed MG-Transformer. The code will be publicly available at https://github.com/One-paper-luck/MG-Transformer.

Remote sensing image captioning aims to generate meaningful and grammatically accurate sentences for remote sensing images. However, in comparison to natural image captioning, remote sensing image captioning encounters additional challenges due to the unique characteristics of remote sensing images. The first challenge arises from the abundance of objects present in these images. As the number of objects increases, it becomes increasingly difficult to determine the main focus of the description. Moreover, the objects in remote sensing images often share similar appearances, which further complicates the generation of accurate descriptions. To overcome these challenges, we propose a Prior Knowledge-guided Transformer for remote sensing image captioning. Firstly, scene-level and object-level features are extracted in a Multi-level Feature Extraction module. To further refine and enhance the extracted multi-level features, we introduce a Feature Enhancement module. This module utilizes a combination of graph neural networks and attention mechanisms to capture the correlation and difference between different objects or scene regions. Moreover, we propose a Prior Knowledge augmented Attention mechanism to select the objects that are more relevant to the scene regions by establishing the relationships between them. This attention mechanism is seamlessly integrated into the Transformer structure, providing valuable prior knowledge that promotes the caption generation process. Extensive experiments on three remote sensing image captioning datasets verify the superiority of the proposed method. Compared with the baseline methods, the proposed method achieves more impressive performance. The code will be publicly available at https://github.com/One-paper-luck/PKG-Transformer.