Accurate hyperspectral image (HSI) interpretation is critical for providing valuable insights into various earth observation-related applications such as urban planning, precision agriculture, and environmental monitoring. However, existing HSI processing methods are predominantly task-specific and scene-dependent, which severely limits their ability to transfer knowledge across tasks and scenes, thereby reducing the practicality in real-world applications. To address these challenges, we present HyperSIGMA, a vision transformer-based foundation model that unifies HSI interpretation across tasks and scenes, scalable to over one billion parameters. To overcome the spectral and spatial redundancy inherent in HSIs, we introduce a novel sparse sampling attention (SSA) mechanism, which effectively promotes the learning of diverse contextual features and serves as the basic block of HyperSIGMA. HyperSIGMA integrates spatial and spectral features using a specially designed spectral enhancement module. In addition, we construct a large-scale hyperspectral dataset, HyperGlobal-450K, for pre-training, which contains about 450 K hyperspectral images, significantly surpassing existing datasets in scale. Extensive experiments on various high-level and low-level HSI tasks demonstrate HyperSIGMA's versatility and superior representational capability compared to current state-of-the-art methods. Moreover, HyperSIGMA shows significant advantages in scalability, robustness, cross-modal transferring capability, real-world applicability, and computational efficiency.

Recent studies demonstrate the effectiveness of deep learning (DL) methods in crop mapping using Satellite Image Time Series (SITS). Despite this progress, the existing crop mapping methods often neglect prior knowledge, leading to unsatisfactory performance, especially in scenarios with limited data. To address this limitation, this paper introduces the Knowledge-Guided Crop Mapping Network (KGCMNet) as an innovative solution for crop mapping. Beyond a spatiotemporal encoder and a crop type prediction decoder, KGCMNet incorporates a knowledge reconstruction (KR) module for introducing essential crop-related prior knowlege. The KR module, utilizing NDVI as the chosen knowledge, aids the encoder in learning discriminative phenology from the SITS data. Experimental results demonstrate the outperformance of KGCMNet compared to other methods, showcasing the effectiveness of KR module in improving the accuracy across various crop types.

Weakly supervised temporal action localization (WTAL) aims to detect action instances from untrimmed videos. It may cause two problems, namely action incompleteness and background disturbance, due to only video-level class labels given. In this paper, we propose a context driven network with Bayes to alleviate the two problems, in which an attention module is used to predict coarse probability for each snippet, and then a Bayesian refinement module is designed to refine the coarse results by capturing the relationship between context snippets. Finally, the coarse and refined probabilities are combined as the inputs of the classifier for training. Quantitative and qualitative studies on two benchmark datasets, i.e., THUMOS’14 and ActivityNet 1.2, demonstrate that the proposed approach exceeds state-of-the-art methods.

Soft-attention mechanism has attracted a lot of attention in recent years due to its ability to capture the most discriminative image features for understanding actions. However, soft-attention tends to focus on fine-grained parts on images and ignores global information, which can lead to totally wrong classification results. To address this issue, we propose a novel deep selective feature learning network (DSFNet), which can automatically learn the feature maps with both fine-grained and global information. Specially, DSFNet is designed to have the ability to learn to adjust the actions for feature map selection by maximizing the cumulative discounted rewards. Moreover, the DSFNet is an easy-to-use extension of state-of-the-art base architectures of multiple tasks. Extensive experiments show that the proposed method has achieved superior performance on two standard action recognition benchmarks across still images (PPMI) and videos (HMDB51).

Weakly supervised temporal action localization (WSTAL) is a practical but challenging issue in video understanding. However, most existing methods have to activate background snippets or deactivate action snippets in cases of no boundary annotations, which inevitably affects the localization of action instances. In this letter, we propose a spatial enhancement and temporal constraint (SETC) model to address this problem from three aspects. Specifically, we first propose a spatial enhancement module to enhance the discrimination of the extracted features. Then we leverage the instance sparse constraint to restrain the drastic fluctuation class activation sequence (CAS). Finally, we use the confidence connectivity enhancement to connect the snippets that are broken up by mistake. Experiments on THUMOS'14 and ActivityNet datasets validate the efficacy of SETC against existing state-of-the-art WSTAL algorithms.

Recent approaches in RGB-based and depth-based human action recognition achieved outstanding performance respectively, which demonstrate the effectiveness of RGB and depth modalities for action classification, however it is infrequent to consider them both. Currently, available multimodal-based methods of action recognition suffer from some limitations, including non-end-to-end training, violent fusion and inefficiency. In this paper, we propose a novel joint deep learning (JDL) model which is capable of: 1) jointly optimizing the object of classification and feature extraction through a novel end-to-end two-stream deep learning model, 2) refining common-specific features via introducing the constraint of similarity loss in high-level, and 3) using 2D convolution kernel instead of 3D convolution kernel during feature extraction for gaining the efficiency. The experiments on two challenging datasets show the promising performance of our architecture.