The misalignment between RGB and thermal images significantly impairs RGB-Thermal semantic segmentation accuracy. Current non-end-to-end methods treat RGB-Thermal registration independently of semantic segmentation, resulting in fusion errors, redundant computations, and poor real-time performance. Semantic segmentation accuracy directly correlates with registration precision: better registration yields more accurate segmentation. Moreover, regions with identical semantic labels, indicating the same object, tend to share similar registration offsets. Based on these correlations, we propose an end-to-end multimodal registration and segmentation method using flexible deformation fields. Our method utilizes a shared encoder for registration and semantic segmentation to reduce redundancy. Unlike traditional non-end-to-end approaches, it directly registers high-level perceptual features, thereby optimizing computational efficiency and real-time performance. Additionally, we employ a flexible deformation field to register RGB-Thermal data, addressing limitations of traditional affine transformations in handling non-coplanar and non-rigid registrations. However, the increased flexibility of deformation fields compared to affine transformations, and the sacrificing of geometric feature preservation, pose training challenges. To overcome this, we introduce a semantic alignment loss function to train the alignment module. This function calculates the semantic segmentation loss between the predictions from registered thermal features and RGB semantic labels. It shortens the gradient backpropagation path, aligning the objectives of registration and segmentation. We validate our end-to-end approach through extensive experiments, achieving significant performance enhancements. On the IR SEG dataset, our end-to-end method achieves state-of-the-art results with a mean Intersection over Union (mIoU) of 61.1% and a mean accuracy (mAcc) of 76.0%.

Masked video modeling has shown remarkable performance in downstream tasks by predicting masked video tokens from visible ones. However, training models from scratch on large-scale unlabeled data remains computationally challenging and time-consuming. Moreover, the commonly used random-based sampling techniques may lead to the selection of redundant or low-information regions, hindering the model from learning discriminative representations within the limited training epochs. To achieve efficient pre-training, we propose MaskAgain, an efficient feature-based knowledge distillation framework for masked video pre-training that facilitates knowledge transfer from a pre-trained teacher model to a student model. In contrast to previous approaches that align all visible token features with the teacher model at output layers, MaskAgain adopts a selective approach by masking visible tokens again at both the hidden and output layers of the transformer block. Attention mechanisms are utilized for informative feature selection. At the hidden level, attention maps generated by the transformer's multi-head attention structure are utilized to select crucial token information at both temporally-global and temporally-local levels. Additionally, at the output level, an activation-based attention map is generated using token features, enabling us to focus on important tokens while preserving feature similarity and the relationship matrix similarity between patches. Extensive experimental results show that MaskAgain achieves comparable or even better performance than existing methods on benchmark datasets with much fewer training epochs and much less memory, which demonstrates that MaskAgain allows for efficient pre-training of accurate video models, reducing computational resources and training time significantly. Code is released at https://github.com/xiaojieli0903/MaskAgain.

Self-supervised learning methods have shown significant promise in acquiring robust spatiotemporal representations from unlabeled videos. In this work, we address three critical limitations in existing self-supervised video representation learning: 1) insufficient utilization of contextual information and lifelong memory, 2) lack of fine-grained visual concept alignment, and 3) neglect of the feature distribution gap between encoders. To overcome these limitations, we propose a novel memory-enhanced predictor that leverages key-value memory networks with separate memories for the online and target encoders. This design enables the effective storage and retrieval of contextual knowledge, facilitating informed predictions and enhancing overall performance. Additionally, we introduce a visual concept alignment module that ensures fine-grained alignment of shared semantic information across segments of the same video. By employing coupled dictionary learning, we effectively decouple visual concepts, enriching the semantic representation stored in the memory networks. Our proposed approach is extensively evaluated on widely recognized benchmarks for action recognition and retrieval tasks, demonstrating its superiority in learning generalized video representations with significantly improved performance compared to existing state-of-the-art self-supervised learning methods. Code is released at https://github.com/xiaojieli0903/FGKVMemPred_video.

Semantic segmentation using RGB and thermal images is crucial in a variety of applications, including autonomous driving and video surveillance. However, the validity of information differs between modalities, which is typically addressed by weighting image features using complex and inefficient networks. To address this issue, we propose a Multi-Expert Fusion Network (MEFNet) that decouples the three-dimensional attention matrix of image features into a two-dimensional modal weight matrix and a channel attention vector. This approach focuses more on modal and channel differences while excluding interferences from other factors. Specifically, MEFNet multiplies RGB and thermal features by their respective modal weights, and then uses channel attention to select important feature channels. Comprehensive experiments demonstrate that MEFNet is competitive with state-of-the-art methods, achieving 62.6% mIoU on the IR SEG dataset.