Visible light and thermal infrared tracking combines the characteristics of visible light and thermal infrared modalities to achieve robust target tracking in all-weather and all-day scenarios. However, most existing visible light and thermal infrared tracking methods rely on either full fine-tuning or attention mechanisms, which introduce a large number of parameters and are predominantly influenced by the visible modality. This results in challenges such as high computational complexity, slower processing speeds, and limited exploitation of multimodal information. To address these issues, this paper proposes a lightweight multimodal tracking model based on feature fusion and enhancement. The model consists of a feature fusion adapter and a joint enhancement adapter, designed to integrate and refine information across modalities. It employs a dual-stream transformer encoder with shared parameters across modality branches, utilizing a frozen pre-trained foundation model to independently extract features from visible light and thermal infrared inputs. The lightweight fusion adapter combines modality-specific information, while the joint enhancement adapter refines unimodal features, introducing only 0.23M trainable parameters. Experimental results on the LasHeR benchmark demonstrate that the proposed method outperforms prompt learning and other adapter-based methods, achieving a 4.4% improvement in PR and a 3.3% increase in SR while maintaining computational efficiency. With a real-time inference speed of 28.60 FPS, the proposed method balances accuracy and efficiency effectively. The source code will be available at https://github.com/huxue/MFJA.

Video-based person re-identification (Re-ID) aims at associating the video sequences of the identical person across multiple cameras. The ubiquitous appearance misalignment poses a major obstacle for video person Re-ID. Existing alignment-based methods generally rely on off-the-shelf semantic parsing models to locate visible human parts, which ignore identifiable personal belongings and cannot handle various interferences (e.g., pedestrian detection errors and occlusions) in video clips. In this work, we propose a novel framework termed Context-Aided Semantic-Aware Self-Alignment (CSSA) for video-based person Re-ID. First, we propose to jointly learn pixel-level part-aligned representations and semantic-aligned global-level representations in an end-to-end manner. Unlike most existing approaches that depend on prior information in terms of pose for part estimation, CSSA can locate different body parts and achieve the pixel-level semantic alignment without extra human topology semantics. Second, a Context-Aided Region Enhancement (CARE) module is proposed to efficiently highlight macro-visual patterns associated with the target pedestrian and suppress noise caused by factors like background clutters and occlusions. Third, we propose a Semantic-Aware Global Feature Alignment (SGFA) method for generating pair-wise semantic-aligned global representations, which play an essential role in both the training and inference phases. Extensive experimental results on multiple challenging benchmarks indicate the superiority and effectiveness of the proposed CSSA.

Video-based person re-identification (Re-ID) aims at matching the video snippets of the same person across multiple cameras. The ubiquitous appearance misalignment is a critical challenge in video person re-identification. Existing alignment-based methods rely on off-the-shelf human parsing models and cannot handle anomalous appearance information (e.g., obstacles and pedestrian interference) in video sequences. In this paper, we propose Anomaly-Aware Semantic Self-Alignment (ASSA), a novel video-based person Re-ID framework that seeks out body parts without prior human topology information and learns part-based feature representations against anomalous information. The proposed ASSA performs part classifier training and part-aligned representation learning alternately. For the classifier training, we design a Salient Region Extraction module to segment the entire foreground from the background in each input frame. Furthermore, a novel Anomaly-Aware Refinement module is proposed to suppress the influence of anomalous interference. Extensive experiments on three prevalent benchmarks demonstrate the effectiveness and superiority of the proposed framework.

Video-based person re-identification (Re-ID) aims at retrieving the video clips of the same person across multiple cameras. Since video clips are captured at various spatial resolutions (scales), learning multi-scale person appearance features while constructing the cross-scale information interaction is pivotal for video-based person Re-ID. In this paper, we propose an efficient framework, Multi-Scale Aligned Spatial-Temporal Interaction (MS-STI), which not only exchanges the spatial-temporal information within a scale, but also mines implicit related complementary knowledge across scales. MS-STI presents a hierarchical multi-branch architecture that designs the branches with fewer convolutional layers for lower spatial resolution inputs. In this way, the framework enables inter-scale feature size matching for exchanging information across multiple scale-specific branches. We share the parameters of branched sub-networks to optimize the efficiency of person feature extraction. Furthermore, we propose two modules, Spatial Interaction (SI) and Multi-Scale Temporal Interaction (MSTI), which can realize spatial-temporal interaction across multiple branches. SI performs point-wise spatial information transfer within a frame. While MSTI focuses on inter-frame and inter-scale information interaction. Extensive experiments on three challenging benchmarks demonstrate the effectiveness and superiority of the proposed MS-STI.

The accuracy of the visual tasks for top-view fisheye cameras is limited by the Euclidean geometry for pose-distorted objects in images. In this paper, we demonstrate the analogy between the fisheye model and the Poincaré ball and that learning the shape of convolution kernels in the Poincaré Ball can alleviate the spatial distortion problem. In particular, we propose the Deformable Convolution based on the Poincaré Ball, named DCPB, which conducts the Graph Convolutional Network (GCN) in the Poincaré ball and calculates the geodesic distances to Poincaré hyperplanes as the offsets and modulation scalars of the modulated deformable convolution. Besides, we explore an appropriate network structure in the baseline with the DCPB. The DCPB markedly improves the neural network’s performance. Experimental results on the public dataset THEODORE show that DCPB obtains a higher accuracy, and its efficiency demonstrates the potential for using temporal information in fisheye videos.

Oil is of great significance in the key components like the hydraulic damper of high-speed trains and its leaks suggest that some key components may malfunction, bringing potential danger to the safety of the train. It is indispensable to diagnose oil leakage timely and faults can be discovered by detecting oil stains to avoid possible accidents caused by the breakdown. But it is challenging to discover the oil stains due to their irregularity of shape and diversity of size in complex environments. To deal with these intractable problems, we propose the Multifaceted Feature Aggregation Network (MFANet) which is composed of Multifaceted Refinement Feature (MRF) module, Cross Layer Feature Attention (CLFA) module, and Cross Layer Feature Enhancement (CLFE) module. The MRF solves the limitation of the single convolution method’s inability to catch changeable oil stains better and captures more expressive features. The CLFA and the CLFE obtain richer information and compensate for the shortcomings of insufficient information on single-layer features. Furthermore, we propose a loss function to guide the learning of our network in the case of the unbalanced dataset, which surpasses the useful Binary Cross-entropy loss. We conduct experiments based on four widely used backbone networks and achieve cutting-edge performance with fewer FLOPs and Parameters compared with many advanced methods. In addition, we perform our MFANet on several challenging saliency detection datasets, the rail surface defects detection dataset, and the COCO-Stuff segmentation dataset, which outperforms some progressive approaches and demonstrates powerful generalization capability and adaptation.