The demonstrated success of sparsely-gated Mixture-of-Experts (MoE) architectures, exemplified by models such as DeepSeek and Grok, has motivated researchers to investigate their adaptation to diverse domains. In real-world image super-resolution (Real-ISR), existing approaches mainly rely on fine-tuning pre-trained diffusion models through Low-Rank Adaptation (LoRA) module to reconstruct high-resolution (HR) images. However, these dense Real-ISR models are limited in their ability to adaptively capture the heterogeneous characteristics of complex real-world degraded samples or enable knowledge sharing between inputs under equivalent computational budgets. To address this, we investigate the integration of sparse MoE into Real-ISR and propose a Mixture-of-Ranks (MoR) architecture for single-step image super-resolution. We introduce a fine-grained expert partitioning strategy that treats each rank in LoRA as an independent expert. This design enables flexible knowledge recombination while isolating fixed-position ranks as shared experts to preserve common-sense features and minimize routing redundancy. Furthermore, we develop a degradation estimation module leveraging CLIP embeddings and predefined positive-negative text pairs to compute relative degradation scores, dynamically guiding expert activation. To better accommodate varying sample complexities, we incorporate zero-expert slots and propose a degradation-aware load-balancing loss, which dynamically adjusts the number of active experts based on degradation severity, ensuring optimal computational resource allocation. Comprehensive experiments validate our framework's effectiveness and state-of-the-art performance.

This paper focuses on jailbreaking attacks against large language models (LLMs), eliciting them to generate objectionable content in response to harmful user queries. Unlike previous LLM-jailbreak methods that directly orient to LLMs, our approach begins by constructing a multimodal large language model (MLLM) built upon the target LLM. Subsequently, we perform an efficient MLLM jailbreak and obtain a jailbreaking embedding. Finally, we convert the embedding into a textual jailbreaking suffix to carry out the jailbreak of target LLM. Compared to the direct LLM-jailbreak methods, our indirect jailbreaking approach is more efficient, as MLLMs are more vulnerable to jailbreak than pure LLMs. Additionally, to improve the attack success rate of jailbreak, we propose an image-text semantic matching scheme to identify a suitable initial input. Extensive experiments demonstrate that our approach surpasses current state-of-the-art jailbreak methods in terms of both efficiency and effectiveness. Moreover, our approach exhibits superior cross-class generalization abilities.

Video-based visible-infrared person re-identification (VVI-ReID) aims to match pedestrian sequences across modalities for all-day surveillance. While supervised methods have shown progress, their dependence on large-scale cross-modal annotations limits scalability. We investigate the task of unsupervised domain adaptation for VVI-ReID (UDA-VVI-ReID), where a model trained on a labeled source domain is adapted to an unlabeled target domain. Directly extending existing image-based unsupervised VI-ReID methods to video scenarios by simply averaging frame-level features is suboptimal, as this naive strategy neglects the rich temporal dynamics in video data and leads to unreliable pseudo-labels due to occlusion-induced noise. To overcome these limitations, we propose a Dynamic-Static Collaboration (DSC) framework that explicitly leverages the complementary strengths of motion and appearance cues. The Dynamic-Static Label Unification (DSLU) module refines pseudo-labels by validating the consistency between static and dynamic predictions. Based on these labels, the Dynamic-Static Joint Learning (DSJL) module performs neighbor-aware contrastive learning in both feature spaces, promoting robust representation learning under cross-modal and temporal variations. Experiments on HITSZ-VCM and BUPTCampus show that DSC sets a strong baseline for this new task, enabling robust cross-modal video ReID without target labels.

Decoding visual features from EEG signals is a central challenge in neuroscience, with cross-modal alignment as the dominant approach. We argue that the relationship between visual and brain modalities is fundamentally asymmetric, characterized by two critical gaps: a Fidelity Gap (stemming from EEG's inherent noise and signal degradation, vs. vision's high-fidelity features) and a Semantic Gap (arising from EEG's shallow conceptual representation, vs. vision's rich semantic depth). Previous methods often overlook this asymmetry, forcing alignment between the two modalities as if they were equal partners and thereby leading to poor generalization. To address this, we propose the adaptive teaching paradigm. This paradigm empowers the ``teacher" modality (vision) to dynamically shrink and adjust its knowledge structure under task guidance, tailoring its semantically dense features to match the ``student" modality (EEG)'s capacity. We implement this paradigm with the ShrinkAdapter, a simple yet effective module featuring a residual-free design and a bottleneck structure. Through extensive experiments, we validate the underlying rationale and effectiveness of our paradigm. Our method achieves a top-1 accuracy of 60.2% on the zero-shot brain-to-image retrieval task, surpassing previous state-of-the-art methods by a margin of 9.8%. Our work introduces a new perspective for asymmetric alignment: the teacher must shrink and adapt to bridge the vision-brain gap.

Jiawei Zhou, Chi Zhang, Xiang Feng, Qiming Zhang, Haibo Qiu, Lihuo He, Dengpan Ye, Xinbo Gao, Jing Zhang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Real-world surveillance often operates in low-light environments, where degraded visual evidence can make training-free anomaly reasoning unreliable. However, current training-free methods typically assume sufficiently clean inputs, which can lead to hallucinated semantics and unstable anomaly scores under visual degradation. To address this issue, we propose Retrieval-augmented Contextual Inference (ReCI), a training-free framework that leverages retrieval-augmented context for robust low-light video anomaly detection. ReCI constructs Semantic Context (SC) through hierarchical captioning by aggregating clip-level local captions into a video-level global description. It then performs contextualized anomaly inference using Reference Context (RC) retrieved from a reference pool built during inference. The VLM outputs anomaly scores and associated confidence values, which we use as a heuristic reliability signal for temporal refinement. Experiments on XD-Violence and UCF-Crime show that ReCI consistently improves over prior training-free baselines, with particularly clear gains on the low-light subset.

Accurate 3D hand pose estimation faces inherent challenges, including self-occlusions, joint similarities, and high degrees of freedom. Although most existing CNN-based or Transformer-based methods leverage global contexts, they often fail to capture fine-grained local details and robustly handle occluded joints. To address these limitations, we propose a novel Transformer framework for depth-based 3D hand pose estimation, which incorporates two key designs: the Bidirectional Multiscale Fusion and Learnable Anchor Guidance. Firstly, we propose bidirectional multiscale fusion that sequentially propa-gates features from different encoder levels in both top-down and bottom-up directions, followed by a final aggregation of all scale features. Such an intricate feature interaction eventually enables effective joint modeling of fine-grained local details (e.g., fingertip positions) and high-level semantic context information (e.g., palm orientation). Secondly, we introduce the learnable anchor query as prior guidance to dynamically guide the decoder to localize ambiguous joints better. The learnable anchors are derived from joint-specific attention maps under 3D ground-truth supervision and then are concatenated with static grid anchors to form hybrid anchors, which effectively enable more precise 3D hand pose estimation, especially for occlusions. To further demonstrate the practicality of our framework for IoT-oriented deployment, we conduct edge-device experiments to validate its deployment feasibility. Experiments on benchmark datasets (including NYU, ICVL, MSRA, and DexYCB) demonstrate the superiority of our method over previous state-of-the-art approaches.

This paper studies the problem of zero-shot sketch-based image retrieval (ZS-SBIR), which aims to use sketches from unseen categories as queries to match the images of the same category. Due to the large cross-modality discrepancy, ZS-SBIR is still a challenging task and mimics realistic zero-shot scenarios. The key is to leverage transferable knowledge from the pre-trained model to improve generalizability. Existing researchers often utilize the simple fine-tuning training strategy or knowledge distillation from a teacher model with fixed parameters, lacking efficient bidirectional knowledge alignment between student and teacher models simultaneously for better generalization. In this paper, we propose a novel Symmetrical Bidirectional Knowledge Alignment for zero-shot sketch-based image retrieval (SBKA). The symmetrical bidirectional knowledge alignment learning framework is designed to effectively learn mutual rich discriminative information between teacher and student models to achieve the goal of knowledge alignment. Instead of the former one-to-one cross-modality matching in the testing stage, a one-to-many cluster cross-modality matching method is proposed to leverage the inherent relationship of intra-class images to reduce the adverse effects of the existing modality gap. Experiments on several representative ZS-SBIR datasets (Sketchy Ext dataset, TU-Berlin Ext dataset and QuickDraw Ext dataset) prove the proposed algorithm can achieve superior performance compared with state-of-the-art methods. The source code is publicly available at https://github.com/zermatt-luo/SBKA.

Cloth-Changing Person Re-Identification (CC-ReID) aims to recognize individuals across camera views despite clothing variations, a crucial task for surveillance and security systems. Existing methods typically frame it as a cross-modal alignment problem but often overlook explicit modeling of interference factors such as clothing, viewpoints, and pedestrian actions. This oversight can distort their impact, compromising the extraction of robust identity features. To address these challenges, we propose a novel framework that systematically disentangles interference factors from identity features while ensuring the robustness and discriminative power of identity representations. Our approach consists of two key components. First, a dual-stream identity feature learning framework leverages a raw image stream and a cloth-isolated stream, to extract identity representations independent of clothing textures. An adaptive cloth-irrelevant contrastive objective is introduced to mitigate identity feature variations caused by clothing differences. Second, we propose a Text-Driven Conditional Generative Adversarial Interference Disentanglement Network (T-CGAIDN), to further suppress interference factors beyond clothing textures, such as finer clothing patterns, viewpoint, background, and lighting conditions. This network incorporates a multi-granularity interference recognition branch to learn interference-related features, a conditional adversarial module for bidirectional transformation between identity and interference feature spaces, and an interference decoupling objective to eliminate interference dependencies in identity learning. Extensive experiments on public benchmarks demonstrate that our method significantly outperforms state-of-the-art approaches, highlighting its effectiveness in CC-ReID.

Domain Generalization (DG) seeks to develop models that perform well on unseen target domains by learning domain-invariant representations. Recent advances in pre-trained Visual Foundation Models (VFMs), such as CLIP, have shown strong potential for enhancing DG through prompt tuning. However, existing VFM-based prompt tuning methods often focus on task-specific adaptation rather than disentangling domain-invariant features, leaving cross-domain generalization insufficiently explored. In this paper, we address this challenge by fully leveraging the controllable and flexible language prompt in VFMs. Observing that the text modality is inherently rich in semantics and easier to disentangle, we propose a novel framework termed Prompt Disentanglement via Language Guidance and Representation Alignment (PADG). PADG first employs a large language model (LLM) to disentangle textual prompts into domain-invariant and domain-specific components, which then guide the learning of domain-invariant visual representations. To complement the limitations of text-only guidance, we further introduce the Worst Explicit Representation Alignment (WERA) module, which enhances visual invariance by simulating bounded domain shifts through learnable stylization prompts and aligning representations between original and perturbed samples. Extensive experiments on mainstream DG benchmarks, including PACS, VLCS, OfficeHome, DomainNet, and TerraInc, demonstrate that PADG consistently outperforms existing state-of-the-art methods, validating its effectiveness in robust domain-invariant representation learning.

Adversarial attack is a major obstacle to the deployment of deep neural networks (DNNs) for security-sensitive applications. To address these adversarial perturbations, various adversarial defense strategies have been developed, with Adversarial Training (AT) being one of the most effective methods to protect neural networks from adversarial attacks. However, existing AT methods struggle against training-agnostic attacks due to their limited generalizability. This suggests that the AT models lack a unified perspective for various attacks to conduct universal defense. This paper sheds light on a generalizable prior under various attacks: consistent class confusion (3C), i.e., an AT classifier often confuses the predictions between correct and ambiguous classes in a highly similar pattern among diverse attacks. Relying on this latent prior as a bridge between seen and agnostic attacks, we propose a more generalized AT model by mitigating consistent class confusion (M3C) to resist training-agnostic attacks. Specifically, we optimize an Adversarial Confusion Loss (ACL), which is weighted by uncertainty, to distinguish the most confused classes and encourage the AT model to focus on these confused samples. To suppress malignant features affecting correct predictions and producing significant class confusion, we propose a Gradient-Aware Attention (GAA) mechanism to enhance the classification confidence of correct classes and eliminate class confusion. Experiments on multiple benchmarks and network frameworks demonstrate that our M3C model significantly improves the generalization of AT robustness against agnostic attacks. The finding of the 3C prior reveals the potential and possibility for defending against a wide range of attacks, and provides a new perspective to overcome such challenge in this field.

Existing weakly supervised video violence detection (VVD) methods primarily rely on Euclidean representation learning, which often struggles to distinguish visually similar yet semantically distinct events due to limited hierarchical modeling and insufficient ambiguous training samples. To address this challenge, we propose PiercingEye, a novel dual-space learning framework that synergizes Euclidean and hyperbolic geometries to enhance discriminative feature representation. Specifically, PiercingEye introduces a layer-sensitive hyperbolic aggregation strategy with hyperbolic Dirichlet energy constraints to progressively model event hierarchies, and a cross-space attention mechanism to facilitate complementary feature interactions between Euclidean and hyperbolic spaces. Furthermore, to mitigate the scarcity of ambiguous samples, we leverage large language models to generate logic-guided ambiguous event descriptions, enabling explicit supervision through a hyperbolic vision-language contrastive loss that prioritizes high-confusion samples via dynamic similarity-aware weighting. Extensive experiments on XD-Violence and UCF-Crime benchmarks demonstrate that PiercingEye achieves state-of-the-art performance, with particularly strong results on a newly curated ambiguous event subset, validating its superior capability in fine-grained violence detection.

Semi-supervised learning (SSL) provides a practical framework for leveraging massive unlabeled samples, especially when labels are expensive for facial expression recognition (FER). Typical SSL methods like FixMatch select unlabeled samples with confidence scores above a fixed threshold for training. However, these methods face two primary limitations: failing to consider the varying confidence across facial expression categories and failing to utilize unlabeled facial expression samples efficiently. To address these challenges, we propose an Enhanced Adaptive Confidence Margin (EACM), consisting of dynamic thresholds for different categories, to fully learn unlabeled samples. Specifically, we employ the predictions on labeled samples at each training iteration to learn an EACM. It then partitions unlabeled samples into two subsets: (1) subset I, including samples whose confidence scores are no less than the margin; (2) subset II, including samples whose confidence scores are less than the margin. For samples in subset I, we constrain their predictions on strongly-augmented versions to match the pseudo-labels derived from the predictions on weakly-augmented versions. Meanwhile, we introduce a feature-level contrastive objective to enhance the similarity between two weakly-augmented features of a sample in subset II. We extensively evaluate EACM on image-based and video-based facial expression datasets, showing that our method achieves superior performance, significantly surpassing fully-supervised baselines in a semi-supervised manner. Additionally, our EACM is promising to leverage cross-dataset unlabeled samples for practical training to boost fully-supervised performance.

High-dimensional and incomplete (HDI) data are ubiquitous in various Big Data-related industrial applications, such as drug innovation and recommender systems. Hash-learning is the most efficient representation learning approach to extract hidden information from HDI data owing to its fast reasoning and low storage. However, an existing hash learning approach commonly employs gradient-based optimization techniques to address the discrete objective caused by the binary nature of hash factors, where the Quantization (i.e., quantizing the real values to binary codes) loss is inevitable, resulting in accuracy loss when representing HDI data. Motivated by these critical and vital issues, this paper proposes a non-gradient hash factor (NGHF) model with three-fold ideas: a) innovating a discrete differential evolution (DDE) algorithm able to simulate the continuous optimization via disabling bits of binary codes based on the projected Hamming dissimilarity, thus enabling an effective discrete optimizer, b) applying the proposed DDE algorithm to directly optimize the discrete learning objective of NGHF defined on HDI data, thereby facilitating its efficient and precise training without any Quantization loss, and c) theoretically proving the convergence of NGHF. As such, NGHF possesses high representation learning ability comparable to that of a real-valued model, making it able to achieve precise binary representation to HDI data. Extensive experimental results on nine real-world datasets demonstrate that NGHF significantly outperforms eight state-of-the-art hash learning models. Moreover, its accuracy is amazingly comparable to that of a real-valued model for HDI data representation learning.

The rapid development of generative AI techniques enables the synthesis of highly realistic facial images, posing significant challenges for the accurate detection of face forgeries. In contrast to solely elevating detector awareness, proactively reducing the intrinsic difficulty of forgery detection can streamline detector complexity while improving both generalization and robustness. This insight motivates our defense strategy to make face forgery clues more evident. Specifically, a novel proactive approach dubbed Self-Steganographic Detection (SSD) is proposed to imperceptibly embed facial images into themselves as a form of detection evidence. The recovery process is designed to remain robust under normal manipulations while exhibiting deliberate degradation under malicious manipulations, thereby clearly revealing potential forgeries. Unlike embedding bit-level vectors, pixel-level images are informative to ensure the generalization of our approach. Due to the similarity between the protected and embedded images, SSD performs detection without storing any embedded information in advance. To support practical deployment, our approach incorporates a dual detection scheme that aims to identify unprotected images and determine the authenticity of protected images. Extensive experiments using 8 face forgery techniques demonstrate the effectiveness of our approach compared to state-of-the-art methods.

Cross-domain medical image segmentation has gained increasing interest for its potential to reduce annotation efforts and improve clinical generalization capabilities. Domain adaptation aims to tackle the domain shift that appears in different image modalities. In cross-domain segmentation, generative models often suffer from limited accuracy due to their lack of domain-specific representations. Besides, many transfer learning approaches rely on additional manual annotations for supervision, emerging paradigms such as Unsupervised Domain Adaptation (UDA) facilitate effective knowledge transfer even when labels in the target domain are entirely absent. In this study, we propose a novel Point-cloud Structure Constrained Unsupervised Domain Adaptation (PSC-UDA) framework based on a Contour-Aware Segmentation (CAS) model with a 3D contour point cloud to bridge the domain gaps appearing in cross-site and cross-domain medical images. The CAS model distills the domain-invariant kidney structure from image texture to distinguish the point cloud and characterize the kidney contour in a coarse-to-fine way. With point-to-voxel self-learning on 3D structure constraints, the proposed PSC-UDA framework addresses visual domain shift, adapting discriminative information of the kidney from the labeled source domain (CT) to the unlabeled target domain (CT/MRI), so that it realizes precise cross-domain kidney segmentation with limited labels. Experimental results prove that the proposed method outperforms the generative UDA methods and the source-free methods on three cross-domain kidney segmentation datasets, outperforming even without a target domain adaptation strategy. The source code is available at https://github.com/zzs95/PSC-UDA .

Most existing deepfake detection methods learn visual artifacts in Euclidean space, overlooking the intrinsic hierarchical relationships among forgery samples. In practice, manipulated images deviate progressively from real ones, forming a hierarchy that Euclidean geometry struggles to model, while hyperbolic space better captures such structures due to its exponential capacity. Motivated by this observation, we propose a dual-space hierarchical learning framework that jointly models Euclidean visual semantics and hyperbolic hierarchical representations. Specifically, Euclidean features are projected into hyperbolic space, where a Hyperbolic Hierarchy-Aware Attention (HHAA) module is introduced to capture hierarchical dependencies among samples. To effectively integrate complementary information from the two spaces, a Dual-Space Gated Fusion (DSGF) module is designed to adaptively inject hierarchical cues into Euclidean features. A joint supervision strategy is further applied to enhance discriminative representation learning. Extensive experiments on multiple deepfake detection benchmarks demonstrate that the proposed method improves detection performance and generalization ability.

Recent video frame interpolation (VFI) methods rely on computationally heavy modules (e.g., global attention module) to handle large motions, incurring prohibitive costs which hinders their practical real-time deployment. In this work, we revisit the core objective of VFI: enhancing the temporal resolution of videos. We identify that previous VFI’s frame-isolated processing ignores continuous temporal modeling, introduces computational redundancy in video streaming scenarios. To address this, we propose an online learning recurrent net with inertia-aware motion estimation(IMEVSI). It consists of implicit motion propagation( IMP), explicit motion propagation(EMP) and adaptive online learning strategy(AOL). IMP and EMP are used to high order inter-frame motion modeling considering motion inertia, AOL are proposed to bridge the motion domain gap between training and deployment. For IMP, we initiate from explicit physical motion modeling, progressively integrating learnable parameters into inertia-ware motion extraction and finally unify motion propagation and extraction within our recurrent motion propagation Transformer(RMPT). For EMP, we directly inject adjacent motion into current flow estimation recognizing its inertia contribution. For AOL,we leverage cycle consistency to dynamically adjust intermediate flow estimator and maintains an adaptive threshold to control parameter update. Extensive experiments demonstrate that our method outperforms state-of-the-art (SOTA) approaches on regular, large-motion, and high-resolution benchmarks while achieving excellent inference speed and FLOPs.

Existing depth-based 3D hand pose estimation methods typically estimate hand joints from either 2D depth images or 3D point clouds, whereas the approaches that fuse multimodal data remain underexplored. Furthermore, previous methods often struggle to learn geometric-facilitated features and precise joint correlations, especially for occluded hands, due to the lack of explicit prior guidance and insufficient cross-dimensional interaction. By taking advantage of multi-modal fusion, cross-dimensional interaction, and prior guidance, we propose a novel joint-guided keypoint denoising Transformer (named HandJoKe) to achieve more precise hand pose estimation, which can iteratively estimate hand poses based on keypoint features from both 2D depth images and 3D point clouds under explicit joint guidance within only several denoising steps. Rather than directly applying existing multi-modal fusion to perform redundant interactions among many background pixels and irrelevant points, HandJoKe focuses on modeling correlations and capturing dependencies among local informative hand regions (i.e., keypoints), thus attaining higher learning capability with lower computation redundancy. Moreover, a novel joint-guided denoising estimation strategy is introduced to adequately fuse cross-modal keypoint features under explicit joint guidance, achieving geometric-facilitated cross-modal keypoint interaction in both 2D and 3D spaces. The effectiveness of joint guidance can be further strengthened through iterative denoising, since it can subsequently update cross-modal keypoint features based on previous denoised hand poses and thus can help better locate confused joints, especially for occluded hands. Extensive experiments show that HandJoKe has achieved state-of-the-art performance on four public challenging benchmarks, including single-hand datasets NYU and ICVL, and hand-object datasets DexYCB and HO3D.

Recent advancements in deep learning have significantly lowered the cost of generating and processing facial images, but this progress has also presented a growing challenge for face forgery detection. Existing detection methods, however, often struggle to generalize effectively to forged samples generated by previously unseen forgery techniques. Additionally, transferring knowledge through knowledge distillation to enhance generalization remains a persistent challenge. To address these issues, this paper proposes Adaptive Consensus Multi-teacher Knowledge Distillation (ACMD), a novel framework aimed at improving the generalization capabilities of face forgery detection models. ACMD leverages multiple teacher models with varying network architectures and introduces a consensus mechanism to resolve knowledge conflicts arising from the diversity of teacher models. It further adapts the assignment of sample-specific distillation weights, based on each teacher model’s prediction confidence. By combining this with feature-based knowledge distillation, the student model gains a deeper understanding of the teacher’s knowledge. Extensive experiments on the DeepfakeBench benchmark demonstrate that ACMD not only outperforms a single teacher model but also achieves state-of-the-art performance in dataset evaluations. Specifically, in a cross-dataset evaluation from FaceForensics++ to CDFv2 and DFDC, ACMD achieves frame-level AUCs of 84.22% and 74.43%, respectively, surpassing all baseline models. Ablation studies further validate the effectiveness of each component and highlight their complementary contributions to the overall detection performance.

In the context of zero-shot learning, retrieving natural images using sketch queries is referred to as Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR). The core problems of ZS-SBIR arise from the significant domain gap between sketches and photographs. The knowledge gap between seen and unseen categories is a crucial problems as well. To address these two issues, we propose a modality-aware bidirectional mining strategy aimed at enhancing the model’s understanding of multimodal data and mitigating the modality gap between sketches and photographs. Additionally, we introduce an improved Mixed Sample Data Augmentation (MSDA) technique to generate mixed samples as supplementary training data. Based on these generated mixed samples, we propose a mixed embedding mining strategy. This strategy aims to optimize the embedding space and enhance the model’s generalization ability from seen to unseen categories. Extensive experimental results on the TU-Berlin Ext., Sketchy Ext., and QuickDraw Ext. datasets demonstrate that our proposed model outperforms existing methods.

Image aesthetic assessment (IAA) is a challenging task due to the subjectivity and abstraction of aesthetic perception. Psychological studies reveal that aesthetic experiences often trigger emotional responses, while comment texts directly reflect people’s expressions of aesthetics and emotions. However, existing multimodal IAA methods neglect the alignment between modalities. To address this, we propose a multimodal emotion-alignment cognitive network (MEC-Net) for IAA, employing strategies of emotion alignment, subjective–objective interaction, and multimodal fusion. First, an emotion alignment module is introduced to align image and text modalities using emotional stimuli, enhancing the consistency of heterogeneous modal features. Then, a subjective and objective representation module is proposed to extract multi-source information from text and images separately. Next, a subjective-objective interactive LSTM (SO-LSTM) is designed to capture the deep interaction between images and text in aesthetic understanding. Finally, an dynamic multimodal fusion (DMF) based on low-rank decomposition is proposed to integrate subjective, objective, and subjective-objective interactive modal features for aesthetic distribution prediction. Extensive experiments and qualitative analysis on image aesthetic benchmarks indicate that the proposed MEC-Net outperforms the state-of-the-art on three IAA tasks. Further, we increase emotion classification task-driven evaluation metrics to verify the strong generalizability of the proposed MEC-Net.

Deepfake detection refers to detecting artificially generated or edited faces in images or videos, which plays an essential role in visual information security. Despite promising progress in recent years, Deepfake detection remains a challenging problem due to the complexity and variability of face forgery techniques. Existing Deepfake detection methods are often devoted to extracting features by designing sophisticated networks but ignore the influence of perceptual quality of faces. Considering the complexity of the quality distribution of real and fake faces, we propose a deepfake detection framework called DeepFidelity, which mines the perceptual forgery fidelity of face images and introduces a quality-aware scoring mechanism to distinguish real and fake faces of different image qualities. Specifically, we improve the model’s ability to identify complex samples by mapping real and fake face data of different qualities to different scores to distinguish them in a more detailed way. In addition, we propose a network structure called Symmetric Spatial Attention Augmentation based vision Transformer (SSAAFormer), which uses the symmetry of face images to promote the network to model the geographic long-distance relationship at the shallow level and augment local features. Extensive experiments on multiple benchmark datasets demonstrate the superiority of the proposed method over state-of-the-art methods. The code is available athttps://github.com/shimmer-ghq/DeepFidelity.

Inverse problems in medical imaging, such as undersampled magnetic resonance imaging (MRI) and sparse-view computed tomography (CT) reconstruction, are essential yet challenging tasks for achieving accurate and reliable diagnostic images. Traditional reconstruction approaches, including iterative optimization algorithms and supervised deep learning methods, often struggle with limited adaptability across imaging protocols, substantial computational requirements, and poor generalization between different imaging modalities. Diffusion-based generative models have recently demonstrated promising results; however, these methods frequently suffer from cumulative estimation errors in their sampling processes, limiting their practical performance and robustness. In this paper, we propose a novel framework called Parallel Trajectory Constrained Sampling (PCS), which substantially enhances image reconstruction quality by explicitly enforcing consistency with the underlying physical measurement process. Specifically, PCS introduces a measurement-domain diffusion model whose reverse stochastic differential equation (SDE) trajectory is analytically determinable, thus obviating the need for a learned score estimator within the measurement domain. Furthermore, a parallel trajectory constraint is formulated to rigorously align the reverse sampling paths of the measurement and image diffusion processes, ensuring strict adherence to the known physical model at every sampling step. The proposed PCS method is flexible and can seamlessly integrate various SDE-based diffusion priors. Extensive experiments on representative inverse problems—including undersampled MRI reconstruction, sparse-view CT reconstruction, and image super-resolution—demonstrate that PCS consistently outperforms existing state-of-the-art diffusion-based reconstruction methods. Although current evaluations focus specifically on MRI and CT modalities, the PCS framework holds considerable promise for broader applicability to other imaging modalities and inverse problems, which we plan to investigate in future studies.

Modeling visual perception in a manner consistent with human subjective evaluation has become a central direction in both video quality assessment (VQA) and broader visual understanding tasks. While free-energy-guided self-repair mechanisms—reflecting human observational experience—have proven effective in image quality assessment, extending them to VQA remains non-trivial. In addition, biologically inspired paradigms such as holistic perception, local analysis, and gaze-driven scanning have achieved notable success in high-level vision tasks, yet their potential within the VQA context remains largely underexplored. To address these issues, we propose EyeSimVQA, a novel VQA framework that incorporates free-energy-based self-repair. It adopts a dual-branch architecture, with an aesthetic branch for global perceptual evaluation and a technical branch for fine-grained structural and semantic analysis. Each branch integrates specialized enhancement modules tailored to distinct visual inputs—resized full-frame images and patch-based fragments—to simulate adaptive repair behaviors. We also explore a principled strategy for incorporating high-level visual features without disrupting the original backbone. In addition, we design a biologically inspired prediction head that models sweeping gaze dynamics to better fuse global and local representations for quality prediction. Experiments on five public VQA benchmarks demonstrate that EyeSimVQA achieves competitive or superior performance compared to state-of-the-art methods, while offering improved interpretability through its biologically grounded design. Our code will be publicly available at https://github.com/handsomewzy/EyeSim-VQA.

Despite the considerable advancements in cross-domain image translation, a significant challenge remains in addressing information asymmetric translation tasks such as SAR-to-Optical and Sketch-to-Instance conversions. These tasks involve transforming data from a domain with limited information into one with more detailed and richer content. Traditional CNN-based methods, while effective at capturing intricate details, often struggle to grasp the overall structural composition of the image, leading to unintended blending or merging of distinct regions within the generated images. In light of these limitations, research has increasingly turned toward Transformers. Though Transformers excel at capturing global structures, they often lack the ability to preserve fine-grained details. Recognizing the importance of both detailed features and structural relationships in information asymmetric translation tasks, we introduce the CNN-Swin Hybrid Network (CSHNet). This network employs a novel bottleneck architecture featuring two key modules: Swin Embedded CNN (SEC) and CNN Embedded Swin (CES), which together form the SEC-CES-Bottleneck (SCB). Within this structure, SEC capitalizes on CNN’s capability for detailed feature extraction while incorporating the Swin Transformer’s inherent structural bias. In contrast, CES preserves the Swin Transformer’s strength in maintaining global structural integrity, while compensating for CNN’s tendency to emphasize detail. In addition to the SCB architecture, CSHNet integrates two essential components designed to improve cross-domain information retention and ensure structural consistency. The Interactive Guided Connection (IGC) fosters dynamic information exchange between SEC and CES, encouraging a deeper understanding of image details. At the same time, Adaptive Edge Perception Loss (AEPL) is implemented to preserve well-defined structural boundaries throughout the translation process. Experimental evaluations demonstrate that CSHNet surpasses current state-of-the-art methods, achieving superior results in both visualization and performance metrics across scene-level and instance-level datasets. Our code is available at: https://github.com/XduShi/CSHNet.

Recently, lightweight networks for single image super-resolution (SISR) have surged due to the need of resource-constrained devices, where divide-and-conquer multi-route model exhibits impressive trade-off between performance and computational cost. However, most existing divide-and-conquer multi-route models face two key limitations: (1) possible suboptimal decoupling of image components (e.g. smooth regions, edges and texture details) due to spatial-domain-only processing, and (2) inability to model global dependencies explicitly and capture structural information, hindering further performance gains. To address these drawbacks, we propose a lightweight frequency-selection-based progressive patch Transformer network (FSPPTN) for higher-quality SISR reconstruction. Specifically, we first propose a frequency selection module, in which we develop a frequency enhancement branch (FEB) to dynamically decouple different image components by introducing the window-based Fast Fourier transform (WFFT) and a learnable weight matrix, and a spatial restoration branch (SRB) to recalibrate and fuse cross-granularity features by designing a multi-gate mechanism for reconstructing the component information screened out by the FEB at current level. Secondly, we propose a lightweight multi-branch gradient-guided inter-patch self-attention to explicitly capture global structural similarities by summarizing structural information of each patch into a lower-dimensional space using the statistical properties of first-order gradients, thereby achieving explicit global dependencies modeling and lightweight. Extensive experimental results demonstrate that, in the vast majority of cases, FSPPTN outperforms state-of-the-art lightweight SISR methods in terms of both performance and computational overhead, especially for ×3 and ×4 SR, e.g. FSPPTN outperforms MaIR-Small by even 0.14dB PSNR on Manga109 dataset for ×4 SR even with 48.3% fewer parameters and 63.5% lower FLOPs. The code is available at: https://github.com/yslyangshuli/FSPPTN-main.

Generating high-quality facial photos from fine-detailed sketches is a long-standing research topic that remains unsolved. The scarcity of large-scale paired data due to the cost of acquiring hand-drawn sketches poses a major challenge. Existing methods either lose identity information with oversimplified representations, or rely on costly inversion and strict alignment when using StyleGAN-based priors, limiting their practical applicability. Our primary finding in this work is that the discrete codebook and decoder trained through self-reconstruction in the photo domain can learn rich priors, helping to reduce ambiguity in cross-domain mapping even with current small-scale paired datasets. Based on this, a cross-domain mapping network can be directly constructed. However, empirical findings indicate that using the discrete codebook for cross-domain mapping often results in unrealistic textures and distorted spatial layouts. Therefore, we propose a Hierarchical Adaptive Texture-Spatial Correction (HATSC) module to correct the flaws in texture and spatial layouts. Besides, we introduce a Saliency-based Key Details Enhancement (SKDE) module to further enhance the synthesis quality. Overall, we present a “reconstruct-cross-enhance” pipeline for synthesizing facial photos from fine-detailed sketches. Experiments demonstrate that our method generates high-quality facial photos and significantly outperforms previous approaches across a wide range of challenging benchmarks. The code is publicly available at: https://github.com/Gardenia-chen/DECP.

Deep neural networks have recently achieved promising performance in the vein recognition task and have shown an increasing application trend. However, they are prone to adversarial attacks by adding imperceptible perturbations to the input, resulting in incorrect recognition. To address this issue, we propose a novel defense model named MsMemoryGAN, which aims to filter the perturbations from adversarial samples before recognition. First, we design a multiscale memory autoencoder (MsMemoryAE) to achieve high-quality reconstruction, where the memory module (MM) within it is capable of learning the detailed patterns of normal samples at different scales. Second, to overcome the limitations of handcrafted similarity metrics, we propose an MM with learnable similarity (LSMM), which retrieves the most relevant memory items to purify the input feature. Finally, the perceptual loss and adversarial loss are integrated with the pixel loss to further enhance the quality of the reconstructed image. During the training phase, the MsMemoryGAN learns to reconstruct the input by merely using fewer prototypical elements of the normal patterns recorded in the memory. At the testing stage, given an adversarial sample, the MsMemoryGAN retrieves its most relevant normal patterns in MMs for reconstruction. Perturbations in the adversarial sample are usually not reconstructed well, resulting in adversarial purification. We conduct extensive experiments on two public vein datasets under different adversarial attack methods to evaluate the performance of the proposed approach. The experimental results show that our approach removes a wide variety of adversarial perturbations, allowing vein classifiers to achieve the highest recognition accuracy.

Large-scale dynamic weighted directed network (DWDN) is commonly utilized to illustrate the temporal interactions between nodes in numerous applications. Latent factorization of tensors (LFT) is a typical representation learning approach to extract the desired knowledge from a DWDN via low-rank tensor embedding. However, an existing LFT approach requires the target DWDN to be maintained in one central place like a central server, which is becoming unacceptable for users who are getting increasingly privacy-sensitive. To address this vital issue, this paper innovatively proposes a federated latent factorization of tensors (FLFT) model. It can perform accurate and privacy-preserving representation learning to a DWDN based on four-fold ideas: 1) establishing a data-density-oriented federated learning framework to enable different users to efficiently and cooperatively build a shared LFT model with keeping raw data privacy, 2) incorporating the linear biases into the local training of each user to eliminate the personalized biases or local fluctuations, 3) adopting an effective hybrid filling strategy to further protect each user's private interaction information, and 4) designing a customized nonlinear activation function to capture the nonlinear characteristics of users' interactions. Extensive experiments on four DWDNs collected from industrial applications validate that FLFT demonstrates a notable increase in accuracy compared with state-of-the-art federated and non-federated learning approaches in representing a DWDN. The source code of the proposed FLFT model is shared at the following link:https://github.com/wudi1989/FLFT.

With the development of deep learning technology, the facial images generated by deepfake technology have reached a level of authenticity that is difficult to distinguish, posing a serious threat to personal privacy and data security. Therefore, it is of great significance to develop efficient and reliable deepfake detection technology. In recent years, Visual Language Models (VLM) have been applied to deepfake detection tasks due to their powerful multimodal understanding capabilities. However, the existing VLM have not been specifically optimized for deepfake detection tasks. When directly applied to this task, there are problems such as insufficient model accuracy and insufficient feature extraction, especially when dealing with complex forgery scenes. In response to these challenges, this paper proposes an innovative deepfake face detection method based on VLM and component-specific prompt tuning. We transform the deepfake detection task into a Visual Question Answering (VQA) task, making full use of the multimodal understanding capabilities of VLM and the flexibility of prompt tuning technology. This method uses a local prompt strategy to customize specific prompt questions for key facial components such as eyes, nose, and mouth, guiding the model to focus on the local features of these areas, thereby accurately capturing forgery traces. In addition, we introduced a feature extraction module Q-Former based on instructions, which can flexibly adjust the focus area of visual features according to prompts, significantly improving the model’s perception of locally forged features. By fusing these local features extracted by Q-Former and combining them with the language model to judge the authenticity of the overall face image, we can finally generate accurate prediction results. A large number of experimental results show that our method is significantly better than existing technologies in terms of detection accuracy and robustness.

Network traffic anomaly detection is critical for cybersecurity but faces challenges in accurately identifying malicious activities. Recent zero-positive approaches, which use only normal training data under the reconstruction paradigm, have shown progress. However, encrypted network traffic obscures normal–anomalous distinctions, causing confused modeling. In addition, the “identical shortcut” problem, where models reconstruct any input with similar fidelity, produces suboptimal representations and indistinguishable detection. To address these limitations, this paper introduces ConMD, a novel Contextual Masking Knowledge Distillation framework. ConMD features distillation paradigm for discriminative representations and then pursues two objectives: effective contextual information modeling and a comprehensive anomaly metric. Specifically, we introduce context-aware local-global attention mechanisms for the student network's backbone, which capture both intra-packet and inter-packet dependencies. Additionally, a context-enhanced masking training strategy is designed to facilitate contextual interactions in normal flows. Given the structural characteristics of network traffic, we also present a new anomaly scoring with multi-view awareness, which perceive comprehensive traffic patterns. ConMD combines insights from both packet- and flow-level views to highlight deviations in anomalous network flows, thereby improving detection accuracy. Extensive experiments on three real-world datasets validate the effectiveness of ConMD, yielding consistent improvements over state-of-the-art baselines, achieving up to 2.8% and 5.1% AUC gains on the DataCon2020 and CIC-IDS2017 datasets, respectively. Our model code will be released at https://github.com/ikun0124/ConMD.

Owing to its inherent attributes of high security, privacy preservation, and liveness detection, vein recognition has garnered significant attention, with deep learning (DL) models prevailing in the field. In particular, Mamba, a recent DL architecture showing robust feature representation with linear computational complexity, has been applied successfully for visual tasks. However, Vision Mamba captures long-distance feature dependencies but deteriorates local feature details. Besides, manually designing Mamba architecture based on human prior knowledge is very time-consuming and error-prone. To address these limitations, we propose a hybrid network structure named Global-local Vision Mamba (GLVM) to learn both local correlations and global dependencies within images for comprehensive vein feature representation. Second, we design a Multi-head Mamba to learn the dependencies along different directions, so as to improve the feature representation of Vision Mamba. Third, to learn complementary features, we propose a ConvMamba block consisting of three branches: Multi-head Mamba branch (MHMamba), Feature Iteration Unit branch (FIU), and Convolutional Neural Network (CNN) branch, with FIU aiming to fuse convolutional local features with Mamba global representations. Finally, we propose a Global-local Alternate Neural Architecture Search (GLNAS) method, which alternately searches for the optimal architecture of GLVM through weight entanglement strategy and evolutionary algorithm. We have carried out rigorous experiments on five public vein datasets to assess performance. Our approach achieves the highest 96.84%, 99.63%, 95.73%, 99.72%, 99.14% accuracies and the lowest 0.27%, 0.07%, 0.48%, 0.07%, 0.12% EER among all existing approaches on five public vein datasets, which demonstrates that our approach is capable of learning more complete features than existing approaches. In addition, the visual assessment experiments also show that our approach extracts more global vein architecture and local vein detail for recognition.

Intermediate flow estimation is an important part of video frame interpolation (VFI). Most previous works use interpolation to derive the intermediate flow assuming localized linear motion. However, this method is not effective when dealing with extreme motions. In this work, we assume that the motion trajectory of an object is determined by the appearance characteristics of this object. Based on this assumption, we propose a new intermediate flow estimation method, which obtains the motion features of intermediate frames from image appearance and inter-frame motion features. In addition, in order to fully extract the inter-frame features, we rethink the difference of VFI and previous works on using Swin-Transformer and compute the appearance features and motion features within the adaptive neighborhood by cyclically shifting the window. Experimental results show that our method achieves state-of-the-art performance on different datasets for both fixed-time and arbitrary-time interpolation. Moreover, our proposed method outperforms models that require inputting a sequence of four frames when handling videos with extremely large motion. The source code is available from https://github.com/chen12304/IFE-VFI.

Efficiently and accurately recognizing interesting objects within the image and regressing bounding boxes to enclose them has been a persistent pursuit in object detection. However, existing detectors fail to achieve both aspects simultaneously due to insufficient task interaction and suboptimal classification behavior. To solve the problem, this paper proposes a novel detector with Efficient Asymmetric Progressive Semi-Decoupled Head (EAPSDH) and Harmonic Focal Loss (HFL). Specifically, we generalize the detection head into a progressive asymmetric paradigm that performs hierarchical and dynamically recalibrated interaction between classification and localization, enabling iterative mutual enhancement in an efficient manner beyond the prior designs. Meanwhile, HFL is proposed to improve classifier optimization by addressing the imbalance between positive and negative samples. HFL dynamically increases the loss weights of positive samples, amplifying their gradient contributions during classifier training, which significantly reduces classification error. By jointly improving task-specific feature representation and classification optimization, EAPSDH and HFL complement each other to alleviate the inconsistency between classification and localization performance, resulting in an efficient and accurate one-stage detector termed EADet. Experimental results on the MS COCO database demonstrate that EADet effectively mitigates the inconsistency between classification and localization performance. Furthermore, EADet achieves a strong trade-off between accuracy and speed, reaching 47.4 AP at 33.2 FPS on the MS COCO with ResNet-101 under the $2\times $ training schedule, demonstrating its effectiveness compared with recent state-of-the-art detectors. Code will be available at https://github.com/HB-X/EADet.

iffusion-based image super-resolution (SR) has shown strong potential in recovering high-fidelity details from low-resolution inputs. However, the need for tens or hundreds of sampling steps leads to substantial inference latency. Recent works attempt to accelerate this process via knowledge distillation, but often rely solely on pixel-level loss or overlook the fact that diffusion models capture different information across time steps. To address this, we propose TAD-SR, a time-aware diffusion distillation framework. Specifically, we introduce a novel score distillation strategy to align the score functions between the outputs of the student and teacher models after minor noise perturbation. This distillation strategy eliminates the inherent bias in score distillation sampling (SDS) and enables the student models to focus more on high-frequency image details by sampling at smaller time steps. We further introduce a time-aware discriminator that exploits the teacher’s knowledge to differentiate real and synthetic samples across different noise scales, using explicit temporal conditioning. Extensive experiments on SR tasks demonstrate that TAD-SR outperforms existing singl-estep diffusion methods and achieves performance on par with multi-step state-of-the-art models.iffusion-based image super-resolution (SR) has shown strong potential in recovering highfidelity details from low-resolution inputs. However, the need for tens or hundreds of sampling steps leads to substantial inference latency. Recent works attempt to accelerate this process via knowledge distillation, but often rely solely on pixel-level loss or overlook the fact that diffusion models capture different information across time steps. To address this, we propose TADSR, a time-aware diffusion distillation framework. Specifically, we introduce a novel score distillation strategy to align the score functions between the outputs of the student and teacher models after minor noise perturbation. This distillation strategy eliminates the inherent bias in score distillation sampling (SDS) and enables the student models to focus more on highf-requency image details by sampling at smaller time steps. We further introduce a time-aware discriminator that exploits the teacher’s knowledge to differentiate real and synthetic samples across different noise scales, using explicit temporal conditioning. Extensive experiments on SR tasks demonstrate that TAD-SR outperforms existing single-step diffusion methods and achieves performance on par with multi-step state-of-the-art models D.

Low-light Image Enhancement Algorithms (LIEAs) aim to improve the visibility and visual quality of images captured in low-light environments. However, none of the existing LIEAs can comprehensively restore all visual contents, which makes it inevitable for the Enhanced Low-light Images (ELIs) to have different degrees of distortion, thereby affecting the visual quality. Currently, there is little research focusing on the quality assessment of these ELIs, partly due to the lack of publicly available datasets. Moreover, existing quality assessment methods primarily focus on a single visual modality and fail to sufficiently exploit the structural information across multiple image attributes, consequently resulting in suboptimal prediction performance. To this end, this paper conducts a systematic study on both subjective and objective quality assessment of ELIs. Firstly, we construct the first Multi-annotated and multi-modal Low-light image Enhancement quality dataset (MLE), which contains 1,000 ELIs, along with subjective studies to obtain multiple attribute annotations, quality scores, and textual descriptions. Based on this, we further propose an Attribute-guided Vision-Language Graph Reasoning Network (AVGR-Net) for ELI quality prediction, which effectively integrates multi-attribute visual and textual information through cross-modal graph reasoning and alignment. Extensive data analysis and experimental results validate both the reliability of the MLE dataset and the superior performance of the AVGR-Net compared to state-of-the-art methods.

Existing meta-learning based few-shot object detection methods suffer from limitations in learning representative prototypes. Specifically, directly aggregating bounding box contents from support images into prototypes renders these methods vulnerable to background noise and the morphological intricacies of objects. Furthermore, these methods neglect the varied contributions of intra-class image-specific prototypes and fail to leverage semantic information effectively during prototype generation, resulting in suboptimal class representations due to naive average aggregation. To address these issues, we propose a Dual Prototype-Enhancement Network (DPENet), designed to optimize prototypes by improving support feature representation and enhancing prototype discriminability. Specifically, we introduce an Object Enhancement Module (OEM) based on dynamic hypergraph construction. This module employs hypergraph convolution to adaptively capture complex high-order semantic interactions among highly similar regions within support features, thereby highlighting salient features of target regions, suppressing background noise, and enhancing support feature representation. Moreover, we propose a Semantic Fusion Perception Module (SFPM) that generates more discriminative class-specific prototypes by integrating weighted intra-class prototype representations with text-based semantic embeddings. Experimental results demonstrate that DPENet significantly outperforms existing methods on the PASCAL VOC and MS COCO datasets.

Existing object detection methods struggle to generalize across increasingly data domains while simultaneously adapting to the emergence of novel categories. To tackle this challenge, adaptive open-set object detection (AOOD) has been introduced, which employs supervised training on base categories within the source domain while enabling unsupervised adaptation to both base and novel categories in the target domain. However, existing AOOD approaches are still hindered by several limitations, including insufficient cross-domain feature representation, inter-category ambiguity in novel classes, and inherent feature bias toward the source domain. To overcome these issues, this paper proposes a category-level collaboration knowledge mining strategy designed to comprehensively exploit both inter-class and intra-class feature relationships across domains. Specifically, a clustering-based memory bank (CMB) is initially constructed to aggregate class prototype features, class auxiliary features, and intra-class disparity features, thereby embedding rich category-level knowledge into a unified memory structure. The CMB is iteratively updated through unsupervised clustering, which facilitates the modeling of intra-category relationships and enhances its capacity for cross-domain knowledge representation. Subsequently, a base-to-novel selection metric (BNSM) is designed to identify features corresponding to novel categories within the source domain by regulating the relationships between the novel categories and each base category. The selected features are then leveraged to initialize the object detector for the classification of novel categories. Finally, an adaptive feature assignment (AFA) strategy is introduced to transfer the learned category-level knowledge to the target domain, enabling the assignment of category labels to features. The memory bank is updated asynchronously with these assigned features to mitigate source domain bias. Extensive experiments conducted on diverse domain datasets demonstrate that the proposed method consistently outperforms state-of-the-art AOOD approaches, achieving performance gains of 1.1 to 5.5 mAP. Code is available at https://github.com/Jandsome/CCKM.

The goal of compositional zero-shot learning (CZSL) is to train a model to recognize images containing known attribute-object pairs. This reduces the reliance on extensive training data and enables the model to identify unseen combinations. Current CZSL methods face several challenges, including multiple attributes for a single object, disconnected training and test sets, long-tailed distribution of visual categories, and substantial differences in state representation between different objects. These factors collectively impede the precise identification of new combinations. In response to these challenges, we propose a Semi-Negative Contrastive Subclass Discriminative Network (SN-CSDN) based on contrastive learning. Firstly, we propose a semi-negative sampling strategy that incorporates carefully selected negative samples into the training process. This approach enables the model to effectively distinguish between different classes while enhancing its ability to capture fine-grained subclass features. By improving the model's sensitivity to inter-class differences and refining its recognition of subtle intra-class variations, this strategy significantly boosts overall discrimination performance. Additionally, we introduce a decoupled network branch designed to capture the intricate relationships between attributes and objects by generating more representative compositional embeddings. This branch leverages subclass information to ensure an accurate classification of synthesized embeddings while preserving the inherent visual distinctions of the original decoupled embeddings across different combinations. By improving feature representation capacity and mitigating sample imbalance, this design effectively improves model performance in long-tailed distributions. Our method has been comprehensively evaluated on three benchmark datasets, with results showing significant performance improvements that demonstrate the method's effectiveness and reliability.

Unsupervised visible-infrared person re-identification (USVI-ReID) aims to learn modality-invariant image features from unlabeled cross-modal person datasets by reducing the modality gap while minimizing reliance on costly manual annotations. Existing methods typically address USVI-ReID using cluster-based contrastive learning, which represents a person by a single cluster center. However, they primarily focus on the commonality of images within each cluster while neglecting the finer-grained differences among them. To address the limitation, we propose a Hierarchical Identity Learning (HIL) framework. Since each cluster may contain several smaller sub-clusters that reflect fine-grained variations among images, we generate multiple memories for each existing coarse-grained cluster via a secondary clustering. Additionally, we propose Multi-Center Contrastive Learning (MCCL) to refine representations for enhancing intra-modal clustering and minimizing cross-modal discrepancies. To further improve cross-modal matching quality, we design a Bidirectional Reverse Selection Transmission (BRST) mechanism, which establishes reliable cross-modal correspondences by performing bidirectional matching of pseudo-labels. Extensive experiments conducted on the SYSU-MM01 and RegDB datasets demonstrate that the proposed method outperforms existing approaches. The source code is available at: https://github.com/haonanshi0125/HIL.

As a fundamental task in image processing, blind image restoration (BIR) faces significant challenges due to the unknown nature of the degradation process. While transformer-based methods have shown promise in various applications, they encounter difficulties in BIR. One key challenge is that the complexity of degradation easily leads to incorporate irrelevant information into their attention mechanisms, thereby hindering restoration performance. To address this challenge, sparsification strategies have been commonly adopted. However, existing sparse transformer-based methods typically determine sparse members through fixed patterns such as constant thresholds or predefined sources, making their sparsification strategies too rigid. To tackle this issue, we propose WSformer, a Wavelet-based Sparse transformer tailored for BIR, which offers three key advantages. First, we design a Sparse Reciprocal Multi-head Self-Attention (SR-MSA) mechanism in the attention layer. This mechanism employs sparse and reciprocal strategies to adaptively select reliable information, while operating across channels to reduce computational complexity. Second, recognizing that feed-forward networks in existing transformer blocks fail to effectively leverage global information, we develop a Recalibrated Feed-Forward Network (RFFN). It fully exploits the fusion of local and global information, enhancing the robustness of feature learning. Finally, to mitigate the increased computational burden introduced by these innovations, we equip WSformer with wavelet transform. Combined with a U-shaped architecture, it enables WSformer to achieve an optimal balance between performance and inference time. Extensive experiments on multiple BIR tasks validate WSformer's effectiveness in both quantitative metrics and visual quality. The code is available at https://github.com/CanZhang01/WSformer.

Existing image fusion methods have developed increasingly sophisticated network architectures for exploiting modality-shared and modality-specific features. However, despite these advancements in feature extraction, most methods ultimately rely on relatively simple implicit or explicit fusion strategies, which can compromise interpretability and limit fusion accuracy. In this paper, we incorporate visual autoregressive modeling to bridge the gap between implicit feature extraction and explicit modality fusion. First, the proposed approach conducts a low-to-high resolution autoregressive objective with modality-specific features, introducing a scalable feature autoregressive mechanism. It aggregates local and global contextual dependencies while enhancing implicit cross-scale interaction. Furthermore, to promote the consistency and complementarity across modalities, we embed an explicit high-order fusion strategy within the progressive modality-specific feature extraction process. This integration facilitates a next-scale synergistic relationship between implicit learning and explicit fusion. Our High-order Feature AutoRegressive Fusion framework (HFARFusion) provides a robust and interpretable solution for general image fusion tasks, effectively balancing fusion performance and transparency through the strengths of autoregressive learning. Extensive experiments demonstrate the outstanding performance of the proposed method in several classical fusion tasks, including infrared-visible, medical, multi-focus, and multi-exposure image fusion. Our code is available at https://github.com/happysbn/HFARFusion.

Reconstructing high-quality garment models from monocular images is important as it provides a practical and effective solution for human digitization and virtual try-on etc. Recent implicit function-based garment reconstruction methods recover free-form geometry but struggle to reconstruct individual garment meshes from human images and tend to produce disembodied limbs or degenerate shapes for novel views. In contrast, explicit parametric garment template models can be utilised to construct separate meshes and constrain the shape reconstruction robustly. However, this limits the reconstruction of garment details and shape variations, such as the wrinkles and pockets etc. To address this problem, in this paper, we introduce a novel explicit garment template that is designed for both closed and open garment topology. Powered by our new garment template, we further propose a detailed garment reconstruction method based on a monocular view that can process both the closed and open types for shape recovery. To capture those challenging parts with unknown geometry and topology, we predict displacement maps on the parameterization domain for the target garment from the monocular image and elaborate it to the 3D garment surface via the UV coordinates, achieving realistic details on the 3D garment shape. Extensive experiments demonstrate the accuracy and robustness of our method and show that realistic details like garment wrinkles and pockets can be faithfully recovered in an explicit way. The code and dataset are available at https://github.com/worryDes/GarmentRec.

Multi-temporal hyperspectral imagery (HSI) has become a powerful tool for change detection (CD) owing to its rich spectral signatures and detailed spatial information. Nevertheless, the application of paired HSIs is constrained by the scarcity of annotated training data. While unsupervised domain adaptation (UDA) offers a potential solution by transferring change detection knowledge from source to target domains, two critical limitations persist: 1) the labor-intensive process of acquiring and annotating source-domain paired samples, and 2) the suboptimal transfer performance caused by substantial cross-domain distribution discrepancies. To address these challenges, we present a Temporal Self-Construction Cross-Domain learning (TSCCD) framework for UDA-based HSI-CD. Our TSCCD framework introduces an innovative temporal self-construction mechanism that synthesizes bi-temporal source-domain data from existing HSI classification datasets while simultaneously performing initial data-level alignment. Furthermore, we develop a reweighted amplitude maximum mean discrepancy (MMD) metric to enhance feature-level domain adaptation. The proposed architecture incorporates an attention-based Kolmogorov-Arnold network (KAN) with high-frequency feature augmentation within an encoder-decoder structure to effectively capture change characteristics. Comprehensive experiments conducted on three benchmark HSI datasets demonstrate that TSCCD achieves superior performance compared to current state-of-the-art methods in HSI change detection tasks. Codes are available at https://github.com/Zhoutya/TSCCD.

State Space Models (SSMs), particularly Mamba, have emerged as promising alternatives to Transformers for lightweight Single Image Super-Resolution (SISR) due to their linear complexity in modeling long-range dependencies. Advanced Mamba-based Super-Resolution models, such as MambaIR and VMambaIR, have demonstrated compelling performance by integrating bidirectional and multi-directional scanning mechanisms. However, these methods are still constrained by several critical challenges that limit their ultimate reconstruction capabilities. Firstly, the inherent requirement for 2D-to-1D serialization disrupts local spatial coherence, which is vital for preserving fine-grained texture details during reconstruction. Second, stacking multiple SSM blocks in deep networks leads to progressive decay of fine-grained information, as hidden states are typically confined within individual layers without cross-layer propagation. To address these challenges, we propose PropMambaSR, a novel hybrid architecture that synergistically combines global context modeling with local feature extraction. Our core contribution is the Propagation State Space Model (PropSSM), which establishes explicit cross-layer hidden state routing to enable direct propagation of fine-grained information from shallow to deep layers, effectively mitigating information decay while preserving reconstruction details. Additionally, we introduce an Enhanced Spatial Feature Block (ESFB) that employs multi-scale residual distillation to explicitly capture local texture information compromised by the serialization process. Finally, a dynamic Fusion Model (FM) adaptively integrates global contextual features from PropSSM with local spatial features from ESFB through learnable gating mechanisms. Extensive experiments demonstrate that PropMambaSR achieves state-of-the-art performance across multiple benchmarks while maintaining computational efficiency. Notably, on the Urban100 dataset for$\times 2$SR, our model surpasses the next-best method by 0.33 dB in PSNR, demonstrating superior capability in reconstructing high-frequency textures and regular structural patterns.

3D hand pose estimation is crucial for many human-computer interaction applications. However, existing deep neural networks (DNNs) for 3D hand pose estimation suffer from poor generalizability due to data scarcity and a lack of domain-specific knowledge. In contrast, humans remain far better than DNNs at learning; Humans require fewer samples for learning new concepts under the guidance of their prior knowledge. Inspired by this, we propose a graph-enhanced CLIP to deliver visual-semantic priors to DNNs, and provide refined domain-specific knowledge for better 3D hand pose estimation. Specifically, we first introduce a pre-trained CLIP to guide the hand estimation model in learning the semantic-aware visual features, and text-free contrastive learning is proposed to effectively transfer high-level visual-semantic priors from the pre-trained large multimodal models. Notably, our strategy is data-agnostic and avoids designing hand-crafted text prompts for various visual inputs. Second, we introduce novel graph Transformers to refine the domain-specific knowledge by fully exploiting the local adjacent relations of hand joints and capturing the global structure representations of hand poses. The introduced graph Transformers are supposed to further refine the generalized CLIP feature for the downstream task (i.e., hand pose estimation) with better performance. Experiments show that our proposed graph-enhanced CLIP achieves state-of-the-art performances on benchmark datasets, demonstrating its effectiveness for 3D hand pose estimation. The source code is available athttps://github.com/TLiu2832/TRVSP-GE-CLIP.

Existing defense methods fail to defend against un known attacks and thus raise generalization issue of adversarial robustness. To remedy this problem, we attempt to delve into some underlying common characteristics among various attacks for generality. In this work, we reveal the commonly overlooked low entropy prior (LE) implied in various adversarial samples, and shed light on the universal robustness against unseen attacks in inference phase. LE prior is elaborated as two properties across various attacks as shown in Fig. 1 and 2: 1) low entropy misclassification for adversarial samples and 2) lower entropy prediction for higher attack intensity. This phenomenon stands in stark contrast to the naturally distributed samples. The LE prior can instruct existing test-time defense methods, thus we propose a two-stage REAL approach: Rectify Adversarial sample based on LE prior for test-time adversarial rectification. Specifically, to align adversarial samples more closely with clean samples, we propose to first rectify adversarial samples misclassified with low entropy by reverse maximizing prediction entropy, thereby eliminating their adversarial nature. To ensure the rectified samples can be correctly classified with low entropy, we carry out secondary rectification by forward minimizing prediction entropy, thus creating a Max-Min entropy optimization scheme. Further, based on the second property, we propose an attack aware weighting mechanism to adaptively adjust the strengths of Max-Min entropy objectives. Experiments on several datasets show that REAL can greatly improve the performance of existing sample rectification models.

Video Quality Assessment (VQA) technology is of significant importance for improving video transmission, storage, and processing. Although Convolutional Neural Networks (CNNs)-based and Transformer-based methods have achieved significant progress, they still suffer from some drawbacks. The previous methods treated video data as independent samples, thereby neglecting the close or distant relationships between different quality levels and consequently constraining the model’s discriminative ability; meanwhile, most existing fusion strategies utilize fixed architectures that lack the ability to adapt to the data for optimal integration, resulting in insufficient utilization of spatio-temporal information and limited expression capabilities of the fused features. To address the above issues, this article proposes a VQA method with In-Batch Contrastive Learning and Two-Phase Feature Fusion (IBCL-VQA). Firstly, the spatial features are extracted through two branches, which not only preserve the global semantics but also focus on the local regions. The temporal characteristics are obtained through a pre-trained video recognition model. Secondly, we propose an in-batch contrastive learning mechanism which, through the principles of maximizing intra-class similarity and minimizing inter-class similarity, combined with a dynamically adjusted penalty strategy, models the correlation between video quality levels. Thirdly, a two-phase feature fusion strategy, consisting of the Gated Spatio-temporal Attention Unit (GSTU) and the Adaptive Fusion Cell (AFC), is further proposed. The former achieves spatio-temporal feature fusion through dynamic weight allocation, and the latter adaptively integrates the features from the two branches based on data characteristics. Finally, a regression module outputs the quality score. Experimental results on five real-world VQA datasets demonstrate the superior performance of the IBCL-VQA. Furthermore, the strong generalizability is verified through cross-database testing. The code and pre-trained weights will be publicly available at: https://github.com/BoHu90/IBCL-VQA .

Recent advances indicate that diffusion model holds great promise in image super-resolution. While latest methods are primarily based on latent diffusion models with convolutional neural networks, there are few attempts to explore transformers, which have demonstrated remarkable performance in image generation. In this work, we design an effective diffusion transformer for image super resolution (DiT-SR) that achieves the visual quality of prior-based methods, but through a training-from-scratch manner. In practice, DiT-SR leverages an overall U-shaped architecture, and adopts uniform isotropic design for all the transformer blocks across different stages. The former facilitates multi-scale hierarchical feature extraction, while the latter reallocate the computational resources to critical layers to further enhance performance. Moreover, we thoroughly analyze the limitation of the widely used AdaLN, and present a frequency-adaptive time-step conditioning module, enhancing the model's capacity to process distinct frequency information at different time steps. Extensive experiments demonstrate that DiT-SR outperforms the existing training-from-scratch diffusion-based SR methods significantly, and even beats some of the prior-based methods on pretrained Stable Diffusion, proving the superiority of diffusion transformer in image super resolution.

Due to the successful development of deep image generation technology, forgery detection plays a more important role in social and economic security. Racial bias has not been explored thoroughly in the deep forgery detection field. In the paper, we first contribute a dedicated dataset called the Fair Forgery Detection (FairFD) dataset, where we prove the racial bias of public state-of-the-art (SOTA) methods. Different from existing forgery detection datasets, the self-constructed FairFD dataset contains a balanced racial ratio and diverse forgery generation images with the largest-scale subjects. Additionally, we identify the problems with naive fairness metrics when benchmarking forgery detection models. To comprehensively evaluate fairness, we design novel metrics including Approach Averaged Metric and Utility Regularized Metric, which can avoid deceptive results. We also present an effective and robust post-processing technique, Bias Pruning with Fair Activations (BPFA), which improves fairness without requiring retraining or weight updates. Extensive experiments conducted with 12 representative forgery detection models demonstrate the value of the proposed dataset and the reasonability of the designed fairness metrics. By applying the BPFA to the existing fairest detector, we achieve a new SOTA. Furthermore, we conduct more in-depth analyses to offer more insights to inspire researchers in the community.

Motion artifacts present in magnetic resonance imaging (MRI) can seriously interfere with clinical diagnosis. Removing motion artifacts is a straightforward solution and has been extensively studied. However, paired data are still heavily relied on in recent works and the perturbations in k-space (frequency domain) are not well considered, which limits their applications in the clinical field. To address these issues, we propose a novel unsupervised purification method which leverages pixel-frequency information of noisy MRI images to guide a pre-trained diffusion model to recover clean MRI images. Specifically, considering that motion artifacts are mainly concentrated in high-frequency components in k-space, we utilize the low-frequency components as the guide to ensure correct tissue textures. Additionally, given that high-frequency and pixel information are helpful for recovering shape and detail textures, we design alternate complementary masks to simultaneously destroy the artifact structure and exploit useful information. Quantitative experiments are performed on datasets from different tissues and show that our method achieves superior performance on several metrics. Qualitative evaluations with radiologists also show that our method provides better clinical feedback.

Electroencephalogram (EEG) signals have attracted significant attention from researchers due to their non-invasive nature and high temporal sensitivity in decoding visual stimuli. However, most recent studies have focused solely on the relationship between EEG and image data pairs, neglecting the valuable "beyond-image-modality" information embedded in EEG signals. This results in the loss of critical multimodal information in EEG. To address the limitation, this paper proposes a unified framework that fully leverages multimodal data to represent EEG signals, named CognitionCapturer. Specifically, CognitionCapturer trains modality expert encoders for each modality to extract cross-modal information from the EEG modality. Then, it introduces a diffusion prior to map the EEG embedding space to the CLIP embedding space, followed by using a pretrained generative model, the proposed framework can reconstruct visual stimuli with high semantic and structural fidelity. Notably, the framework does not require any fine-tuning of the generative models and can be extended to incorporate more modalities. Through extensive experiments, we demonstrate that CognitionCapturer outperforms state-of-the-art methods both qualitatively and quantitatively.

Adversarial fine-tuning methods enhance adversarial robustness via fine-tuning the pre-trained model in an adversarial training manner. However, we identify that some specific latent features of adversarial samples are confused by adversarial perturbation and lead to an unexpectedly increasing gap between features in the last hidden layer of natural and adversarial samples. To address this issue, we propose a disentanglement-based approach to explicitly model and further remove the specific latent features. We introduce a feature disentangler to separate out the specific latent features from the features of the adversarial samples, thereby boosting robustness by eliminating the specific latent features. Besides, we align clean features in the pre-trained model with features of adversarial samples in the fine-tuned model, to benefit from the intrinsic features of natural samples. Empirical evaluations on three benchmark datasets demonstrate that our approach surpasses existing adversarial fine-tuning methods and adversarial training baselines.

Single domain generalization (SDG) aims to learn a robust model, which could perform well on many unseen domains while there is only one single domain available for training. One of the promising directions for achieving single-domain generalization is to generate out-of-domain (OOD) training data through data augmentation or image generation. Given the rapid advancements in AI-generated content (AIGC), this paper is the first to propose leveraging powerful pre-trained text-to-image (T2I) foundation models to create the training data. However, manually designing textual prompts to generate images for all possible domains is often impractical, and some domain characteristics may be too abstract to describe with words. To address these challenges, we propose a novel Progressive Adversarial Prompt Tuning (PAPT) framework for pre-trained diffusion models. Instead of relying on static textual domains, our approach learns two sets of abstract prompts as conditions for the diffusion model: one that captures domain-invariant category information and another that models domain-specific styles. This adversarial learning mechanism enables the T2I model to generate images in various domain styles while preserving key categorical features. Extensive experiments demonstrate the effectiveness of the proposed method, achieving superior performances to state-of-the-art single-domain generalization approaches.

Infrared Small Target Detection (IRSTD) aims to identify low signal-to-noise ratio small targets in infrared images with complex backgrounds, which is crucial for various applications. However, existing IRSTD methods typically rely solely on image modalities for processing, which fail to fully capture contextual information, leading to limited detection accuracy and adaptability in complex environments. Inspired by vision-language models, this paper proposes a novel framework, SAIST, which integrates textual information with image modalities to enhance IRSTD performance. The framework consists of two main components: Scene Recognition Contrastive Language-Image Pretraining (SR-CLIP) and CLIP-guided Segment Anything Model (CG-SAM). SR-CLIP generates a set of visual descriptions through object-object similarity and object-scene relevance, embedding them into learnable prompts to refine the textual description set. This reduces the domain gap between vision and language, generating precise textual and visual prompts. CG-SAM utilizes the prompts generated by SR-CLIP to accurately guide the Mask Decoder in learning prior knowledge of background features, while incorporating infrared imaging equations to improve small target recognition in complex backgrounds and significantly reduce the false alarm rate. Additionally, this paper introduces the first multimodal IRSTD dataset, MIRSTD, which contains abundant image-text pairs. Experimental results demonstrate that the proposed SAIST method outperforms existing state-of-the-art approaches.

The emphasis on no-reference image quality assessment has often overshadowed the significance of Full-Reference Image Quality Assessment (FR-IQA), which generally better reflects human contrastive perception mechanism. However, FRIQA presents challenges in obtaining content-aligned reference images. To tackle these issues, a novel Bidirectional Reference Image Quality Assessment (BRIQA) method is proposed, centering on leveraging bidirectional reference images and content-quality correlation modeling. First, triplets of content-aligned low-quality and content-non-aligned high-quality reference images are generated using two easily accessible approaches. To prevent the extraction of redundant information, two feature extractors pretrained through unsupervised contrastive learning are utilized to independently extract content and quality features for the triplet images. Then, an attention-mixer is introduced to further mine quality difference information and enhance content feature. Finally, a content-quality correlation modeler is proposed to model the relationship between quality differences and visual contents. Experimental results on benchmark datasets demonstrate that the BRIQA outperforms existing state-of-the-art methods.

With the advancement of AI-generated content technologies, AI-generated images (AGIs) have become increasingly influential in artistic creation and visual communication. However, the aesthetic quality of AGIs varies significantly due to technical limitations and the influence of user input, underscoring the urgent need for systematic aesthetic evaluation of AGIs. In addition, it is difficult to ensure the consistency of text-to-image, which compresses the application space of AGIs. To address these issues, a fine-grained dataset for Aesthetic and Alignment evaluation of AGIs (AGIAA-2K) is presented. This dataset contains 2,064 images generated using 172 well-designed prompts across six different AGI models, with each image annotated based on the subjective experiment. Then, the rationality of the dataset is verified by data analysis. Finally, the performances of the existing algorithms are evaluated in terms of image aesthetic assessment and text-to-image alignment of AGIs. The results demonstrate that these algorithms cannot effectively evaluate these two aspects. The AGIAA-2K is available at https://github.com/BoHu90/AGIAA-2K.

Wide-angle video is favored for its wide viewing angle and ability to capture a large area of scenery, making it an ideal choice for sports and adventure recording. However, wide-angle video is prone to deformation, exposure and other distortions, resulting in poor video quality and affecting the perception and experience, which may seriously hinder its application in fields such as competitive sports. Up to now, few explorations focus on the quality assessment issue of wide-angle video. This deficiency primarily stems from the absence of a specialized dataset for wide-angle videos. To bridge this gap, we construct the first Multi-annotated and multi-modal Wide-angle Video quality assessment (MWV) dataset. Then, the performances of state-of-the-art video quality methods on the MWV dataset are investigated by inter-dataset testing and intra-dataset testing. Experimental results show that these methods impose significant limitations on their applicability.

Image quality assessment (IQA) of artificial intelligence-generated content (AIGC) has recently attracted significant research attention. Unlike general-purpose IQA, which primarily focuses on evaluating image content, AIGCIQA often requires addressing both the Text-to-Image (T2I) correspondence and the perceptual quality of images. To address this requirement, this paper proposes a novel two-stage AIGCIQA method. The first stage evaluates the alignment of the AI-generated images (AIGIs) with their corresponding descriptions, serving as an indicator of overall image quality. Specifically, positive and negative prompts are constructed to describe the T2I correspondence degree, and then a CLIP model is employed to predict the degree based on these image-prompt pairs. The second stage refines the perceptual quality assessment by integrating both global and local degradation features of AIGIs. Importantly, the contribution of local features is measured according to their correlation with the overall image, ensuring key regions are adequately represented in the quality prediction. Experimental results on AGIQA-1K, AGIQA-3K, and AIGCIQA2023 demonstrate the superior performance of the proposed method.

Image fusion combines images from multiple domains into one image, containing complementary information from source domains. Existing methods take pixel intensity, texture and high-level vision task information as the standards to determine preservation of information, lacking enhancement for human perception. We introduce an image fusion method, Hierarchical Perception Fusion (HPFusion), which leverages Large Vision-Language Model to incorporate hierarchical human semantic priors, preserving complementary information that satisfies human visual system. We propose multiple questions that humans focus on when viewing an image pair, and answers are generated via the Large Vision-Language Model according to images. The texts of answers are encoded into the fusion network, and the optimization also aims to guide the human semantic distribution of the fused image more similarly to source images, exploring complementary information within the human perception domain. Extensive experiments demonstrate our HPFusoin can achieve high-quality fusion results both for information preservation and human visual enhancement Our code is available at https://github.com/SSyangguangZHPFusion.

User-generated content (UGC) videos often exhibit complex distortions and diverse content, posing significant challenges for traditional video quality assessment (VQA) methods. Approaches that directly merge distortion and semantic information risk feature conflicts and the loss of details. In addition, simple concatenation of spatiotemporal features fails to capture vital interactions, limiting predictive accuracy. Motivated by these challenges, this paper proposes a Multi-level Distortion Adaptation and Spatiotemporal Cross-Attention Fusion framework for VQA, named MACA-VQA. Specifically, a novel multi-level adaptive strategy progressively incorporates distortion information into each Transformer layer of the CLIP model, enabling layer-wise fusion of semantic and distortion features. Furthermore, a newly introduced cross-attention fusion mechanism dynamically integrates spatiotemporal features, capturing complex, multidimensional interactions. Extensive experiments demonstrate that MACA-VQA achieves state-of-the-art performance on multiple public datasets, validating its effectiveness and robustness in both intra-dataset and inter-dataset scenarios. The source code is available at https://github.com/BoHu90/MACA-VQA

Deep point cloud registration methods encounter challenges due to partial overlaps and are heavily reliant on labeled data. In this paper, we propose ALCReg, an active label correction method for partial point cloud registration learning. ALCReg utilises a multimodal approach to generate pseudo labels, mitigating the cold-start issue in active learning. To ensure the diversity and representativeness of selected samples, we propose an inlier ratio based query strategy for manual correction. Furthermore, an innovative self-correction mechanism based on consistency is introduced, allowing the model to refine pseudo labels autonomously and further improve model performance. Experimental results on the 3DMatch and 3DLoMatch datasets demonstrate that ALCReg achieves comparable performance with the fully-supervised registration methods, even with only 5% of labeled samples, making it the first active learning method tailored for partial point cloud registration. Code is available at https://github.com/Jiang0903/ALCReg.

Prompt learning is a cutting-edge parameter-efficient fine-tuning technique for pre-trained vision-language models (VLMs). Instead of learning a single text prompt, recent works have revealed that learning diverse text prompts can effectively boost the performances on downstream tasks, as the diverse prompted text features can comprehensively depict the visual concepts from different perspectives. However, diverse prompt learning demands enormous computational resources. This efficiency issue still remains unexplored. To achieve efficient and diverse prompt learning, this paper proposes a novel Surrogate Prompt Learning (SurPL) framework. Instead of learning diverse text prompts, SurPL directly generates the desired prompted text features via a lightweight Surrogate Feature Generator (SFG), thereby avoiding the complex gradient computation procedure of conventional diverse prompt learning. Concretely, based on a basic prompted text feature, SFG can directly and efficiently generate diverse prompted features according to different pre-defined conditional signals. Extensive experiments indicate the effectiveness of the surrogate prompted text features, and show compelling performances and efficiency of SurPL on various benchmarks.

Diffusion models have transformed generative modeling but suffer from scalability limitations due to computational overhead and inflexible architectures that process all generative stages and tokens uniformly. In this work, we introduce Diff-MoE, a novel framework that combines Diffusion Transformers with Mixture-of-Experts to exploit both temporarily adaptability and spatial flexibility. Our design incorporates expert-specific timestep conditioning, allowing each expert to process different spatial tokens while adapting to the generative stage, to dynamically allocate resources based on both the temporal and spatial characteristics of the generative task. Additionally, we propose a globally-aware feature recalibration mechanism that amplifies the representational capacity of expert modules by dynamically adjusting feature contributions based on input relevance. Extensive experiments on image generation benchmarks demonstrate that Diff-MoE significantly outperforms state-of-the-art methods. Our work demonstrates the potential of integrating diffusion models with expert-based designs, offering a scalable and effective framework for advanced generative modeling.

Existing handwriting recognition methods only focus on learning visual patterns by modeling low-level relationships of adjacent pixels, while overlooking the intrinsic geometric structures of characters. In this paper, we propose a novel graph-enhanced cross-modal mutual learning network GCM to fully process handwritten text images alongside their corresponding geometric graphs, which consists of one shared cross-modal encoder and two parallel inverse decoders. Specifically, the encoder simultaneously extracts visual and geometric information from the cross-modal inputs, and the decoders fuse the multi-modal features for prediction under the guidance of cross-modal fusion. Moreover, two parallel decoders sequentially aggregate cross-modal features in inverse orders (V→G and G→V) but are enhanced through mutual distillation at each time-step, which involves one-to-one knowledge transfer and fully leverages complementary cross-modal information from both directions. Notably, only one branch of GCM is activated in inference, thus avoiding the increase of the model parameters and computation costs for testing. Experiments show that our method outperforms previous state-of-the-art methods on public benchmarks such as IAM, RIMES, and ICDAR-2013 when no extra training data is utilized.

Autonomous exploration of unknown environments is a critical task in robotic search and rescue operations. Recently, hierarchical planning frameworks have gained significant attention for their potential to enhance exploration efficiency. However, most existing approaches struggle with efficient exploration due to two key limitations: (1) neglecting subregion environmental information and (2) inconsistency between local and global paths. To overcome these challenges, we propose an Information Entropy-assisted Hierarchical Planning (IEHP) framework for efficient autonomous exploration. Specifically, we introduce an efficient subregion arrangement method that considers total travel distance, path similarity, and information entropy. Additionally, we propose a globally consistent frontier selection method to minimize redundant local paths, improving alignment between local and global planning. We validate the feasibility and efficiency of our approach through a series of complex simulation scenarios, with experimental results demonstrating the superiority of the proposed method.

Low-light Image Enhancement (LIE) technology adaptively improves brightness while preserving texture details and suppressing noise artifacts, thereby reducing visual degradation caused by insufficient illumination. While deep learning-based image enhancement algorithms have made significant progress, a key gap remains in establishing standardized methods for fairly evaluating and comparing their performance. To bridge this gap, this paper systematically investigates enhanced low-light image quality assessment from both subjective and objective dimensions. First, we introduce a Real-world Low-light Image Enhancement quality assessment dataset (RLIE), which contains 1540 images from 154 scenarios, each with a subjective score given by the subjects. Based on this, we propose a low light enhanced image quality assessment method based on Multi-level Illumination Injection and Hierarchical Discrepancy Perception (MIIHDP). The core idea of this method is to hierarchically inject separated illumination information into the feature extraction process, then tailor the processing of difference information at different scales to obtain a more comprehensive representation. Finally, extensive statistical analyses demonstrate the rationality of the proposed RLIE dataset, and experimental results show the superior performance of the proposed MIIHDP compared with state-of-the-arts. Our dataset and code are released at: https://github.com/BoHu90/RLIE.

Current medical vision-language pre-training models primarily follow two paradigms: report-supervised cross-modal alignment pre-training and reconstruction-based self-supervised pre-training. The former enhances the discriminative power of representations, while the latter facilitates fine-grained representation learning. However, naively combining these two paradigms inherits their inherent limitations: reconstruction-based methods treat all image patches equally during reconstruction, failing to effectively capture critical pathological details-since disease-related regions typically occupy only a small fraction of the image. Meanwhile, alignment-based methods suffer from suboptimal representations due to the presence of false negatives. To address these challenges, we propose a novel pre-training framework that integrates two key components: Pathology-Aware Reconstruction (PAR) and Discriminative Knowledge-Boosted Alignment (DKBA). Through a cascaded training strategy, our framework effectively combines the strengths of both paradigms while mitigating their inherent limitations. During the reconstruction pre-training stage, PAR incorporates pathology-aware priors to enhance the model's ability to capture fine-grained pathological details. In the alignment pre-training stage, DKBA leverages a medical knowledge graph as external supervision to improve cross-modal clustering alignment, thereby reducing the negative impact of false negatives. Extensive experiments on diverse downstream medical imaging tasks including image classification, object detection, and semantic segmentation, demonstrate the superior generalization capabilities of our method. Our code is publicly available at https://github.com/Felix1118/PADKB.

Temporal Sentence Grounding (TSG) aims to identify relevant moments in an untrimmed video that semantically correspond to a given textual query. Despite existing studies having made substantial progress, they often overlook the issue of spurious correlations between video and textual queries. These spurious correlations arise from two primary factors: (1) inherent biases in the textual data, such as frequent co-occurrences of specific verbs or phrases, and (2) the model's tendency to overfit to salient or repetitive patterns in video content. Such biases mislead the model into associating textual cues with incorrect visual moments, resulting in unreliable predictions and poor generalization to out-of-distribution examples. To overcome these limitations, we propose a novel TSG framework, causal intervention and counterfactual reasoning that utilizes causal inference to eliminate spurious correlations and enhance the model's robustness. Specifically, we first formulate the TSG task from a causal perspective with a structural causal model. Then, to address unobserved confounders reflecting textual biases toward specific verbs or phrases, a textual causal intervention is proposed, utilizing do-calculus to estimate the causal effects. Furthermore, visual counterfactual reasoning is performed by constructing a counterfactual scenario that focuses solely on video features, excluding the query and fused multi-modal features. This allows us to debias the model by isolating and removing the influence of the video from the overall effect. Experiments on public datasets demonstrate the superiority of the proposed method. The code is available at https://github.com/Tangkfan/CICR.

While unmanned aerial vehicles (UAVs) offer wide-area, high-altitude coverage for anomaly detection, they face challenges such as dynamic viewpoints, scale variations, and complex scenes. Existing datasets and methods, mainly designed for fixed ground-level views, struggle to adapt to these conditions, leading to significant performance drops in drone-view scenarios.To bridge this gap, we introduce A2Seek (Aerial Anomaly Seek), a large-scale, reasoning-centric benchmark dataset for aerial anomaly understanding. This dataset covers various scenarios and environmental conditions, providing high-resolution real-world aerial videos with detailed annotations, including anomaly categories, frame-level timestamps, region-level bounding boxes, and natural language explanations for causal reasoning. Building on this dataset, we propose A2Seek-R1, a novel reasoning framework that generalizes R1-style strategies to aerial anomaly understanding, enabling a deeper understanding of “Where” anomalies occur and “Why” they happen in aerial frames.To this end, A2Seek-R1 first employs a graph-of-thought (GoT)-guided supervised fine-tuning approach to activate the model's latent reasoning capabilities on A2Seek. Then, we introduce Aerial Group Relative Policy Optimization (A-GRPO) to design rule-based reward functions tailored to aerial scenarios. Furthermore, we propose a novel “seeking” mechanism that simulates UAV flight behavior by directing the model's attention to informative regions.Extensive experiments demonstrate that A2Seek-R1 achieves up to a 22.04\% improvement in AP for prediction accuracy and a 13.9\% gain in mIoU for anomaly localization, exhibiting strong generalization across complex environments and out-of-distribution scenarios. Our dataset and code are released at https://2-mo.github.io/A2Seek/.

Advances in image generation technologies have raised growing concerns about their potential misuse, particularly in producing misinformation and deepfakes. This creates an urgent demand for effective methods to detect AI-generated images (AIGIs). While progress has been made, achieving reliable performance across diverse generative models and scenarios remains challenging due to the absence of source-invariant features and the limited generalization of existing approaches. In this study, we investigate the potential of using image entropy as a discriminative cue for AIGI detection and propose Multi-granularity Local Entropy Patterns (MLEP), a set of feature maps computed based on Shannon entropy from shuffled small patches at multiple image scales. MLEP effectively captures pixel dependencies across scales and dimensions while disrupting semantic content, thereby reducing potential content bias. Based on MLEP, we can easily build a robust CNN-based classifier capable of detecting AIGIs with enhanced reliability. Extensive experiments in an open-world setting, involving images synthesized by 32 distinct generative models, demonstrate that our approach achieves substantial improvements over state-of-the-art methods in both accuracy and generalization. Our code and models are available at https://www.github.com/fkeufss/MLEP/.

Face super-resolution (FSR) is critical for bolstering intelligent security in Internet of Things (IoT) systems. Recent deep learning-driven FSR algorithms have attained remarkable progress. However, they always require separate model training and optimization for each scaling factor or input resolution, leading to inefficiency and impracticality. To overcome these limitations, we propose SAFNet, an innovative framework tailored for scale-adaptive FSR with arbitrary input resolution. SAFNet integrates scale information into representation learning to enable adaptive feature extraction and introduces dual-embedding attention to boost adaptive feature reconstruction. It leverages facial self-similarity and spatial-frequency collaboration to achieve precise scale-aware SR representations. This is attained through three key modules: 1) the scale adaption guidance unit (SAGU); 2) the scale-aware nonlocal self-similarity (SNLS) module; and 3) the spatial-frequency interactive modulation (SFIM) module. SAGU imports scaling factors using frequency encoding, SNLS exploits self-similarity to enrich feature representations, and SFIM incorporates spatial and frequency information to predict target pixel values adaptively. Comprehensive evaluations across four benchmark datasets reveal that SAFNet outperforms the second-best compared state-of-the-art (SOTA) method by about 0.2 dB/0.007 in PSNR/SSIM ($\times 4$on CelebA) with reduced 18.68%/42.64% computational complexity/time cost. This demonstrates SAFNet’s effectiveness and superiority, showcasing its potential as a promising solution for scale and input resolution adaptation challenges in FSR. The code will be available athttps://github.com/ICVIPLab/SAFNet.

Although numerous clustering algorithms have been developed, many existing methods still rely on the K-means technique to identify clusters of data points. However, the performance of K-means is highly dependent on the accurate estimation of cluster centers, which is challenging to achieve optimally. Furthermore, it struggles to handle linearly non-separable data. To address these limitations, from the perspective of manifold learning, we reformulate multi-view K-means into a manifold-based multi-view clustering formulation that eliminates the need for computing centroid matrix. This reformulation ensures consistency between the manifold structure and the data labels. Building on this, we propose a novel multi-view K-means model incorporating the tensor rank constraint. Our model employs the indicator matrices from different views to construct a third-order tensor, whose rank is minimized via the tensor Schatten p-norm. This approach effectively leverages the complementary information across views. By utilizing different distance functions, our proposed model can effectively handle linearly non-separable data. Extensive experimental results on multiple databases demonstrate the superiority of our proposed model.

This paper targets to learn multiple comprehensive text prompts that can describe the visual concepts from coarse to fine, thereby endowing pre-trained VLMs with better transfer ability to various downstream tasks. We focus on exploring this idea on transformer-based VLMs since this kind of architecture achieves more compelling performances than CNN-based ones. Unfortunately, unlike CNNs, the transformer-based visual encoder of pre-trained VLMs cannot naturally provide discriminative and representative local visual information. To solve this problem, we propose Frequency-based Comprehensive Prompt Learning (FCPrompt) to excavate representative local visual information from the redundant output features of the visual encoder. FCPrompt transforms these features into frequency domain via Discrete Cosine Transform (DCT). Taking the advantages of energy concentration and information orthogonality of DCT, we can obtain compact, informative and disentangled local visual information by leveraging specific frequency components of the transformed frequency features. To better fit with transformer architectures, FCPrompt further adopts and optimizes different text prompts to respectively align with the global and frequency-based local visual information via a dual-branch framework. Finally, the learned text prompts can thus describe the entire visual concepts from coarse to fine comprehensively. Extensive experiments indicate that FCPrompt achieves the state-of-the-art performances on various benchmarks.

In the current scenario, a vast amount of unlabeled high-dimensional data exhibits intrinsic relationships, making it suitable for information extraction through graph-based clustering methods. However, these datasets often lack edge structure information and contain numerous irrelevant features. To address these challenges, we propose a comprehensive solution that involves: (1) applying a feature weighting approach to manage features, (2) constructing edges based on weighted granular-balls, and (3) integrating graph convolutional networks (GCNs) with edge generation to develop an autoencoder network. Our method significantly enhances the extraction of relevant information from high-dimensional, unlabeled data, improving the overall performance and reliability of the clustering process. Extensive experimental results demonstrate that our model, AW-GBGAE, excels in clustering tasks and exhibits strong competitiveness compared to baseline models. The code is publicly available at https://github.com/xjnine/AWGBGAE.

Adversarial training has been proposed and widely recognized as a very effective method to defend against adversarial noise. However, the label flipping pattern on different classes still need deeper exploration to identify potential problems and assist in further enhancing robustness. In this work, we model the class-flipping distribution via statistical investigations and find this distribution reveals two shortcomings: the highly misleading category is present in the model's predictions for data in each class, and the trend in class flipping are significantly different across classes. Based on these observations, we propose a Class-Flipping-aware Adversarial Training (CFAT) method. On the one hand, we obtain the most misleading categories for the data in each class by counting the samples flipped to different wrong categories, and utilize them as the target to construct corresponding targeted adversarial samples, respectively. On the other hand, we take the proportions of samples flipped to the most misleading category as factors to scale the perturbation budgets of adversarial training samples for the data with corresponding classes. Experimental results on datasets with different class number validate the effectiveness of the proposed method.

As deepfake technologies continue to evolve, proactive defense techniques have gained increasing attention for their potential to either neutralize deepfake operations or simplify detection through pre-embedded signals. In this paper, inspired by watermark-based forensic methods, we explore a novel detection framework built on the concept of “hiding a learnable face within a face”. Specifically, we use a semi-fragile invertible steganography network to imperceptibly embed a learnable template image within a host face image to be protected. This template serves as an indicator revealing signs of tampering when recovered through the inverse steganography process. Unlike manually designed templates, it is optimized during training to resemble a neutral facial appearance—functioning like a subtle “big brother” hidden within the image. Through a self-blending mechanism and robustness learning strategy with a simulated transmission channel, we develop a robust detector that accurately distinguishes between malicious tampering and benign processing of the steganographic image. Extensive experiments across multiple datasets validate the superiority of the proposed approach over competing passive and proactive detection methods.

Abstract Detecting fake news early is challenging due to the absence of labeled articles for emerging events in training data. To address this, we propose a Disentangled Event-Agnostic Representation (DEAR) learning approach. Our method begins with a BERT-based adaptive multi-grained semantic encoder that captures hierarchical and comprehensive textual representations of the input news content. To effectively separate latent authenticity-related and event-specific knowledge within the news content, we employ a disentanglement architecture. To further enhance the decoupling effect, we introduce a cross-perturbation mechanism that perturbs authenticity-related representation with the event-specific one, and vice versa, deriving a robust and discerning authenticity-related signal. Additionally, we implement a refinement learning scheme to minimize potential interactions between two decoupled representations, ensuring that the authenticity signal remains strong and unaffected by event-specific details. Experimental results demonstrate that our approach effectively mitigates the impact of event-specific influence, outperforming state-of-the-art methods. In particular, it achieves a 6.0% improvement in accuracy on the PHEME dataset over MDDA, a similar approach that decouples latent content and style knowledge, in scenarios involving articles from unseen events different from the topics of the training set.

Catastrophic forgetting is the crucial challenge for continual learning. One of the state-of-the-art approaches is the orthogonal projection, which aims to learn each task by updating model parameters in the direction orthogonal to the subspace spanned by the previous task input. Although such strict orthogonal weight constraints ensure no interference with tasks that have been learned to achieve model stability, they greatly sacrifice model plasticity. In this paper, we propose an adaptive balanced orthogonal projection (AdaBOP) method, to search for the optimal network parameter updating direction to address the plasticity-stability dilemma in continual learning. The proposed AdaBOP method can adaptively adjust its tendency towards plasticity-stability trade-off based on the layer-wise feature space correlations of the model between old and new tasks. To further improve the training efficiency, we also implement the AdaBOP method in the uncentered covariance matrix space of the previous tasks, and finally achieve a better stability-plasticity trade-off in continual learning efficiently. Experimental results greatly demonstrate the effectiveness of the proposed method, which achieves superior performances to state-of-the-art continual learning approaches. The code is available athttps://github.com/hyscn/AdaBOP.

Transductive zero-shot learning (TZSL) has been proposed to address the domain shift problem by leveraging additional unlabeled unseen data to enhance the generalization ability from seen classes to unseen target classes. Existing TZSL methods primarily focus on mitigating the distribution bias problem by incorporating these unlabeled samples into the generative models. Although these methods have achieved great success, they do not fully exploit the potential of these unlabeled target data. In this paper, we propose a bidirectional weakly guided conditional generative modeling approach, which utilizes the attribute regressor and the visual generator to synthesize paired training data of unseen classes for each other, thus converting unlabeled target data into matched feature-attribute pairs. Additionally, on top of the generative modeling, we also propose to progressively estimate the associations between visual features and attributes among the unlabeled target data through a semi-supervised pseudo-labeling approach, so as to further facilitate the generative model and enhance the learning of target distributions. Extensive experimental results on four benchmark datasets demonstrate the effectiveness of the proposed method, achieving superior performances to state-of-the-art methods. Our source code is released in https://github.com/LevisWei/semi-zero-master.

Current deepfake detection methods commonly use data augmentation and authenticity-content disentanglement to extract more generalized features for detection tasks. However, these methods rely exclusively on low-level spatial artifacts to distinguish real from fake images, which presents significant challenges in accurately capturing the rich forgery cues. Deepfakes create discrepancies between forged and original facial features within the face-recognition (FR) embedding space, which can serve as an additional cue for detection. To better exploit the artifacts in deepfake images, we propose a novel detection method that enhances the detector’s perception capability by incorporating not only the real and fake samples during training, but also the visual residual between real and fake images. Meanwhile, we integrate the discrepancy in facial embedding between the real and fake samples into the training procedure of artifact extraction, serving as a guidance signal with strong knowledge provided by the pretrained face recognition model. Specialized distillation loss along with additional cross-entropy losses are designed to enhance the detection capability. Experiments on multiple benchmarks demonstrate the superiority of the proposed approach in deepfake detection over literature methods.

Recently, unsupervised face super-resolution (FSR) has attracted significant attention due to its remarkable generalization performance. However, existing methods neglect the incorporation of facial priors, which can effectively guide the restoration of face images. The root cause of this issue lies in the significant challenges associated with incorporating facial priors into unsupervised frameworks. First, unsupervised methods often face the challenge of real-world low-quality (LQ) images that are severely corrupted, making it unrealistic to extract reliable prior information from them. Second, the estimation of facial priors exponentially increases the model’s parameters and computational complexity, contradicting the purpose of unsupervised methods for practical deployment. In this work, we fundamentally address the aforementioned challenges and proposeFaith3D-FSR, a novel approach that incorporates faithful 3D facial priors into unsupervised FSR. Specifically, we introduceFaith3Dmechanism for faithful prior integration, which deconstructs super-resolution images into 3D elements and uses the 3D priors from real high-quality (HQ) images as reference for calibration solely during the training phase. This strategy enables more precise guidance on the super-resolution in a high-dimensional space, without requiring additional prior estimation during inference. It successfully overcomes the aforementioned challenges, making it more suitable for real-world applications, and offers a plug-and-play solution for incorporating 3D priors into unsupervised FSR. Extensive experiments demonstrate that our approach achieves state-of-the-art (SOTA) performance on multiple benchmark datasets and across a range of evaluation metrics. The code is available here.

Wide-angle videos shot with short-focus lenses often exhibit deformation distortions, which poses significant challenges for video quality assessment (VQA). Although current VQA methods focus primarily on video content and distortion perception, there has been little explicit research on the impact of deformation characteristics on the perception of wide-angle video quality. To this end, this paper makes the first attempt to construct a novel wide-angle video quality assessment method based on deformation representation learning and multi-dimensional feature fusion, termed DRLMF. Specifically, we first analyze the deformation distribution characteristics of wide-angle videos based on the deformation camera model. Based on this, a three-stream video perception and assessment network is proposed. The first branch extracts global semantics using the image encoder of CLIP. The second branch introduces an effective deformation region selection strategy and proposes an interpretable deformation representation learning module. This module leverages the perception advantages of convolutional neural networks (CNNs) in local distortions and considers the correlation between patch size and distortion perception. The third branch extracts motion features using an action recognition network. Finally, an effective multi-dimensional feature fusion module is proposed to integrate more refined and richer semantic, deformation, and motion features. Extensive experiments on wide-angle VQA datasets and standard video datasets show that the DRLMF outperforms the state-of-the-arts in terms of prediction monotonicity and accuracy. The codes will be available at https://github.com/BoHu90/DRLMF.

Current few-shot learning techniques predominantly leverage amortization techniques based on meta-learning frameworks, which effectively adapt to unknown tasks with limited examples. However, these approaches face significant challenges in cross-domain scenarios, where the data distributions between the source domain (training data) and the target domain (testing data) differ substantially. This domain shift can lead to models that overfit the global discriminative model while underfitting their local amortization on the adaptable few-shot structure. To mitigate this problem, our proposal makes an upgrade on Conditional Neural Adaptive Processes, reformulating its conditioning mechanism to better handle cross-domain adaptation. This results in calibrated amortization of task-specific feature extractors and the construction of a robust non-parametric classifier. In our implementation, we first employ generative modeling or deterministic self-attention to all labeled context features, establishing a strong task-level alignment that adapts the extractor across domains. Additionally, we introduce a novel channel-wise normalization to further enhance the adaptation process. Our experiments on the Meta-dataset benchmark demonstrate an average$6.9\sim 9$% improvement in out-of-distribution tasks, underscoring the effectiveness of exploiting calibrated adaptation in few-shot cross-domain classification.

Sketch-based image retrieval (SBIR) relies on free-hand sketches to retrieve natural photos within the same class. However, its practical application is limited by its inability to retrieve classes absent from the training set. To address this limitation, the task has evolved into Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR), where model performance is evaluated on unseen categories. Traditional SBIR primarily focuses on narrowing the domain gap between photo and sketch modalities. However, in the zero-shot setting, the model not only needs to address this cross-modal discrepancy but also requires a strong generalization capability to transfer knowledge to unseen categories. To this end, we propose a novel framework for ZS-SBIR that employs a pair-based relation-aware quadruplet loss to bridge feature gaps. By incorporating two negative samples from different modalities, the approach prevents positive features from becoming disproportionately distant from one modality while remaining close to another, thus enhancing inter-class separability. We also propose a Relation-Aware Meta-Learning Network (RAMLN) to obtain the margin, a hyper-parameter of cross-modal quadruplet loss, to improve the generalization ability of the model. RAMLN leverages external memory to store feature information, which it utilizes to assign optimal margin values. Experimental results obtained on the extended Sketchy and TU-Berlin datasets show a sharp improvement over existing state-of-the-art methods in ZS-SBIR.

Near infrared-visible (NIR-VIS) heterogeneous face recognition aims to match face identities in cross-modality settings, which has achieved significant development recently. The work on adversarial attack and security issues of the heterogeneous face recognition task is still lacking. Existing adversarial face generation methods can’t deploy directly because of the inevitable large modality discrepancy. Besides, the ideal adversarial attacking generated images should maintain both high capabilities and low detectability. Considering the properties of near-infrared face images, our basic idea is to construct adversarial shadows for good stealthiness and high attack capability. In this paper, we propose a novel face adversarial shadow generation framework for NIR-VIS heterogeneous face recognition, which can synthesize fine-crafted lighting conditions containing strong identity attacking ability. Specifically, we design the variance consistency-based symmetric face attacking loss to improve the attacking generalization and the synthesized image quality. Extensive qualitative and quantitative experiments on the public large-scale NIR-VIS heterogeneous face dataset prove the proposed method achieves superior performance compared with the state-of-the-art methods. The source code is publicly available athttps://github.com/GEaMU/Devil-in-Shadow.

Diffusion models show great potential in solving inverse problems, including MRI reconstruction. With its unique characteristics, medical imaging demands both efficiency and accuracy in the reconstruction process. However, existing MRI reconstruction methods based on diffusion models often fall short of fully leveraging the available measurements during sampling. Consequently, these methods suffer from compromised reconstruction quality and elevated bias, especially when dealing with large acceleration factors. In response to these challenges, we propose Dual Manifold Constraints (DMC), a fast MRI reconstruction method based on diffusion models. We treat the sampling process as a combination of denoising and adding noise processes, and we constrain these two processes using both pristine measurements and their noisy counterparts to adapt to the geometry of diffusion. It’s worth noting that we propose a method to estimate the noisy measurement that satisfies the sub-sampling process to maintain the current data manifold when performing data consistency constraints. Experimental results show that our method outperforms the latest diffusion-based methods regarding both reconstruction speed and accuracy, and exhibits strong out-of-distribution generalization performance.

Semi-supervised learning (SSL) has proven effective in assigning a pseudo-label to a confident sample whose largest class probability is above a fixed threshold. However, in the context of semi-supervised facial attribute recognition (SSFAR), where a sample is associated with multiple presence and absence pseudo-labels, directly applying existing SSL methods is challenging due to two issues: 1) the lack of a clear boundary between presence and absence predictions for an attribute makes it difficult to distinguish them using a single threshold; 2) the learning difficulty varies across attributes, so the fixed strategy fails to adaptively learn different attributes. To address these challenges, we propose Dynamic thrEShold Pairs (DESP), a simple yet effective method to handle the SSFAR problem. Specifically, during each training stage, we derive two sets for each attribute from labeled samples, which contain the predicted probabilities of presence and absence, respectively. We then compute the mid-ranges of the two sets as paired presence and absence thresholds. Finally, we assign a presence or absence pseudo-label for the attribute to an unlabeled sample when its prediction exceeds the presence threshold or falls below the absence threshold. Extensive experiments on the CelebA and LFWA datasets demonstrate that DESP achieves superior performance compared to state-of-the-art methods, especially in the case of scarce labeled samples. Also, DESP performs well on multi-label datasets such as Pascal VOC and MS-COCO. The code will be publicly available athttps://github.com/yihanxxu/DESP.

We propose CatVersion, an inversion-based method that learns the personalized concept through a handful of examples. Subsequently, users can utilize text prompts to generate images that embody the personalized concept, thereby achieving text-to-image personalization. In contrast to existing approaches that emphasize word embedding learning or parameter fine-tuning for the diffusion model, which potentially causes concept dilution or overfitting, our method concatenates embeddings on the feature-dense space of the text encoder in the diffusion model to learn the gap between the personalized concept and its base class, aiming to maximize the preservation of prior knowledge in diffusion models while restoring the personalized concepts. To this end, we first dissect the text encoder’s integration in the image generation process to identify the feature-dense space of the encoder. Afterward, we concatenate embeddings on the Keys and Values in this space to learn the gap between the personalized concept and its base class. In this way, the concatenated embeddings ultimately manifest as a residual on the original attention output. To more accurately and unbiasedly quantify the results of personalized image generation, we improve the CLIP image alignment score based on masks. Qualitatively and quantitatively, CatVersion helps to restore personalization concepts more faithfully and enables more robust editing.

Existing active learning methods for object detection face challenges, such as the lack of ground truth labels for regression loss, insufficient representation of unlabeled instance samples information, and discrepancies in information quality between image-level and multiple anchor-level instances. To address these issues, we propose an active learning method for object detection with vectorized dual pseudo loss and multiple instance offset constraint. This method implements a two-stage framework. The first stage focuses on evaluating the information quality of detection images. We first pioneer a dual pseudo loss formulation that provides theoretically grounded regression loss estimation. The regression loss is calculated as the norm of the offset discrepancy loss vector between the enhanced and original base box vector, further constrained by the cosine value of the angle between the anchor box feature and regressor parameters vector. The distance entropy from the base box feature vector to each category's feature prototype vector is used as a weighting factor for the regression and classification information quality of instance samples. Subsequently, the second stage employs diversity-driven sampling on high-information images, leveraging instance-level cosine similarity to effectively remove redundant images. The proposed method outperforms state-of-the-art active learning approaches for object detection on PASCAL VOC and MS COCO datasets. Additionally, the proposed dual pseudo regression loss robustly captures regression information quality, demonstrating its effectiveness for active learning in object detection.

Malicious applications of visual manipulation have raised serious threats to the security and reputation of users in many fields. To alleviate these issues, adversarial noise-based defenses have been enthusiastically studied in recent years. However, “data-only” methods tend to distort fake samples in the low-level feature space rather than the high-level semantic space, leading to limitations in resisting malicious manipulation. Frontier research has shown that integrating knowledge in deep learning can produce reliable and generalizable solutions. Inspired by these, we propose aknowledge-guided adversarial defense (KGAD) to actively force malicious manipulation models to output semantically confusing samples. Specifically, in the process of generating protective adversarial noise, we focus on constructing significant semantic confusions at the domain-specific knowledge level, and exploit a metric closely related to visual perception to replace the general pixel-wise metrics. The generated adversarial noise can actively interfere with the malicious manipulation model by triggering knowledge-guided and perception-related disruptions in the fake samples. To validate the effectiveness of the proposed method, we conduct qualitative and quantitative experiments on human perception and visual quality assessment. The results on two different tasks both show our defense achieves competitive performances and generalizability, indicating that it can effectively resist malicious visual manipulation.

Object detection in remote sensing images is crucial in numerous fields; however, it becomes highly challenging under adverse weather circumstances. Given that previous remote sensing image object detection methods were designed based on normal weather conditions and ideal datasets, they are not beneficial for detection under real-world haze conditions and with class-imbalanced data. In this work, an adaptive dehazing centroid contrastive network (DHC-Net) is proposed to address the aforementioned issues. This network consists of an adaptive dehazing module and a centroid-guided contrastive learning approach. The adaptive dehazing module learns the image content to generate adaptive dehazing parameters, thus alleviating the influence of haze on the quality of remote sensing images. The centroid-guided contrastive learning approach is particularly designed to address the issue of imbalanced datasets. Integrating centroid vectors with actual samples in each training batch guarantees that each class is sampled at least once, effectively preventing the undersampling of minority classes. Moreover, dynamic weighted sampling based on prediction confidence guides the model to give priority to smaller classes, remarkably improving its ability to handle imbalanced data. Extensive experiments on the DOTA-v2.0, DOTA-v2.0Haze, RTTS, and HazeNet datasets demonstrate that DHC-Net is outstanding in handling haze conditions in remote sensing data, substantially enhancing target detection accuracy, even in the presence of imbalanced object classes. The source code will be available athttps://github.com/Linghuaqian1/DHC_Net

Hyperspectral image (HSI) and light detection and ranging (LiDAR) data joint classification is a challenging task. Existing multisource remote sensing data classification methods often rely on human-designed frameworks for feature extraction, which heavily depend on expert knowledge. To address these limitations, we propose a novel dynamic cross-modal feature interaction network (DCMNet), the first framework leveraging a dynamic routing mechanism for HSI and LiDAR classification. Specifically, our approach introduces three feature interaction blocks: bilinear spatial attention block (BSAB), bilinear channel attention block (BCAB), and integration convolutional block (ICB). These blocks are designed to effectively enhance spatial, spectral, and discriminative feature interactions. A multilayer routing space with routing gates is designed to determine optimal computational paths, enabling data-dependent feature fusion. Additionally, bilinear attention mechanisms are employed to enhance feature interactions in spatial and channel representations. Extensive experiments on three public HSI and LiDAR datasets demonstrate the superiority of DCMNet over the state-of-the-art methods. Our codes are available athttps://github.com/oucailab/DCMNet.

With the rapid development of remote sensing technology, the range of its applications, such as urban planning and environmental monitoring, has broadened considerably. However, the vast increase in remote sensing data poses challenges for training deep neural networks, primarily due to the extensive labeled datasets required. To address this issue, zero-shot learning (ZSL) emerges as a viable approach, enabling recognition of unseen categories without relying on training samples. In this paper, we introduce a Robust and Discriminative Visual-Semantic Alignment (RDVSA) model, which utilizes attribute-guided contrastive learning for enhanced performance. By incorporating category centroids into the embedding space of visual features and attribute descriptions, our model enhances the discriminative power of learned features and stabilizes the learning process, particularly in the presence of complex data distributions and noise in remote sensing images. Additionally, we employ a Vision Transformer (ViT) and a “mean-teacher" framework to capture both global context and local details, ensuring robustness in occluded or irrelevant attribute regions. Comprehensive experiments on large-scale remote sensing benchmarks reveal that our approach surpasses current state-of-the-art methods in both ZSL and generalized zero-shot learning (GZSL) scenarios. These results underscore the effectiveness of using attribute-based contrastive learning alongside consistency constraints.

Change detection (CD) is one of the prominent research topics in the fields of Earth science and remote sensing. Recently, an increasing number of deep learning-based CD methods have been developed. Most of the current CD methods require lots of pixel-level labels for supervised learning. However, annotating all the changed pixels in bitemporal images is both challenging and time-consuming. In this work, as a first attempt in the field of CD, we propose a novel CD framework, similarity-awareness weakly supervised CD (SWCD) to achieve accurate CD, which uses weakly supervised learning as an auxiliary task to guide the model in both semi-supervised and supervised learning. In the weakly supervised branch (WSB), we incorporate the concept of similarity and introduce similarity information into the supervised branch to guide pixel-level CD learning, thus enhancing feature continuity. Moreover, large kernel convolution attention is introduced to enhance multiscale feature learning. In the supervised branch, we re-evaluate the approach to multiscale feature aggregation and introduce an adaptive feature module to integrate features from both global and local perspectives. Furthermore, our method can serve as a general framework that is compatible with the existing CD approaches. Experimental results on four CD datasets demonstrate the superior effectiveness and generalization of our proposed method. The code is available athttps://github.com/ZijunTan/SWCD.

Prevailing pan-sharpening methods tackle the ill-posed challenge of reconstructing high-resolution multispectral (HRMS) images from low-resolution multispectral (LRMS) and panchromatic (PAN) inputs. This ill-posed nature introduces distortions such as blurred spatial edges and spectral color deviations, which compromise the fidelity of the reconstructed image. To address this, we propose S2CMamba, a dual-branch framework that leverages Mamba’s efficient contextual modeling and enforces spatial and spectral consistency through tailored priors, effectively alleviating the ill-posed nature of the pan-sharpening. The spatial context branch utilizes the Windowed Spatial Local Mamba (WSLM) for local details and the Global Spatial Interaction Mamba (GSIM) for long-range structures. Within WSLM, a Manifold Preservation (MP) constraint is proposed to align HRMS features with the low-dimensional manifold consistency of PAN and LRMS, thereby mitigating high-dimensional distortions and enhancing spatial consistency. Meanwhile, the spectral branch integrates multi-scale feature extraction and designed Spectral Context Mamba (SCM) to capture spectral context. Moreover, the spatial and spectral properties of multispectral images are investigated, and the wavelet transforms are introduced for better accomplish consistency. By incorporating contextual information, S2CMamba reduces the ambiguity of the solution space, while wavelet-based consistency constraints and designed MP prior further alleviate the ill-posed nature. Extensive experiments on benchmark datasets demonstrate that S2CMamba surpasses state-of-the-art methods, validating the efficacy of this approach in addressing the ill-posed nature of pan-sharpening tasks.

Remote sensing (RS) image deblurring and super-resolution (SR) are common tasks in computer vision that aim at restoring RS image detail and spatial scale, respectively. However, real-world RS images often suffer from a complex combination of global low-resolution (LR) degeneration and local blurring degeneration. Although carefully designed deblurring and SR models perform well on these two tasks individually, a unified model that performs jointly RS image deblurring and SR (JRSIDSR) task is still challenging due to the vital dilemma of reconstructing the global and local degeneration simultaneously. In addition, existing methods struggle to capture the interrelationship between deblurring and SR processes, leading to suboptimal results. To tackle these issues, we give a unified theoretical analysis of RS images’ spatial and blur degeneration processes and propose a dual-branch parallel network named adjustable-kernel and multi-domain network (AKMD-Net) for the JRSIDSR task. AKMD-Net consists of two main branches: deblurring and SR branches. In the deblurring branch, we design a pixel-adjustable kernel block (PAKB) to estimate the local and spatial-varying blur kernels. In the SR branch, a multi-domain attention block (MDAB) is proposed to capture the global contextual information enhanced with high-frequency details. Furthermore, we develop an adaptive feature fusion (AFF) module to model the contextual relationships between the deblurring and SR branches. Finally, we design an adaptive Wiener loss (AW Loss) to depress the prior noise in the reconstructed images. Extensive experiments demonstrate that the proposed AKMD-Net achieves state-of-the-art (SOTA) quantitative and qualitative performance on commonly used RS image datasets. The source code is publicly available at:https://github.com/zpc456/AKMD-Net.

Multitemporal hyperspectral images (HSIs) have been widely applied in change detection (CD) of different land covers for their rich spectral features and image details. However, alignment and labeling pairs of bitemporal HSIs are labor-intensive. In this article, we propose a single-temporal mask-based network (STMNet) for self-supervised HSI CD from a new perspective of detecting masks as changes. STMNet implements self-supervised by treating artificially constructed masks attached to single-temporal HSI as changed regions. To this end, we design a multiscale mask change simulation (MMCS) strategy to generate pseudo-second-temporal HSI closer to the real case. Meanwhile, a global-local feature aggregation network is proposed to enhance long-distance and local spatial-spectral feature extraction. To the best of our knowledge, this is the first work in the field of HSI CD that uses single-temporal HSIs and eliminates the need for labeling and pairing samples, alleviating the problem of difficult multitemporal HSI annotation. The visual and quantitative experimental results on three HSI datasets show that the proposed STMNet outperforms the compared state-of-the-art methods for HSI CD. Codes are available athttps://github.com/Zhoutya/ChangeDetection-STMNet.

Video-based Visible-Infrared Person Re-Identification (VVI-ReID) aims to match pedestrian sequences across modalities by extracting modality-invariant sequence-level features. As a high-level semantic representation, language provides a consistent description of pedestrian characteristics in both infrared and visible modalities. Leveraging the Contrastive Language-Image Pre-training (CLIP) model to generate video-level language prompts and guide the learning of modality-invariant sequence-level features is theoretically feasible. However, the challenge of generating and utilizing modality-shared video-level language prompts to address modality gaps remains a critical problem. To address this problem, we propose a simple yet powerful framework, video-level language-driven VVI-ReID (VLD), which consists of two core modules: invariant-modality language prompting (IMLP) and spatial-temporal prompting (STP). IMLP employs a joint fine-tuning strategy for the visual encoder and the prompt learner to effectively generate modality-shared text prompts and align them with visual features from different modalities in CLIP’s multimodal space, thereby mitigating modality differences. Additionally, STP models spatiotemporal information through two submodules, the spatial-temporal hub (STH) and spatial-temporal aggregation (STA), which further enhance IMLP by incorporating spatiotemporal information into text prompts. The STH aggregates and diffuses spatiotemporal information into the [CLS] token of each frame across the vision transformer (ViT) layers, whereas STA introduces dedicated identity-level loss and specialized multihead attention to ensure that the STH focuses on identity-relevant spatiotemporal feature aggregation. The VLD framework achieves state-of-the-art results on two VVI-ReID benchmarks. On the HITSZ-VCM dataset, it improves the Rank-1 accuracy by 7.3% and mAP by 7.6% (infrared-to-visible) and the Rank-1 accuracy by 10.4% and the mAP accuracy by 9.3% (visible to infrared) and requires only 2 hours of training, 2.39M additional parameters, and 0.12G FLOPs. The code will be released at https://github.com/Visuang/VLD.

Deep neural networks (DNNs) have been widely used in remote sensing but demonstrated to be sensitive with adversarial examples. By introducing carefully designed perturbations to clean images, DNNs can be led to incorrect predictions. Adversarial patch is commonly used to conduct adversarial attack, where traditional methods optimize its content and position separately, neglecting the coupling relation of two factors. In this paper, we propose a black-box attack framework targeting fine-grained aircraft recognition, named PatchGen, simultaneously optimizing both content and position of physical adversarial patches. For the requirements of physical attack, we further constrain the patch in object region and utilize elaborate criteria to evaluate its naturalness to alleviate the distortion when applying the patch in real world. We comprehensively validate our method in fine-grained aircraft classification, extending to object detection subsequently. Extensive experiments demonstrate that the proposed method achieves superior attack performance efficiently for classification and detection tasks in digital domain. Moreover, we validate the effectiveness of the adversarial patch under diverse circumstances in the physical world and prove that our method can be applied to different models as well as various domains.

Due to the successful development of deep image generation technology, visual data forgery detection would play a more important role in social and economic security. Existing forgery detection methods suffer from unsatisfactory generalization ability to determine the authenticity in the unseen domain. In this paper, we propose a novel Attention Consistency Refined masked frequency forgery representation model toward a generalizing face forgery detection algorithm (ACMF). Most forgery technologies always bring in high-frequency aware cues, which make it easy to distinguish source authenticity but difficult to generalize to unseen artifact types. The masked frequency forgery representation module is designed to explore robust forgery cues by randomly discarding high-frequency information. In addition, we find that the forgery saliency map inconsistency through the detection network could affect the generalizability. Thus, the forgery attention consistency is introduced to force detectors to focus on similar attention regions for better generalization ability. Experiment results on several public face forgery datasets (FaceForensic++, DFD, Celeb-DF, WDF and DFDC datasets) demonstrate the superior performance of the proposed method compared with the state-of-the-art methods. The source code and models are publicly available athttps://github.com/chenboluo/ACMF.

Deep neural networks are proven to be vulnerable to finely designed adversarial examples, and adversarial defense algorithms draw more and more attention nowadays. Pre-processing based defense is a major strategy, as well as learning robust feature representation, has been proven an effective way to boost generalization. However, existing defense works lack considering different depth-level visual features in the training process. In this paper, we first highlight two novel properties of robust features from the feature distribution perspective: 1) Diversity (robust features within the same class should maintain appropriate variety). 2) Discriminability (robust features from different classes should be sufficiently separated). We find that state-of-the-art defense methods aim to address both of these mentioned issues well. It motivates us to increase intra-class variance and decrease inter-class discrepancy simultaneously in adversarial training. Specifically, we propose a simple but effective defense based on decoupled visual representation masking. The designed Decoupled Visual Feature Masking (DFM) block can adaptively disentangle visual discriminative features and non-visual features with diverse mask strategies, while the suitable discarding information can disrupt adversarial noise to improve robustness. Our work provides a generic and easy-to-plugin block unit for any former adversarial training algorithm to achieve better protection integrally. Extensive experimental results prove that the proposed method can achieve superior performance compared with state-of-the-art defense approaches. The code is publicly available at https://github.com/chenboluo/Adversarial-defense.

Deep Neural Networks with natural training are quite vulnerable to adversarial attacks, so it’s necessary to defend these attacks with effective defense methods like adversarial training. However, while defending against adversarial attacks, adversarially trained models’ robustness between classes shows severe unfairness. To mitigate the disparity, a lot of methods have been proposed, while many of them sacrifice the overall accuracy to leverage the worst-class accuracy. Inspired by previous works, we propose a new fairness methodology, and we name it Class Encourage-suppress Robust Learning (CRL). Based on the overall accuracy and the class-wise accuracies of the dataset, we introduce a new measurement named Class Diversity Ratio to adjust the weights of different classes in the loss function. Additionally, we propose a new learning strategy called the Competitor Encourage-suppress Strategy to mitigate the disparity between diverse classes, which is simple but effective. Experimental results on representative datasets show that our method outperforms state-of-the-art (SOTA) methods.

In the era of digital media, the proliferation of forged images and videos poses a significant threat to societal stability. With the rapid advancement of deep learning, the generation of realistic fake images has become increasingly simple, presenting unprecedented challenges in discerning the authenticity of images. While some existing methods have shown promising results in forgery detection, they often underutilize facial semantic information. To address this issue, this paper introduces the Semantic Token Transformer for Face Forgery Detection. By incorporating facial semantic information with a transformer network, the input tokens of the transformer are transformed into tokens of varying shapes and sizes based on their importance, thereby enhancing the accuracy of the detector. To achieve this objective, we first employ an image processing stage to manipulate the image based on facial semantic information. Subsequently, we introduce a scoring network, guided by prior knowledge, which adaptively categorizes tokens into different clusters based on their importance and relevance to the results of the preprocessing stage. Finally, we merge the tokens within the clusters using an attention mechanism and input them into the detector for forgery detection. Through experiments conducted on multiple datasets and cross-dataset evaluations, we demonstrate that our approach outperforms state-of-the-art detection methods.

Recently, the breathtaking development and potential misuse of deepfake technology has raised numerous privacy and security concerns, triggering widespread apprehension. Existing deepfake detection methods focus on the analysis of local regions for faces, such as mouth movement, eye blinking frequency, etc., which, however, are limited in their ability to capture the global inconsistencies present in forged faces. Some researchers attempt to seize 3D artifacts related to facial global information, but typically treat the 3D information as mere input, lacking the in-depth analysis. To address these shortcomings and mine the inherent and delicate 3D artifacts in the forged faces, this paper innovatively proposes the 3D Artifact Detector (3DAD) method, which leverages the spatio-temporal inconsistency on the 3D semantic space in the forgery videos to uncover the deepfake clues. Specifically, we employ 3D Analysis Unit (3DAU) to pre-train the face reconstruction task within 3D Morphable Model (3DMM) space, thereby obtaining the high-level inherent 3d representation. Concurrently, for the multi-levels of information in the face, we utilize the Texture Perception Unit (TPU) to extract the texture information in the low-level semantic space of the images. Ultimately we feed the two distinct modalities into the spatiotemporal fusion model for final detection. Through extensive intra- and cross-dataset experiments on publicly available datasets, we demonstrate the effectiveness and generalizability of the proposed method. The source code is available at https://github.com/Cookie-XT/3DAD.

Text-based person search aims to identify specific target person from the database according to the given text description. Early work adopted separately pretrained encoders to extract visual and textual features, but benefit from the bloom of visual language pre-training, recent work uses unified pretrained visual language models such as CLIP as backbone. However, visual language models are generally pretrained from coarse-grained image-text pairs, while image-text pairs in text-based person search are more fine-grained to distinguish different persons. In addition, visual and linguistic concepts naturally organize themselves in a hierarchy, which is not explicitly captured by current large-scale vision and language models such as CLIP. To bridge this gap, we propose a novel Granularity-Aware Hyperbolic Representation learning method for mining granularity and capturing semantic hierarchy. Notably, we consider both token-level and instance-level granularity. For token-granularity alignment, we present a Bidirectional Attention Interaction module to explicitly learn the matching between fine-grained visual tokens and text tokens. For instance-granularity alignment, we equip the contrastive learning loss with Semantic Margin Softmax so that image-text pairs can perceive the similarity granularity of different samples during training. Besides, the global features of images and texts are mapped into hyperbolic space through Hyperbolic Representation Learning to embed tree-like data to capture semantic hierarchy. Extensive experiments verify the effectiveness of our proposed modules and show that our method achieves state-of-the-art results on the three widely acknowledged benchmarks, namely CUHK-PEDES, ICFG-PEDES, and RST-PReID. Our code is available at https://github.com/7chQ/GAHR.

In an era where personal photos are easily leaked and collected, face de-identification is a crucial method for protecting identity privacy. However, current face de-identification techniques face challenges in preserving attribute details and often produce anonymized results with reduced realistic. These shortcomings are particularly evident when handling occlusions, frequently resulting in noticeable editing artifacts. Our primary finding in this work is that simultaneous training of identity disentanglement and anonymization hinders their respective effectiveness. Therefore, we propose “Disentangle Before Anonymize”, a novel two-stage Framework (DBAF) designed for attribute-preserved and occlusion-robust de-identification. This framework includes a Contrastive Identity Disentanglement (CID) module and a Key-authorized Reversible Identity Anonymization (KRIA) module, achieving faithful attribute preservation and high-quality identity anonymization edits. Additionally, we introduce a Multi-scale Attentional Attribute Retention (MAAR) module to address the issue of reduced anonymization quality under occlusions. Extensive experiments demonstrate that our method outperforms state-of-the-art de-identification approaches, delivering superior quality, enhanced detail fidelity, improved attribute preservation performance, and greater robustness to occlusions. The code ispublicly available at: https://github.com/mrzhu-cool/DBAF.

Multi-modal image synthesis is crucial for obtaining complete modalities due to the imaging restrictions in reality. Current methods, primarily CNN-based models, find it challenging to extract global representations because of local inductive bias, leading to synthetic structure deformation or color distortion. Despite the significant global representation ability of transformer in capturing long-range dependencies, its huge parameter size requires considerable training data. Multi-modal synthesis solely based on one of the two structures makes it hard to extract comprehensive information from each modality with limited data. To tackle this dilemma, we propose a simple yet effective Recursive TransFusion (RTF) framework for multi-modal image synthesis. Specifically, we develop a TransFusion unit to integrate local knowledge extracted from the individual modality by connecting a CNN-based local representation block (LRB) and a transformer-based global fusion block (GFB) via a feature translating gate (FTG). Considering the numerous parameters introduced by the transformer, we further unfold a TransFusion unit with recursive constraint repeatedly, forming recursive TransFusion (RTF), which progressively extracts multi-modal information at different depths. Our RTF remarkably reduces network parameters while maintaining superior performance. Extensive experiments validate our superiority against the competing methods on multiple benchmarks. The source code will be available at https://github.com/guoliangq/RTF.

Graph neural networks (GNNs) encounter challenges in establishing deep structures and managing a large number of parameters effectively to learn node features comprehensively. Consequently, in vision tasks, GNNs often struggle to achieve high classification accuracy compared to convolutional neural networks. Nonetheless, GNNs retain crucial advantages and potential, particularly in lightweight network scale and efficient, reliable decision-making. Thus, improving GNN performance in vision tasks remains a significant research endeavor, with numerous important works exploring the application of GNN models in such contexts, where the graph representation of images poses a key challenge. Existing methods often fall short in adaptively generating blocks of different sizes and their corresponding edges to form graph representations according to graph semantics. To address this issue, we propose a novel method to convert images into graphical forms using granular-ball computing. Our approach does not rely on manual annotation or other learning methods, yet it can dynamically generate block nodes of varying sizes and corresponding edges. Compared to other state-of-the-art methods, our approach better captures semantic information within the graph. Despite having fewer parameters, our method significantly enhances accuracy. Overall, our work holds substantial implications for improving the performance of graph neural networks in vision tasks.

Multi-modal object Re-ID aims to leverage the complementary information provided by multiple modalities to overcome challenging conditions and achieve high-quality object matching. However, existing multi-modal methods typically rely on various modality interaction modules for information fusion, which can reduce the efficiency of real-time monitoring systems. Additionally, practical challenges such as low-quality multi-modal data or missing modalities further complicate the application of object Re-ID. To address these issues, we propose the Complementary Data Enhancement and Modal-Aware Soft Alignment Network (DESANet), which is designed to be independent of interactive networks and adaptable to scenarios with missing modalities. This approach ensures a simple-yet-effective, and efficient multi-modal object Re-ID. DESANet consists of three key components: Firstly, the Dual-Color Space Data Enhancement (DCDE) module, which enhances multi-modal data by performing patch rotation in the RGB space and improving image quality in the HSV space. Secondly, the Salient Feature ReConstruction (SFRC) module, which addresses the issue of missing modalities by reconstructing features from one modality using the other two. Thirdly, the Modal-Aware Soft Alignment (MASA) module, which integrates multi-source data to avoid the blind fusion of features and prevents the propagation of noise from reconstructed modalities. Our approach achieves state-of-the-art performances on both person and vehicle datasets. Source code is available at https://github.com/DWJ11/DESANet.

Current RGB-D methods usually leverage large-scale backbones to improve accuracy but sacrifice efficiency. Meanwhile, several existing lightweight methods are difficult to achieve high-precision performance. To balance the efficiency and performance, we propose a Speed-Accuracy Tradeoff Network (SATNet) for Lightweight RGB-D SOD from three fundamental perspectives: depth quality, modality fusion, and feature representation. Concerning depth quality, we introduce the Depth Anything Model to generate high-quality depth maps,which effectively alleviates the multi-modal gaps in the current datasets. For modality fusion, we propose a Decoupled Attention Module (DAM) to explore the consistency within and between modalities. Here, the multi-modal features are decoupled into dual-view feature vectors to project discriminable information of feature maps. For feature representation, we develop a Dual Information Representation Module (DIRM) with a bi-directional inverted framework to enlarge the limited feature space generated by the lightweight backbones. DIRM models texture features and saliency features to enrich feature space, and employ two-way prediction heads to optimal its parameters through a bi-directional backpropagation. Finally, we design a Dual Feature Aggregation Module (DFAM) in the decoder to aggregate texture and saliency features. Extensive experiments on five public RGB-D SOD datasets indicate that the proposed SATNet excels state-of-the-art (SOTA) CNN-based heavyweight models and achieves a lightweight framework with 5.2 M parameters and 415 FPS. The code is available at https://github.com/duan-song/SATNet.

Current object detectors often suffer performance degradation when applied to cross-domain scenarios, particularly under challenging visual conditions such as nighttime scenes. This is primarily due to the I3 problems: Inadequate sampling of instance-level features, Indistinguishable feature representation across domains and Inaccurate generation for identical category participation. To address these challenges, we propose a domain-adaptive detection framework that enables robust generalization across different visual domains without introducing any additional inference overhead. The framework comprises three key components. Specifically, the centerness-category consistency sampler alleviates inadequate sampling by selecting representative instance-level features, while the paired centerness consistency loss enforces alignment between classification and localization. Second, VLM-based orthogonality enhancement leverages frozen vision-language encoders with an orthogonal projection loss to improve cross-domain feature distinguishability. Third, hallucination feature generator synthesizes robust instance-level features for missing categories, ensuring balanced category participation across domains. Extensive experiments on multiple datasets covering various domain adaptation and generalization settings demonstrate that our method consistently outperforms state-of-the-art detectors, achieving up to 5.5 mAP improvement, with particularly strong gains in nighttime adaptation.

Conventional reconstruction-based video anomaly detection (VAD) methods implicitly model normality in latent spaces, which is limited by the generalization ability of latent features. Normalizing Flow (NF)-based methods have been introduced to address this issue, as they explicitly model the distribution of input data and achieve significant performance in VAD. However, existing NF-based methods are confined to Euclidean space, limiting their ability to model action hierarchies. While effective at capturing local joint dynamics and short-term temporal variations, they fail to encode kinematic dependencies and long-term pose evolution, ultimately struggling to discern ambiguous anomalies that deviate minimally from normal motion. In contrast, hyperbolic representation learning, with its ability to model hierarchical and complex relationships among actions, offers a promising solution to enhance the discriminative power between similar skeletal actions. Motivated by this, we propose a novel Dual-Space Normalizing Flow (DSNF) method. Specifically, we design a Dual-Space Parallel Graph Convolutional Network (DSPGCN) that synergistically integrates the strengths of both Euclidean and hyperbolic geometries to simultaneously capture local detail features of poses and intrinsic hierarchical relationships of actions. To enhance the model's focus on discriminative features, we design an Adaptive Weighted Approximation Mass (AWAM) loss that dynamically adjusts weights to impose stronger constraints on regions with low discriminability in the dual space, encouraging the model to focus more on key discriminative features in hyperbolic space that reflect complex relationships between actions. Extensive experiments on public datasets demonstrate the effectiveness and robustness of our method in various VAD scenarios.

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen attribute-object compositions by leveraging prior knowledge of known primitives. However, real-world visual features of attributes and objects are often entangled, causing distribution shifts between seen and unseen combinations. Existing methods often ignore intrinsic variations and interactions among primitives, leading to poor feature discrimination and biased predictions. To address these challenges, we propose Multi-level Contextual Prototype Modulation (MCPM), a transformer-based framework with a hierarchical structure that effectively integrates attributes and objects to generate richer visual embeddings. At the feature level, we apply contrastive learning to improve discriminability across compositional tasks. At the prototype level, a subclass-driven modulator captures fine-grained attribute-object interactions, enabling better adaptation to long-tail distributions. Additionally, we introduce a Minority Attribute Enhancement (MAE) strategy that synthesizes virtual samples by mixing attribute classes, further mitigating data imbalance. Experiments on four benchmark datasets (MIT-States, C-GQA, UT-Zappos, and VAW-CZSL) show that MCPM brings significant performance improvements, verifying its effectiveness in complex composition scenes.

Face recognition has achieved remarkable progress and is widely deployed in real-world scenarios. Recently more and more attention has been given to individual privacy protection, due to unauthorized sensitive image leakage by malicious attackers. Multi-modality face images captured by diverse sensors, also called heterogeneous faces, bring in more challenges in face privacy protection while lacking related research. In this paper, we propose a novel visual Privacy preserving method for Heterogeneous Face Recognition (Privacy-HFR) to protect perceptual visual information and maintain essential identity information in multi-modality face analysis scenarios. Frequency domain analysis is a vital strategy to bridge the inevitable modality gap for heterogeneous face images. Meanwhile, recent theoretical insights also inspire us to design a suitable frequency component adjustment to balance human visual sensitivity and identity discriminative information. In addition, the ability to defend against recovery attacks has emerged as an essential criterion for privacy preserving face recognition. Noting that there seems to exist a dilemma that reducing accessible information by the attack model will affect the extracted identity information for recognition. It is because these two kinds of information are mutually blended in the frequency domain, which makes it a challenge to simultaneously maintain visual privacy and identity distinguishability. Thus, we provide a novel perspective to leverage the randomly optimal solutions and design the specific adversarial perturbations against the recovery attack. Experiments on several large-scale heterogeneous face datasets (CASIA NIR-VIS 2.0, LAMP-HQ, Tufts Face and CUFSF datasets) prove that the proposed method outperforms existing privacy-preserving face recognition methods in terms of recognition accuracy and privacy protection capability. The code is available in https://github.com/xiyin11/Privacy-HFR.

With the rapid development of Generative Adversarial Networks (GANs), facial sketch generation and age transformation have advanced considerably. These technologies show great potential in digital media, entertainment, and forensic applications, particularly in helping law enforcement reconstruct the appearance of long-term fugitives. However, current methodologies exhibit notable limitations: existing approaches typically specialize in either facial sketch generation or age progression independently, lacking an effective integration for cross-domain synthesis. Moreover, preserving identity information while ensuring high-quality image generation remains a challenge. This paper proposes Multi-Scale Feature Extraction Networks (MSFE), an image-to-image translation framework that enables continuous age transformation while maintaining the stylistic characteristics of sketch domains. The core of the MSFS network uses a Dual Conditional Normalization Attention (DCNA) architecture to extract sketch features and encode facial images into the latent space of a pre-trained StyleGAN based on the desired age change. Experimental results on public datasets demonstrate that our approach outperforms existing methods, achieving superior facial photo-sketch synthesis with enhanced realism, identity preservation, and age accuracy.

With the development of face forgery technology, fake faces are rampant, threatening the security and authenticity of many fields. Therefore, it is of great significance to study face forgery detection. At present, existing detection methods have deficiencies in the comprehensiveness of feature extraction and model adaptability, and it is difficult to accurately deal with complex and changeable forgery scenarios. However, the rise of multimodal models provides new insights for current forgery detection methods. At present, most methods use relatively simple text prompts to describe the difference between real and fake faces. However, these researchers ignore that the CLIP model itself does not have the relevant knowledge of forgery detection. Therefore, our paper proposes a face forgery detection method based on multi-encoder fusion and cross-modal knowledge distillation. On the one hand, the prior knowledge of the CLIP model and the forgery model is fused. On the other hand, through the alignment distillation, the student model can learn the visual abnormal patterns and semantic features of the forged samples captured by the teacher model. Specifically, our paper extracts the features of face photos by fusing the CLIP text encoder and the CLIP image encoder, and uses the dataset in the field of forgery detection to pretrain and fine-tune the Deepfake-V2-Model to enhance the detection ability, which are regarded as the teacher model. At the same time, the visual and language patterns of the teacher model are aligned with the visual patterns of the pretrained student model, and the aligned representations are refined to the student model. This not only combines the rich representation of the CLIP image encoder and the excellent generalization ability of text embedding, but also enables the original model to effectively acquire relevant knowledge for forgery detection. Experiments show that our method effectively improves the performance on face forgery detection.

Existing free-energy guided No-Reference Image Quality Assessment (NR-IQA) methods continue to face challenges in effectively restoring complexly distorted images. The features guiding the main network for quality assessment lack interpretability, and efficiently leveraging high-level feature information remains a significant challenge. As a novel class of state-of-the-art (SOTA) generative model, the diffusion model exhibits the capability to model intricate relationships, enhancing image restoration effectiveness. Moreover, the intermediate variables in the denoising iteration process exhibit clearer and more interpretable meanings for high-level visual information guidance. In view of these, we pioneer the exploration of the diffusion model into the domain of NR-IQA. We design a novel diffusion model for enhancing images with various types of distortions, resulting in higher quality and more interpretable high-level visual information. Our experiments demonstrate that the diffusion model establishes a clear mapping relationship between image reconstruction and image quality scores, which the network learns to guide quality assessment. Finally, to fully leverage high-level visual information, we design two complementary visual branches to collaboratively perform quality evaluation. Extensive experiments are conducted on seven public NR-IQA datasets, and the results demonstrate that the proposed model outperforms SOTA methods for NR-IQA. The codes will be available at https://github.com/handsomewzy/DiffV2IQA.

Recently, AI-generated images (AIGIs), synthesized based on initial textual prompts, have attracted widespread attention. However, due to limitations in current generation techniques, these images often exhibit degraded perceptual quality and semantic misalignment with the guiding prompts. Therefore, evaluating both perceptual quality and text-to-image alignment is essential for optimizing the performance of generative models. Existing methods design textual prompts solely based on the initial prompt for both perceptual and alignment quality tasks, and compute only coarse-grained similarity between the designed prompt and the generated image. However, such task-agnostic prompts overlook the distinctions between the perceptual and alignment quality tasks, and coarse-level similarity fails to capture semantic details, leading to suboptimal evaluation performance. To address these challenges, we propose a novel AIGI quality assessment framework, termed TPMS, which incorporates task-specific prompt and multi-granularity similarity computation. The task-specific prompt constructs dedicated prompts for perceptual and alignment quality respectively, allowing the model to capture distinct quality cues tailored to each evaluation task. Multi-granularity similarity measures the coarse-level similarity between the generated image and task-specific prompts to capture global quality characteristics, and the fine-level similarity between the generated image and the initial prompt to enhance semantic detail awareness. By integrating these two complementary similarities, TPMS enables precise and robust quality prediction. Extensive experiments on four widely-used AIGI quality benchmarks validate the effectiveness and superiority of the proposed framework.

Creating a comprehensively representative image while maintaining the merits of various modalities is a key focus of current Multi-Modality Image Fusion research. Existing unified methods often struggle to handle varying types of degradation while extracting modality-shared and modality-specific information from source images, leading to limitations in their generative or representation capabilities under different conditions. To address the challenge, we propose MVFusion, a novel self-supervised masked variational autoencoder framework that simultaneously enhances generative training and representation learning. It is designed to cope with varying image quality and dataset composition with a unified framework while ensuring effective fusion of modality information. Specifically, MVFusion employs a self-supervised masked autoencoder to reduce the impact of redundancy and degradation in the source images, and thus learns the latent distribution of degraded input images in the generative training stage. In addition, we incorporate variational feature learning to further preserve the distinctive modality features in the representation learning stage. Extensive experiments demonstrate that our model achieves promising results in several classical fusion tasks, including infrared-visible, multi-focus, multi-exposure, and medical image fusion. The code is available at https://github.com/shiboneng/MVFusion.

Unsupervised person re-identification aims to retrieve a given pedestrian image from unlabeled data. For training on the unlabeled data, the method of clustering and assigning pseudo-labels has become mainstream, but the pseudo-labels themselves are noisy and will reduce the accuracy. To overcome this problem, several pseudo-label improvement methods have been proposed. But on the one hand, they only use target domain data for fine-tuning and do not make sufficient use of high-quality labeled data in the source domain. On the other hand, they ignore the critical fine-grained features of pedestrians and overfitting problems in the later training period. In this paper, we propose a novel unsupervised cross-domain person re-identification network (IDENet) based on an inter-domain equilibrium structure to improve the quality of pseudo-labels. Specifically, we make full use of both source domain and target domain information and construct a small learning network to equalize label allocation between the two domains. Based on it, we also develop a dynamic neural network with adaptive convolution kernels to generate adaptive residuals for adapting domain-agnostic deep fine-grained features. In addition, we design the network structure based on ordinary differential equations and embed modules to solve the problem of network overfitting. Extensive cross-domain experimental results on Market1501, PersonX, and MSMT17 prove that our proposed method outperforms the state-of-the-art methods.

Cross-domain few-shot learning aims to achieve swift generalization between a source domain and a target domain using a limited number of images. Current research predominantly relies on generalized feature embeddings, employing metric classifiers in Euclidean space for classification. However, due to existing disparities among different data domains, attaining generalized features in the embedding becomes challenging. Additionally, the rise in data domains leads to high-dimensional Euclidean spaces. To address the above problems, we introduce a cross-domain few-shot learning method named Hyperbolic Insights with Knowledge Distillation (HIKD). By integrating knowledge distillation, it enhances the model's generalization performance, thereby significantly improving task performance. Hyperbolic space, in comparison to Euclidean space, offers a larger capacity and supports the learning of hierarchical structures among images, which can aid generalized learning across different data domains. So we map the Euclidean space features to the hyperbolic space via hyperbolic embedding and utilize hyperbolic fitting distillation method in the meta-training phase to obtain multi-domain unified generalization representation. In the meta-testing phase, accounting for biases between the source and target domains, we present a hyperbolic adaptive module to adjust embedded features and eliminate inter-domain gap. Experiments on the Meta-Dataset demonstrate that HIKD outperforms state-of-the-arts methods with the average accuracy of 80.6%.

Semi-Supervised Object Detection (SSOD) aims to improve the utilization of unlabeled data, and various methods, such as adaptive threshold techniques, have been extensively studied to increase exploitable information. However, these methods are passive, relying solely on the original image data. Additionally, existing approaches prioritize the predicted categories of the teacher model while overlooking the relationships between different categories in the prediction. In this paper, we introduce a novel approach called Dense Information Learning (DIL), which actively generates unlabeled data containing densely exploitable information and forces the network to have relation consistency under different perturbations. Specifically, Dense Information Augmentation (DIA) leverages the prior information of the network to create a foreground bank and actively incorporates exploitable information into the unlabeled data. DIA automatically performs information enhancement and filters noise. Furthermore, to encourage the network to maintain consistency at the manifold level under various perturbations, we introduce Relation Consistency Regularization (RCR). It considers both feature-level and image-level perturbations, guiding the network to focus on more discriminative features. Extensive experiments conducted on multiple datasets validate the effectiveness of our approach in leveraging information from unlabeled images. The proposed DIL improves the mAP by 12.6% and 10.0% relative to the supervised baseline method when utilizing 5% and 10% of labeled data on the MS-COCO dataset, respectively.

Semi-supervised object detection (SSOD) aims to solve the data annotation challenge in object detection and can achieve remarkable progress in natural scenes; however, it remains unexplored in horizontal bounding box (HBB)-based remote sensing imagery where annotation tasks pose greater challenges. In remote sensing scenarios, objects exhibit arbitrary orientations, small scales, and dense distributions, leading to pseudoboxes with fuzzy boundaries and class imbalance issues. Therefore, we propose UNCertainty quantification (UNC) for SSOD in remote sensing images. UNC uses uncertainty to guide the network from both regression and classification perspectives: Semantic alignment SAM calibration (SASC) uses pseudoboxes as box prompts for the input of the segment anything model (SAM), achieving more precise boundaries. Subsequently, boundaries with lower regression uncertainty are selected as the final pseudoboxes, ensuring better alignment between the pseudoboxes and the ground truth. Dynamic uncertainty weighting (DUW) calculates class uncertainty and determines its correlation with the availability of instances per class. High uncertainty implies limited availability of instances, necessitating greater emphasis on instances of that class. Furthermore, we set a percentage uncertainty threshold to avoid overemphasis caused by individual classes. Extensive experiments conducted on the DIOR and DOTA HBB-based datasets demonstrate the effectiveness of our method in leveraging unlabeled image information. Specifically, compared with the supervised baseline method, the UNC method improves mAP by 12.4% and 8.6% when 5% and 10% of labeled data on DIOR, respectively.

Image Super-Resolution (SR) has seen remarkable progress with the emergence of transformer-based architectures. However, due to the high computational cost, many existing transformer-based SR methods limit their attention to local windows, which hinders their ability to model long-range dependencies and global structures. To address these challenges, we propose a novel SR framework named Semantic-Driven Global-Local Fusion Transformer (SGLFT). The proposed model enhances the receptive field by combining a Hybrid Window Transformer (HWT) and a Scalable Transformer Module (STM) to jointly capture local textures and global context. To further strengthen the semantic consistency of reconstruction, we introduce a Semantic Extraction Module (SEM) that distills high-level semantic priors from the input. These semantic cues are adaptively integrated with visual features through an Adaptive Feature Fusion Semantic Integration Module (AFFSIM). Extensive experiments on standard benchmarks demonstrate the effectiveness of SGLFT in producing visually faithful and structurally consistent SR results. The code will be available at https://github.com/kbzhang0505/SGLFT.

Infrared small target detection (IRSTD) is of great practical significance in many real-world applications, such as maritime rescue and early warning systems, benefiting from the unique and excellent infrared imaging ability in adverse weather and low-light conditions. Nevertheless, segmenting small targets from the background remains a challenge. When the subsampling frequency during image processing does not satisfy the Nyquist criterion, the aliasing effect occurs, which makes it extremely difficult to identify small targets. To address this challenge, we propose a novel Wavelet Mamba with Reversible Structure Network (WMRNet) for infrared small target detection in this paper. Specifically, WMRNet consists of a Discrete Wavelet Mamba (DW-Mamba) module and a Third-order Difference Equation guided Reversible (TDE-Rev) structure. DW-Mamba employs the Discrete Wavelet Transform to decompose images into multiple subbands, integrating this information into the state equations of a state space model. This method minimizes frequency interference while preserving a global perspective, thereby effectively reducing background aliasing. The TDE-Rev aims to suppress edge aliasing effects by refining the target edges, which first processes features with an explicit neural structure derived from the second-order difference equations and then promotes feature interactions through a reversible structure. Extensive experiments on the public IRSTD-1k and SIRST datasets demonstrate that the proposed WMRNet outperforms the state-of-the-art methods.

In response to the worldwide COVID-19 pandemic, advanced automated technologies have emerged as valuable tools to aid healthcare professionals in managing an increased workload by improving radiology report generation and prognostic analysis. This study proposes a Multi-modality Regional Alignment Network (MRANet), an explainable model for radiology report generation and survival prediction that focuses on high-risk regions. By learning spatial correlation in the detector, MRANet visually grounds region-specific descriptions, providing robust anatomical regions with a completion strategy. The visual features of each region are embedded using a novel survival attention mechanism, offering spatially and risk-aware features for sentence encoding while maintaining global coherence across tasks. A cross-domain LLMs-Alignment is employed to enhance the image-to-text transfer process, resulting in sentences rich with clinical detail and improved explainability for radiologists. Multi-center experiments validate the overall performance and each module's composition within the model, encouraging further advancements in radiology report generation research emphasizing clinical interpretation and trustworthiness in AI models applied to medical studies.

Automatic road extraction from high-resolution remote sensing images has greatly facilitated the applications of high-precision road mapping in autonomous driving and intelligent transportation. However, challenges such as occlusions from buildings, trees, and complex road shapes bring great difficulties to precise road extraction. Also, existing methods often overlook the integrity of road direction and edge, leading to unsatisfactory extraction results. To alleviate the issue, this paper has presented a direction and edge co-awareness network (DENet). Firstly, the road edge detector (RED) is introduced to extract coarse road edges with abundant directional information. By leveraging the edge enhancement blocks, the edge structures of road can be efficiently refined, achieving the extraction of intricate narrow and elongated road shapes. Secondly, we incorporate the directional spatial attention (DSA) mechanism within the dual encoders and decoders to promote the extraction and fusion of road directional information and elongated features from different orientations, thus greatly mitigating the road occlusion issue. Finally, to fully interlace potential road information, a grouped local-global feature fusion (GLFF) is specifically designed to exchange multi-scale semantic information across different channels, simultaneously emphasizing road features and suppressing irrelevant background features. Numerous experimental results on three public datasets demonstrate the effectiveness and efficiency of the proposed DENet for road extraction, achieving F1 scores of 78.51% on the CHN6-CUG dataset, 79.35% on the Massachusetts road dataset, and 77.90% on the GF2-FC dataset, outperforming several existing state-of-the-art methods. The code is available at:https://github.com/gwy103/DENet.

This paper proposes a novel hierarchical controller for connected truck platoons. To this end, the predecessor following topology is used to characterize the communication connectivity between connected trucks. Then, a longitudinal efficient controller consisting of upper-level and lower-level controllers is proposed. In particular, the upper-level controller is designed based on the kinematic model to handle the car-following interactions between connected trucks and delays in communication and input. The lower-level controller comprises a feedforward and a feedback control law. The feedforward control law converts the desired acceleration from the upper-level controller into the vehicle throttle or braking pressure using the inverse dynamic model, while the feedback control law compensates for the control error caused by unknown vehicle parameters. In addition, in the linear region, the internal stability is analyzed based on the second-order kinematic model using s-domain analysis and linearization method, respectively. Then, the string stability is proved. The influence of parameters on the stability performance is extensively discussed using the stability diagram. Finally, the feasibility of the proposed controller is verified via co-simulations in PreScan and TruckSim, in terms of acceleration, velocity, and spacing error profiles.

Existing studies of multi-modality medical image segmentation tend to aggregate all modalities without discrimination and employ multiple symmetric encoders or decoders for feature extraction and fusion. They often overlook the different contributions to visual representation and intelligent decisions among multi-modality images. Motivated by this discovery, this paper proposes an asymmetric adaptive heterogeneous network for multi-modality image feature extraction with modality discrimination and adaptive fusion. For feature extraction, it uses a heterogeneous two-stream asymmetric feature-bridging network to extract complementary features from auxiliary multi-modality and leading single-modality images, respectively. For feature adaptive fusion, the proposed Transformer-CNN Feature Alignment and Fusion (T-CFAF) module enhances the leading single-modality information, and the Cross-Modality Heterogeneous Graph Fusion (CMHGF) module further fuses multi-modality features at a high-level semantic layer adaptively. Comparative evaluation with ten segmentation models on six datasets demonstrates significant efficiency gains as well as highly competitive segmentation accuracy. (Our code is publicly available at https://github.com/joker-527/AAHN).

Weakly supervised LiDAR semantic segmentation has made significant strides with limited labeled data. However, most existing methods focus on the network training under weak supervision, while efficient annotation strategies remain largely unexplored. To tackle this gap, we implement LiDAR semantic segmentation using scatter image annotation, effectively integrating an efficient annotation strategy with network training. Specifically, we propose employing scatter images to annotate LiDAR point clouds, combining a pre-trained optical flow estimation network with a foundational image segmentation model to rapidly propagate manual annotations into dense labels for both images and point clouds. Moreover, we propose ScatterNet, a network that includes three pivotal strategies to reduce the performance gap caused by such annotations. First, it utilizes dense semantic labels as supervision for the image branch, alleviating the modality imbalance between point clouds and images. Second, an intermediate fusion branch is proposed to obtain multimodal texture and structural features. Finally, a perception consistency loss is introduced to determine which information needs to be fused and which needs to be discarded during the fusion process. Extensive experiments on the nuScenes and SemanticKITTI datasets demonstrate that our method requires less than 0.02% of the labeled points to achieve over 95% of the performance of fully-supervised methods. Notably, our labeled points are only 5% of those used in the most advanced weakly supervised methods.

Inconsistent accuracy between classification and localization tasks is a common challenge in modern object detection. Task decoupling, which employs distinct features or labeling strategies for each task, is a widely used approach to address this issue. Although it has led to noteworthy advancements, this approach is insufficient as it neglects task interdependence and lacks an explicit consistency constraint. To bridge this gap, this paper proposes the Progressive Semi-Decoupled Detector (ProSDD) to enhance both classification and localization accuracy. Specifically, a new detection head is designed that incorporates feature suppression and enhancement mechanism (FSEM) and bidirectional interaction module (BIM). Compared with the decoupled head, it not only filters out task-irrelevant information and enhances task-related information, but also avoids excessive decoupling at the feature level. Moreover, both FSEM and BIM are used multiple times, thus forming a progressive semi-decoupled head. Then, a novel consistency loss is proposed and integrated into the loss function of object detection, ensuring harmonic performance in classification and localization. Experimental results demonstrate that the proposed ProSDD effectively alleviates inconsistent accuracy and achieves high-quality object detection. Taking the pretrained ResNet-50 as the backbone, ProSDD achieves a remarkable 43.3 AP on the MS COCO dataset, surpassing contemporary state-of-the-art detectors by a substantial margin under the equivalent configurations. Code is available athttps://github.com/HB-X/ProSDD.

Multimodal sentiment analysis aims at exploiting complementary information from multiple modalities or data sources to enhance the understanding and interpretation of sentiment. While existing multi-modal fusion techniques offer significant improvements in sentiment analysis, real-world scenarios often involve missing modalities, introducing complexity due to uncertainty of which modalities may be absent. To tackle the challenge of incomplete modality-specific feature extraction caused by missing modalities, this paper proposes a Cosine Margin-Aware Network (CMANet) which centers on the Cosine Margin-Aware Distillation (CMAD) module. The core module measures distance between samples and the classification boundary, enabling CMANet to focus on samples near the boundary. So, it effectively captures the unique features of different modal combinations. To address the issue of modality imbalance during modality-specific feature extraction, this paper proposes a Weak Modality Regularization (WMR) strategy, which aligns the feature distributions between strong and weak modalities at the dataset-level, while also enhancing the prediction loss of samples at the sample-level. This dual mechanism improves the recognition robustness of weak modality combination. Extensive experiments demonstrate that the proposed method outperforms the previous best model, MMIN, with a 3.82% improvement in unweighted accuracy. These results underscore the robustness of the approach under conditions of uncertain and missing modalities.

Early weakly supervised video grounding (WSVG) methods often struggle with incomplete boundary detection due to the absence of temporal boundary annotations. To bridge the gap between video-level and boundary-level annotations, explicit supervision methods (i.e., generating pseudo-temporal boundaries for training) have achieved great success. However, data augmentation in these methods might disrupt critical temporal information, yielding poor pseudo-temporal boundaries. In this paper, we propose a new perspective that maintains the integrity of the original temporal content while introducing more valuable information for expanding the incomplete boundaries. To this end, we proposeETC(ExpandthenClarify), first using the additional information to expand the initial incomplete pseudo-temporal boundaries, and subsequently refining these expanded ones to achieve precise boundaries. Motivated by video continuity, i.e., visual similarity across adjacent frames, we use powerful multi-modal large language models (MLLMs) to annotate each frame within the initial pseudo-temporal boundaries, yielding more comprehensive descriptions for expanded boundaries. To further clarify the noise in expanded boundaries, we combine mutual learning with a tailored proposal-level contrastive objective to use a learnable approach to harmonize a balance between incomplete yet clean (initial) and comprehensive yet noisy (expanded) boundaries for more precise ones. Experiments demonstrate the superiority of our method on two challenging WSVG datasets.

Existing facial expression recognition (FER) methods typically fine-tune a pre-trained visual encoder using discrete labels. However, this form of supervision limits to specify the emotional concept of different facial expressions. In this paper, we observe that the rich knowledge in text embeddings, generated by vision-language models, is a promising alternative for learning discriminative facial expression representations. Inspired by this, we propose a novel knowledge-enhanced FER method with an emotional-to-neutral transformation. Specifically, we formulate the FER problem as a process to match the similarity between a facial expression representation and text embeddings. Then, we transform the facial expression representation to a neutral representation by simulating the difference in text embeddings from textual facial expression to textual neutral. Finally, a self-contrast objective is introduced to pull the facial expression representation closer to the textual facial expression, while pushing it farther from the neutral representation. We conduct evaluation with diverse pre-trained visual encoders including ResNet-18 and Swin-T on four challenging facial expression datasets. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art FER methods. The code is made publicly available athttps://github.com/hangyu94/KE2NT.

This paper targets a novel trade-off problem in generalizable prompt learning for vision-language models (VLM), i.e., improving the performance on unseen classes while maintaining the performance on seen classes. Comparing with existing generalizable methods that neglect the seen classes degradation, the setting of this problem is stricter and fits more closely with practical applications. To solve this problem, we start from the optimization perspective, and leverage the relationship between loss landscape geometry and model generalization ability. By analyzing the loss landscapes of the state-of-the-art method and vanilla Sharpness-aware Minimization (SAM) based method, we conclude that the trade-off performance correlates to bothloss valueandloss sharpness, while each of them is indispensable. However, we find the optimizing gradient of existing methods cannot maintain high relevance to both loss value and loss sharpness during optimization, which severely affects their trade-off performance. To this end, we propose a novel SAM-based method for prompt learning, denoted as Gradient Constrained Sharpness-aware Context Optimization (GCSCoOp), to dynamically constrain the optimizing gradient, thus achieving above two-fold optimization objective simultaneously. Extensive experiments verify the effectiveness of GCSCoOp in the trade-off problem.

Cloth-changing person re-identification (CC-ReID) aims to match persons who change clothes over long periods. The key challenge in CC-ReID is to extract cloth-irrelated features, such as face, hairstyle, body shape, and gait. Current research mainly focuses on modeling body shape using multi-modal biological features (such as silhouettes and sketches). However, it does not fully leverage the personal description information hidden in the original RGB image. Considering that there are certain attribute descriptions that remain unchanged after the changing of cloth, we propose a Masked Attribute Description Embedding (MADE) method that unifies personal visual appearance and attribute description for CC-ReID. Specifically, handling variable cloth-sensitive information, such as color and type, is challenging for effective modeling. To address this, we mask the clothes type and color information (upper body type, upper body color, lower body type, and lower body color) in the personal attribute description extracted through an attribute detection model. The masked attribute description is then connected and embedded into Transformer blocks at various levels, fusing it with the low-level to high-level features of the image. This approach compels the model to discard cloth information. Experiments are conducted on several CC-ReID benchmarks, including PRCC, LTCC, Celeb-reID-light, and LaST. Results demonstrate that MADE effectively utilizes attribute description, enhancing cloth-changing person re-identification performance, and compares favorably with state-of-the-art methods.

Current timestamp-supervised temporal action segmentation (TS-TAS) methods typically follow a two-phase pipeline: initializing the model with timestamp labels and refining it with pseudo-labels. However, limited by the sparsity of timestamp annotations, current methods' performance is sub-optimal. Specifically, initializing the model with only timestamp annotations may cause overfitting to labeled frames. Additionally, sparse timestamp annotations cannot capture the diverse action representations throughout the whole instance, especially those near the ambiguous action boundaries, leading to pseudo-label noise. Inspired by the cluster assumption of semi-supervised learning (SSL) that points within the same manifold likely share the same label, we here model TS-TAS as an SSL problem. Specifically, we propose a Temporal Embedding Consistency (TEC) strategy to mitigate the excessive focus on annotated frames. The TEC strategy encourages frames with similar representations within the video to have similar classification probability distributions, thereby propagating labeled frames' information to implicit ones. Besides, we design a TS-Mix strategy to further leverage unlabeled data to mitigate the influence of pseudo-label noise in a consistency regularization manner. The TS-Mix strategy includes intra-mix, which adds linear interpolation of two adjacent timestamps to every frame between them, and inter-mix, which mixes frames from two different untrimmed videos frame-by-frame. Then the mixed video is trained with the correspondingly mixed pseudo-labels. Comprehensive experimental results on different benchmarks show that we achieve new state-of-the-art performances. Furthermore, the proposed method can seamlessly enhance existing methods, significantly improving their performances.

Food image composition requires the use of existing dish images and background images to synthesize a natural new image, while diffusion models have made significant advancements in image generation, enabling the construction of end-to-end architectures that yield promising results. However, existing diffusion models face challenges in processing and fusing information from multiple images and lack access to high-quality publicly available datasets, which prevents the application of diffusion models in food image composition. In this paper, we introduce a large-scale, high-quality food image composite dataset,FC22 k, which comprises 22,000 foreground, background, and ground truth ternary image pairs. Additionally, we propose a novel food image composition method,Foodfusion, which leverages the capabilities of the pre-trained diffusion models and incorporates a Fusion Module for processing and integrating foreground and background information. This fused information aligns the foreground features with the background structure by merging the global structural information at the cross-attention layer of the denoising UNet. To further enhance the content and structure of the background, we also integrate a Content-Structure Control Module. Extensive experiments demonstrate the effectiveness and scalability of our proposed method.

Low-light Image Enhancement (LLIE) aims to rectify inadequate illumination conditions and achieve superior visual quality in images, which plays a pivotal role in the domain of low-level computer vision. Due to poor illumination in images, many high-frequency details are obscured, which leads to an uneven distribution of low- and high-frequency information. However, most existing LLIE methods do not pay special attention to the restoration of high-frequency detail information and some challenging-to-recover areas in images. To address this issue, we propose a novel progressive prompt-driven LLIE framework with frequency aware learning, through a two-stage coarse-to-fine learning mechanism. Specifically, the proposed method fully utilizes both the specially designed brightness-aware prompt and detail-aware prompt on the prior trained model, to achieve an excellent enhanced image that exhibits more natural brightness and richer detail information. Furthermore, the proposed frequency aware learning objective can adaptively adjust the contribution of individual pixels for image reconstruction based on the statistics of high- and low-frequency features, which enables the network to focus on learning intricate details and other challenging areas in low-light images. Extensive experimental results demonstrate the effectiveness of the proposed method, achieving superior performances to state-of-the-art methods on representative real-world and synthetic datasets. Our source code is available athttps://github.com/MSL502/PPFAL.

Weakly supervised temporal sentence grounding aims to identify semantically relevant video moments in an untrimmed video corresponding to a given sentence query without exact timestamps. Neuropsychology research indicates that the way the human brain handles information varies based on the grammatical categories of words, highlighting the importance of separately considering nouns and verbs. However, current methodologies primarily utilize pre-extracted video features to reconstruct randomly masked queries, neglecting the distinction between grammatical classes. This oversight could hinder forming meaningful connections between linguistic elements and the corresponding components in the video. To address this limitation, this paper introduces the dual semantic reconstruction network (DSRN) model. DSRN processes video features by distinctly correlating object features with nouns and motion features with verbs, thereby mimicking the human brain's parsing mechanism. It begins with a feature disentanglement module that separately extracts object-aware and motion-aware features from video content. Then, in a dual-branch structure, these disentangled features are used to generate separate proposals for objects and motions through two dedicated proposal generation modules. A consistency constraint is proposed to ensure a high level of agreement between the boundaries of object-related and motion-related proposals. Subsequently, the DSRN independently reconstructs masked nouns and verbs from the sentence queries using the generated proposals. Finally, an integration block is applied to synthesize the two types of proposals, distinguishing between positive and negative instances through contrastive learning. Experiments on the Charades-STA and ActivityNet Captions datasets demonstrate that the proposed method achieves state-of-the-art performance.

The objective of blind image quality assessment (BIQA) is to develop a model capable of automatically evaluating image quality without requiring any reference knowledge. While multi-task learning has been widely utilized in BIQA, it has predominantly remained unimodal. This paper delves into the Visual-Language multi-task BIQA model, where distortion knowledge can be captured through image-text contrastive learning. Specifically, Visual-Language auxiliary tasks targeting distortion type and quality level are introduced, respectively, where both positive and negative image-text pairs are constructed for the target distorted image. Subsequently, image-text correspondences are learned in the embedding space while simultaneously evaluating image quality. Notably, in the auxiliary task learning, the proposed method not only brings the image and its corresponding positive text prompt closer but also pushes away the image from its negative text prompts, thereby facilitating the extraction of pertinent distortion features. In the quality assessment task, a patch-wise strategy is employed during the training phase. Differing from conventional BIQA methods, a novel NSS-guided quality weighting is introduced to gauge the correlation between patch quality and global quality, thereby enabling precise quality prediction. Extensive experiments are conducted on six IQA datasets, and the experimental results verify the superiority of the proposed method.

How humans understand and recognize the actions of others is a complex neuroscientific problem that involves a combination of cognitive mechanisms and neural networks. Research has shown that humans have brain areas that recognize actions that process top-down attentional information, such as the temporoparietal association area. Also, humans have brain regions dedicated to understanding the minds of others and analyzing their intentions, such as the medial prefrontal cortex of the temporal lobe. Skeleton-based action recognition creates mappings for the complex connections between the human skeleton movement patterns and behaviors. Although existing studies encoded meaningful node relationships and synthesized action representations for classification with good results, few of them considered incorporating a priori knowledge to aid potential representation learning for better performance. LA-GCN proposes a graph convolution network using large-scale language models (LLM) knowledge assistance. First, the LLM knowledge is mapped into a priori global relationship (GPR) topology and a priori category relationship (CPR) topology between nodes. The GPR guides the generation of new “bone” representations, aiming to emphasize essential node information from the data level. The CPR mapping simulates category prior knowledge in human brain regions, encoded by the PC-AC module and used to add additional supervision—forcing the model to learn class-distinguishable features. In addition, to improve information transfer efficiency in topology modeling, we propose multi-hop attention graph convolution. It aggregates each node's k-order neighbor simultaneously to speed up model convergence. LA-GCN reaches state-of-the-art on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets.

Zero-shot human skeleton-based action recognition aims to construct a model that can recognize actions outside the categories seen during training. Previous research has focused on aligning sequences' visual and semantic spatial distributions. However, these methods extract semantic features simply. They ignore that proper prompt design for rich and fine-grained action cues can provide robust representation space clustering. In order to alleviate the problem of insufficient information available for skeleton sequences, we design an information compensation learning framework from an information-theoretic perspective to improve zero-shot action recognition accuracy with a multi-granularity semantic interaction mechanism. Inspired by ensemble learning, we propose a multi-level alignment (MLA) approach to compensate information for action classes. MLA aligns multi-granularity embeddings with visual embedding through a multi-head scoring mechanism to distinguish semantically similar action names and visually similar actions. Furthermore, we introduce a new loss function sampling method to obtain a tight and robust representation. Finally, these multi-granularity semantic embeddings are synthesized to form a proper decision surface for classification. Significant action recognition performance is achieved when evaluated on the challenging NTU RGB+D, NTU RGB+D 120, and PKU-MMD benchmarks and validate that multi-granularity semantic features facilitate the differentiation of action clusters with similar visual features.

In the era of big data, the problem of facial beauty prediction (FBP) has been addressed using a combination of deep learning and esthetics based on data and models. Most existing methods are based on 2-D unimodal information processing. Owing to the high cost of 3-D data acquisition equipment, studies on the use of multimodal features of 2-D and 3-D for esthetic evaluation are scarce. Moreover, most existing methods are based on self-built 3-D datasets, which are limited to practical application scenarios of 2-D facial images. This study proposed a label distribution-based multimodal facial esthetic analysis framework (LDMFE). The LDMFE performed facial esthetic evaluation by combining 2-D and 3-D information following the process used by the human brain to conduct the 3-D esthetic evaluation. FBP was performed by extracting facial depth structure information using a depth information extraction network, DIENet, which comprises a facial structure perception layer (FSP-Layer) and an attention decision block (AD-Block). Furthermore, to ensure a high degree of agreement between the predicted label distribution of the network and the true distribution, a simple and efficient distribution measurement loss function called ${\mathcal {L}}_{\text {WD}}$ was proposed. Compared with the label distribution-based FBP loss and the latest FBP loss, ${\mathcal {L}}_{\text {WD}}$ was more stable and effective. The performance of LDMFE was evaluated using three datasets. The experimental results demonstrate that the LDMFE exhibits state-of-the-art performance.

An undirected weighted graph (UWG) is the fundamental data representation in various real applications. A graph convolution network is frequently utilized for representation learning to a UWG. Nevertheless, existing graph convolutional networks (GCNs) only consider a node's neighborhood during the embedding propagation, which regrettably decreases its representation learning capability due to the information loss in the modeling phase. Motivated by this discovery, this study proposes an adaptive neighborhood-resonated graph convolution network (ANR-GCN) with the following ideas: 1) establishing the weighted embedding propagation with the consideration of link weights in a UWG, thereby incorporating the interaction strength of each node pair into the ANR-GCN model; 2) building the neighborhood-regularization (NR) to make each node resonate with its neighborhoods, thus reinforcing the informative neighborhood information for improving the ANR-GCN's representation capability to the complex topology of the target UWG; and 3) diversifying the NR effects following the attention principle for guaranteeing the ANR-GCN's learning capacity. The proposed ANR-GCN's representation learning ability to a UWG is theoretically guaranteed from the perspectives of bounded generalization error and uniform stability. Extensive experiments on four UWG datasets illustrate that the proposed ANR-GCN significantly outperforms state-of-the-art GCNs in missing edge detection in a UWG, which evidently demonstrates its superior performance.

Recent blind super-resolution (BSR) methods are explored to handle unknown degradations and achieve impressive performance. However, the prevailing assumption in most BSR methods is the spatial invariance of degradation kernels across the entire image, which leads to significant performance declines when faced with spatially variant degradations caused by object motion or defocusing. Additionally, these methods do not account for the human visual system's tendency to focus differently on areas of varying perceptual difficulty, as they uniformly process each pixel during reconstruction. To cope with these issues, we propose a difficulty-guided variant degradation learning network for BSR, named difficulty-guided degradation learning (DDL)-BSR, which explores the relationship between reconstruction difficulty and degradation estimation. Accordingly, the proposed DDL-BSR consists of three customized networks: reconstruction difficulty prediction (RDP), space-variant degradation estimation (SDE), and degradation and difficulty-informed reconstruction (DDR). Specifically, RDP learns the reconstruction difficulty with the proposed reconstruction-distance supervision. Then, SDE is designed to estimate space-variant degradation kernels according to the difficulty map. Finally, both degradation kernels and reconstruction difficulty are fed into DDR, which takes into account such two prior knowledge information to guide super-resolution (SR). Experimental analysis on various synthetic datasets demonstrates that DDL-BSR invariably surpasses state-of-the-art (SOTA) methods, producing SR images with enhanced realism and texture quality. Code is available at https://github.com/JiaWang0704/DDL-BSR.

Compositional zero-shot learning (CZSL) aims to identify unobservable compositional concepts with prior knowledge of known primitives (attributes and objects). Due to distribution differences between seen and unseen components, existing methods for CZSL often ignore intrinsic variations between primitives and suffer from domain bias problems. To address this challenge, we proposed a concept-aware graph convolutional network (GCN) that utilizes cross-attentions to extract features unique to attributes and objects from paired concept-sharing inputs. The proposed model utilizes the cosine similarity between visual features and synthetic embeddings to estimate the feasibility score for each unseen composition. This score is then employed as a weight in the graph adjacency matrix. Additionally, the proposed model incorporates the Earth mover's distance (EMD) to further limit the concept of learning interest in disentanglers. Experimental results on three challenging dataset benchmarks, including UT-Zappos 50K, C-GQA, and MIT-States, demonstrate that the proposed model outperforms prior work in both closed- and open-world CZSL (OW-CZSL).

The creation of high-quality artistic portraits is a critical and desirable task in the field of computer vision. While recent3-D generative models have achieved impressive results in generating images with view consistency and intricate 3-D shapes, their application for generating artistic portraits is often more challenging than 2-D generative models due to the potentially destructive impact of 3-D structures on human faces. This article introduces a novel approach that leverages a meticulously designed domain feature extraction module to extract the specific feature information from both the source natural face domain and the target artistic portrait domain. These extracted features are seamlessly integrated into a 3-D representation, generating multiview consistent 3-D artistic portraits. To fuse the features of the source and target domains better, we propose a new module for domain adaptation. This module adds a path to the style path established by StyleGAN to introduce the artistic portrait domain information and regulate the target domain's feature information in $\mathcal {S}$ space. Our domain adaptation module is implemented in each StyleBlock of the 3-D representation generator to integrate the target domain information with the original facial information. Experimental results demonstrate that our approach generates high-quality 3-D artistic portraits that outperform existing approaches in preserving 3-D geometric information and multiview consistency.

Quantizing neural network is an efficient model compression technique that converts weights and activations from floating-point to integer. However, existing model quantization methods are primarily designed for high-level visual tasks. They do not sufficiently consider the unique characteristics of feature distribution in image super-resolution (SR) reconstruction models. On the one hand, the objective of SR is to restore high-frequency and fine-detail information while preserving the overall feature distribution. Therefore, the regularization techniques are removed to maintain the original distribution. However, vanilla quantization methods often employ regularization techniques to normalize the features for stable network training, which destroys the inherent information of the feature distribution. On the other hand, the feature distribution in SR models exhibits a nonuniform bell-shaped form. Common quantization methods adopt a uniform quantization strategy with equal quantization intervals. This fails to effectively capture the nonuniform feature distribution in SR. To address the above issue, we propose a novel method named Inter and Intra Nonuniform Quantization, which takes into account the specific characteristics of the feature distribution in the context of SR reconstruction models. Additionally, we propose a weight adjustment method called flex-scale-weight-adjust (FSWA). It can maintain the diversity of weight information and reduce quantization errors. Extensive experiments demonstrate that our proposed method surpasses other quantization methods in both the evaluation of reconstruction metrics and visual reconstruction performance.

An autonomous underwater vehicle (AUV) has shown impressive potential and promising exploitation prospects in numerous marine missions. Among its various applications, the most essential prerequisite is path planning. Although considerable endeavors have been made, there are several limitations. A complete and realistic ocean simulation environment is critically needed. As most of the existing methods are based on mathematical models, they suffer from a large gap with reality. At the same time, the dynamic and unknown environment places high demands on robustness and generalization. In order to overcome these limitations, we propose an information-assisted reinforcement learning path planning scheme. First, it performs numerical modeling based on real ocean current observations to establish a complete simulation environment with the grid method, including 3-D terrain, dynamic currents, local information, and so on. Next, we propose an information compression (IC) scheme to trim the mutual information (MI) between reinforcement learning neural network layers to improve generalization. A proof based on information theory provides solid support for this. Moreover, for the dynamic characteristics of the marine environment, we elaborately design a confidence evaluator (CE), which evaluates the correlation between two adjacent frames of ocean currents to provide confidence for the action. The performance of our method has been evaluated and proven by numerical results, which demonstrate a fair sensitivity to ocean currents and high robustness and generalization to cope with the dynamic and unknown underwater environment.

Granular-ball support vector machine (GBSVM) is a significant attempt to construct a classifier using the coarse-to-fine granularity of a granular ball as input, rather than a single data point. It is the first classifier whose input contains no points. However, the existing model has some errors, and its dual model has not been derived. As a result, the current algorithm cannot be implemented or applied. To address these problems, we fix the errors of the original model of the existing GBSVM and derive its dual model. Furthermore, a particle swarm optimization (PSO) algorithm is designed to solve the dual problem. The sequential minimal optimization (SMO) algorithm is also carefully designed to solve the dual problem. The latter is faster and more stable. The experimental results on the UCI benchmark datasets demonstrate that GBSVM is more robust and efficient. All codes have been released in the open source library available at: http://www.cquptshuyinxia.com/GBSVM.html or https://github.com/syxiaa/GBSVM.

Traditional clustering algorithms often focus on the most fine-grained information and achieve clustering by calculating the distance between each pair of data points or implementing other calculations based on points. This way is not inconsistent with the cognitive mechanism of "global precedence" in the human brain, resulting in those methods' bad performance in efficiency, generalization ability, and robustness. To address this problem, we propose a new clustering algorithm called granular-ball clustering via granular-ball computing. First, clustering algorithm based on granular-ball (GBCT) generates a smaller number of granular-balls to represent the original data and forms clusters according to the relationship between granular-balls, instead of the traditional point relationship. At the same time, its coarse-grained characteristics are not susceptible to noise, and the algorithm is efficient and robust; besides, as granular-balls can fit various complex data, GBCT performs much better in nonspherical datasets than other traditional clustering methods. The completely new coarse granularity representation method of GBCT and cluster formation mode can also be used to improve other traditional methods. All codes can be available at https://github.com/wylbdthxbw/GBC.

Pawlak rough set (PRS) and neighborhood rough set (NRS) are the two most common rough set theoretical models. Although the PRS can use equivalence classes to represent knowledge, it is unable to process continuous data. On the other hand, NRSs, which can process continuous data, rather lose the ability of using equivalence classes to represent knowledge. To remedy this deficit, this article presents a granular-ball rough set (GBRS) based on the granular-ball computing combining the robustness and the adaptability of the granular-ball computing. The GBRS can simultaneously represent both the PRS and the NRS, enabling it not only to be able to deal with continuous data and to use equivalence classes for knowledge representation as well. In addition, we propose an implementation algorithm of the GBRS by introducing the positive region of GBRS into the PRS framework. The experimental results on benchmark datasets demonstrate that the learning accuracy of the GBRS has been significantly improved compared with the PRS and the traditional NRS. The GBRS also outperforms nine popular or the state-of-the-art feature selection methods. We have open-sourced all the source codes of this article at https://www.cquptshuyinxia.com/GBRS.html, https://github.com/syxiaa/GBRS.

Binary neural network (BNN) is an effective approach to reduce the memory usage and the computational complexity of full-precision convolutional neural networks (CNNs), which has been widely used in the field of deep learning. However, there are different properties between BNNs and real-valued models, making it difficult to draw on the experience of CNN composition to develop BNN. In this article, we study the application of binary network to the single-image super-resolution (SISR) task in which the network is trained for restoring original high-resolution (HR) images. Generally, the distribution of features in the network for SISR is more complex than those in recognition models for preserving the abundant image information, e.g., texture, color, and details. To enhance the representation ability of BNN, we explore a novel activation-rectified inference (ARI) module that achieves a more complete representation of features by combining observations from different quantitative perspectives. The activations are divided into several parts with different quantification intervals and are inferred independently. This allows the binary activations to retain more image detail and yield finer inference. In addition, we further propose an adaptive approximation estimator (AAE) for gradually learning the accurate gradient estimation interval in each layer to alleviate the optimization difficulty. Experiments conducted on several benchmarks show that our approach is able to learn a binary SISR model with superior performance over the state-of-the-art methods. The code will be released at https://github.com/jwxintt/Rectified-BSR.

Multimodal person reidentification (ReID), which aims to learn modality-complementary information by utilizing multimodal images simultaneously for person retrieval, is crucial for achieving all-time and all-weather monitoring. Existing methods try to address this issue through modality fusion to absorb complementary information. However, most of these methods are limited to the spatial domain only and usually overlook the intra-/intermodal interactions during feature fusion, resulting in insufficient learning of modality-specific and complementary information. To address these issues, we propose a tri-interaction enhancement network (TIENet), which contains three modules: spatial-frequency interaction (SFI), intermodal mask interaction (IMMI), and intramodal feature fusion (IMFF). Specifically, the SFI boosts the modality-specific representation by integrating the amplitude-guided attention mechanism into the phase space, combined with spatial-domain convolution to achieve fine-grained information learning. Meanwhile, the IMMI enhances the richness of the feature descriptors by embedding the intermodal relationships to preserve complementary information. Finally, the IMFF module considers the structure of the human body and integrates intramodal contextual information. Extensive experimental results demonstrate the effectiveness of our method, achieving superior performances on RGBNT201 and MARKET1501_RGBNT datasets.

Deep learning methods have achieved impressive performance in compressed video quality enhancement tasks. However, these methods rely excessively on practical experience by manually designing the network structure and do not fully exploit the potential of the feature information contained in the video sequences, i.e., not taking full advantage of the multiscale similarity of the compressed artifact information and not seriously considering the impact of the partition boundaries in the compressed video on the overall video quality. In this article, we propose a novel Mixed Difference Equation inspired Transformer (MDEformer) for compressed video quality enhancement, which provides a relatively reliable principle to guide the network design and yields a new insight into the interpretable transformer. Specifically, drawing on the graphical concept of the mixed difference equation (MDE), we utilize multiple cross-layer cross-attention aggregation (CCA) modules to establish long-range dependencies between encoders and decoders of the transformer, where partition boundary smoothing (PBS) modules are inserted as feedforward networks. The CCA module can make full use of the multiscale similarity of compression artifacts to effectively remove compression artifacts, and recover the texture and detail information of the frame. The PBS module leverages the sensitivity of smoothing convolution to partition boundaries to eliminate the impact of partition boundaries on the quality of compressed video and improve its overall quality, while not having too much impacts on non-boundary pixels. Extensive experiments on the MFQE 2.0 dataset demonstrate that the proposed MDEformer can eliminate compression artifacts for improving the quality of the compressed video, and surpasses the state-of-the-arts (SOTAs) in terms of both objective metrics and visual quality.

Infrared small target detection (IRSTD) refers to detecting faint targets in infrared (IR) images, which has achieved notable progress with the advent of deep learning. However, the drive for improved detection accuracy has led to larger, intricate models with redundant parameters, causing storage and computation inefficiencies. In this pioneering study, we introduce the concept of utilizing network pruning to enhance the efficiency of IRSTD. Due to the challenge posed by low signal-to-noise ratios (SNRs) and the absence of detailed semantic information in IR images, directly applying existing pruning techniques yields suboptimal performance. To address this, we propose a novel wavelet structure-regularized multidimensional musical scale soft channel pruning (SCP) method, giving rise to the efficient IRPruneDeXt model. Our approach involves representing the weight matrix in the wavelet domain and formulating a wavelet channel pruning (WCP) strategy. We incorporate wavelet regularization to induce structural sparsity without incurring extra memory usage. Additionally, we design a multidimensional musical scale soft channel reconstruction (MMSCR) method that adapts the strategy across temporal and spatial dimensions to preserve key target information and prevent premature pruning. By leveraging interactions between criteria, it balances pruning and reconstruction through a musical scale feedback effect, achieving an optimal sparse structure while maintaining overall sparsity. Through extensive experiments on many widely used benchmarks, our IRPruneDeXt method surpasses established techniques in both model complexity and accuracy. Specifically, when employing U-net as the baseline network, IRPruneDeXt achieves a 65.68% reduction in parameters and a 51.77% decrease in floating-point operations (FLOPs) while improving intersection over union (IoU) from 73.31% to 76.17% and normalized IoU (nIoU) from 70.92% to 75.08%. The code is available at github.com/hd0013/IRPruneDet.

Infrared small target detection (IRSTD) aims to identify and locate small targets amidst background noise. It is highly valuable in various practical application domains, such as maritime rescue and early warning systems deployed in challenging conditions such as harsh weather, low illumination, and long imaging distances. Different from existing works that either adopt well-designed backbone networks or devise specific modules to improve them from different aspects, in this article, we formulate the learning process of IRSTD from a novel perspective, i.e., the mechanism of pixel movement. Considering that the movement of pixels passing through the layers of the network for IRSTD can be analogized to the flow of particles in a fluid dynamic system, we propose a computational fluid dynamic network (CFD-Net) derived from computational fluid dynamics. Technically, we leverage the superiority of the unilateral difference equation with third-order accuracy and devise a unilateral differential residual structure as the backbone of CFD-Net. This design ensures that the pixel stream only flows in the forward direction. In addition, a switch-controlled multidirectional treatment tank (SMTT) is introduced to CFD-Net to dynamically guide the pixel stream to the appropriate path for different targets with varying shapes and orientations, facilitating learning robust target representation and improving detection performance. The proposed CFD-Net is evaluated on the IRSTD-1k and SIRST datasets and is found to outperform existing state-of-the-art (SOTA) methods.

Face stylization has made notable progress in recent years. However, when training on limited data, the performance of existing approaches significantly declines. Although some studies have attempted to tackle this problem, they either failed to achieve the few-shot setting (less than 10) or can only get suboptimal results. In this article, we propose GAN Prior Distillation (GPD) to enable effective few-shot face stylization. GPD contains two models: a teacher network with GAN Prior and a student network that fulfills end-to-end translation. Specifically, we adapt the teacher network trained on large-scale data in the source domain to the target domain using a handful of samples, where it can learn the target domain's knowledge. Then, we can achieve few-shot augmentation by generating source domain and target domain images simultaneously with the same latent codes. We propose an anchor-based knowledge distillation module that can fully use the difference between the training and the augmented data to distill the knowledge of the teacher network into the student network. The trained student network achieves excellent generalization performance with the absorption of additional knowledge. Qualitative and quantitative experiments demonstrate that our method achieves superior results than state-of-the-art approaches in a few-shot setting.

With the rapid development of the Internet, various types of anomaly traffic are threatening network security. However, the difficulty of collecting and labelling anomalous traffic is a significant challenge, so this paper proposes a semi-supervised anomaly detection framework. Considering normal and abnormal traffic have different data distributions, our framework can generate pseudo anomaly samples without prior knowledge of anomalies to achieve the detection of anomaly data. The framework comprises three principal components. Firstly, a pre-trained feature extractor is employed to extract a feature representation of the network traffic. Secondly, a bidirectional normalizing flow module establishes a reversible transformation between the latent data distribution and a Gaussian space. Through this bidirectional mapping, samples first undergo transformation manipulation within the Gaussian distribution space, and are then transported through the generative direction of normalizing flows, translating mathematical transformations into semantic feature evolutions in the latent data space. Finally, a simple classifier explicitly learns the potential differences between anomaly and normal samples to facilitate better anomaly detection. During inference, our framework requires only two modules to detect anomalous samples, leading to a considerable reduction in model size. According to the experiments, our method achieves the state-of-the-art results on the common benchmarking datasets of anomaly network traffic detection. Furthermore, it exhibits good generalisation performance across datasets.

The global multi-object tracking (MOT) system can consider interaction, occlusion, and other ``visual blur'' scenarios to ensure effective object tracking in long videos. Among them, graph-based tracking-by-detection paradigms achieve surprising performance. However, their fully-connected nature poses storage space requirements that challenge algorithm handling long videos. Currently, commonly used methods are still generated trajectories by building one-forward associations across frames. Such matches produced under the guidance of first-order similarity information may not be optimal from a longer-time perspective. Moreover, they often lack an end-to-end scheme for correcting mismatches. This paper proposes the Composite Node Message Passing Network (CoNo-Link), a multi-scene generalized framework for modeling ultra-long frames information for association. CoNo-Link's solution is a low-storage overhead method for building constrained connected graphs. In addition to the previous method of treating objects as nodes, the network innovatively treats object trajectories as nodes for information interaction, improving the graph neural network's feature representation capability. Specifically, we formulate the graph-building problem as a top-k selection task for some reliable objects or trajectories. Our model can learn better predictions on longer-time scales by adding composite nodes. As a result, our method outperforms the state-of-the-art in several commonly used datasets.

Adversarial attacks involve adding perturbations to the source image to cause misclassification by the target model, which demonstrates the potential of attacking face recognition models. Existing adversarial face image generation methods still can’t achieve satisfactory performance because of low transferability and high detectability. In this paper, we propose a unified framework Adv-Diffusion that can generate imperceptible adversarial identity perturbations in the latent space but not the raw pixel space, which utilizes strong inpainting capabilities of the latent diffusion model to generate realistic adversarial images. Specifically, we propose the identity-sensitive conditioned diffusion generative model to generate semantic perturbations in the surroundings. The designed adaptive strength-based adversarial perturbation algorithm can ensure both attack transferability and stealthiness. Extensive qualitative and quantitative experiments on the public FFHQ and CelebA-HQ datasets prove the proposed method achieves superior performance compared with the state-of-the-art methods without an extra generative model training process. The source code is available at https://github.com/kopper-xdu/Adv-Diffusion.

Recently MLP-based methods have shown strong performance in point cloud analysis. Simple MLP architectures are able to learn geometric features in local point groups yet fail to model long-range dependencies directly. In this paper, we propose Point Deformable Network (PDNet), a concise MLP-based network that can capture long-range relations with strong representation ability. Specifically, we put forward Point Deformable Aggregation Module (PDAM) to improve representation capability in both long-range dependency and adaptive aggregation among points. For each query point, PDAM aggregates information from deformable reference points rather than points in limited local areas. The deformable reference points are generated data-dependent, and we initialize them according to the input point positions. Additional offsets and modulation scalars are learned on the whole point features, which shift the deformable reference points to the regions of interest. We also suggest estimating the normal vector for point clouds and applying Enhanced Normal Embedding (ENE) to the geometric extractors to improve the representation ability of single-point. Extensive experiments and ablation studies on various benchmarks demonstrate the effectiveness and superiority of our PDNet.

Infrared Small Target Detection (IRSTD) refers to detecting faint targets in infrared images, which has achieved notable progress with the advent of deep learning. However, the drive for improved detection accuracy has led to larger, intricate models with redundant parameters, causing storage and computation inefficiencies. In this pioneering study, we introduce the concept of utilizing network pruning to enhance the efficiency of IRSTD. Due to the challenge posed by low signal-to-noise ratios and the absence of detailed semantic information in infrared images, directly applying existing pruning techniques yields suboptimal performance. To address this, we propose a novel wavelet structure-regularized soft channel pruning method, giving rise to the efficient IRPruneDet model. Our approach involves representing the weight matrix in the wavelet domain and formulating a wavelet channel pruning strategy. We incorporate wavelet regularization to induce structural sparsity without incurring extra memory usage. Moreover, we design a soft channel reconstruction method that preserves important target information against premature pruning, thereby ensuring an optimal sparse structure while maintaining overall sparsity. Through extensive experiments on two widely-used benchmarks, our IRPruneDet method surpasses established techniques in both model complexity and accuracy. Specifically, when employing U-net as the baseline network, IRPruneDet achieves a 64.13% reduction in parameters and a 51.19% decrease in FLOPS, while improving IoU from 73.31% to 75.12% and nIoU from 70.92% to 74.30%. The code is available at https://github.com/hd0013/IRPruneDet.

Domain Generalization (DG) aims to develop a versatile model capable of performing well on unseen target domains. Recent advancements in pre-trained Visual Foundation Models (VFMs), such as CLIP, show significant potential in enhancing the generalization abilities of deep models. Although there is a growing focus on VFM-based domain prompt tuning for DG, effectively learning prompts that disentangle invariant features across all domains remains a major challenge. In this paper, we propose addressing this challenge by leveraging the controllable and flexible language prompt of the VFM. Observing that the text modality of VFMs is inherently easier to disentangle, we introduce a novel text feature guided visual prompt tuning framework. This framework first automatically disentangles the text prompt using a large language model (LLM) and then learns domain-invariant visual representation guided by the disentangled text feature. Moreover, we also devise domain-specific prototype learning to fully exploit domain-specific information to combine with the invariant feature prediction. Extensive experiments on mainstream DG datasets, namely PACS, VLCS, OfficeHome, DomainNet and TerraInc, demonstrate that the proposed method achieves superior performances to state-of-the-art DG methods.

In this paper, we addressed facial aesthetic enhancement (FAE). Although existing methods have made great progress, the beautified images generated by them are highly prone to poor beautification, which limits their application to real-world scenes. To tackle this problem, we proposed a new method called the facial aesthetic enhancement network for Asian faces based on differential facial aesthetic activations (Diff-FANet), which comprises three important modules: aesthetic average difference perception block (ADP), aesthetic difference evaluation block (ADE), and aesthetic fusion optimization block (AFO). ADP learns the transformation of the latent code of an image before and after beautification. The ADE learns the features of an enhanced image, which guides image fusion. The AFO was used to eliminate ghosting. To evaluate the effectiveness of Diff-FANet, we utilized the wedding dataset for training and the SCUT-FBP5500 and Asian face datasets for testing. The results of experiments revealed that Diff-FANet achieved excellent results.

In-air handwriting as a new human-computer interaction way plays an important role in many virtual/mixed-reality applications. Existing methods for in-air handwritten text recognition (IAHTR) typically directly process handwriting trajectories with deep neural networks. However, those methods all simply learn discriminative patterns by modelling low-level relationships between adjacent points of trajectories, while completely ignoring the inherent geometric structures of characters. Instead, we propose a novel Graph-guided Cross-modality Translator for IAHTR, which further explicitly exploits the geometric structures of characters for guiding the decoding of trajectories via graph-guided cross-modality attention mechanism without introducing extra annotation costs. Experiments on benchmarks IAHEW-UCAS2016 & IAM-OnDB show that our method has achieved state-of-the-art performance for handwritten text recognition.

Text-to-image person retrieval aims to recognize target pedestrians based on specified text. Existing methods mainly obtain image and text features separately through distinct feature extractors, subsequently embedding them into a unified feature space and calculating their similarity. Despite great success, current methods still suffer from the lack of information interaction between images and text. To address this issue, we propose Mutual-guidance Representation Learning (MGRL) for text-to-image person retrieval, which captures the key features for matching via text-image information interaction. Accordingly, our MGRL consists of two customized modules: iterative text-guided feature extraction (ITFE) and vision-assisted specific mask complement (VSMC). Specifically, ITFE is first designed to extract the matching information between the text and the image concerning the local feature attention of the target pedestrians by iterative text guidance. Then, to further ensure the image features extracted by ITFE contain the text description, VSMC is designed to utilize the extracted image features to help complete masked text where the mask is difficult to complete with only unmasked text information. Experiments are conducted on CUHK-PEDES and ICFG-PEDES datasets, and experimental results demonstrate the superiority of the proposed MGRL.

Existing weighted clustering algorithms often heavily rely on specific parameters. Specifically, in addition to the number of clusters (k), several other parameters need to be manually tuned, which greatly limits their practical applicability. The fundamental issue lies in the fact that most weighted clustering methods derive feature weights through global iterations. To address this challenge, this paper introduces a novel weighted granular-ball structure, continually optimizing weights during the ball splitting process and restricting the calculation of local data point weights to the corresponding weighted granular-ball. We employ local iterations within this structure as an approximation to global weight calculations. This method eliminates the need for parameter tuning during the weight calculation process and incidentally addresses the “curse of dimensionality” in traditional granular-ball computing model. When applied to complex real-world datasets, this method accurately represents high-dimensional data, thereby improving clustering precision and extending the adaptability of the granular-ball computing model in high-dimensional spaces. Comprehensive experimental analysis demonstrates that our W-GBC algorithm performs well in terms of clustering results and competes strongly with baseline algorithms. The code has been released and is now available at https://github.com/xjnine/W-GBC.

Current data stream clustering algorithms face low efficiency in both the online and offline phases, and struggle to address the problem of cluster boundary overlap caused by concept drift. Specifically, in the online phase, the majority of existing data stream clustering algorithms require each newly arriving sample to be scanned and inserted into the appropriate micro-clusters. In offline clustering, algorithms typically require all sample points as input. Moreover, most data stream clustering algorithms struggle to effectively deal with the problem of cluster boundary overlap caused by the concept drift. To tackle these challenges, we use a granular-ball structure for the coarse-grained representation of data stream. This structure eliminates the need for computations on all data points in both the online and offline phases. Additionally, we introduce fuzziness into the granular-ball structure to resolve the issue of cluster boundary overlap caused by the concept drift. Experimental results on both synthetic and real-world datasets demonstrate that our approach achieves efficient and accurate clustering performance when compared to existing data stream clustering algorithms. Our source code is publicly available at https://github.com/xjnine/GBFuzzyStream.

Data mixing augmentation has been widely applied to improve the generalization ability of deep neural networks. Recently, offline data mixing augmentation, e.g. handcrafted and saliency information-based mixup, has been gradually replaced by automatic mixing approaches. Through minimizing two sub-tasks, namely, mixed sample generation and mixup classification in an end-to-end way, AutoMix significantly improves accuracy on image classification tasks. However, as the optimization objective is consistent for the two sub-tasks, this approach is prone to generating consistent instead of diverse mixed samples, which results in overfitting for target task training. In this paper, we propose AdAutomixup, an adversarial automatic mixup augmentation approach that generates challenging samples to train a robust classifier for image classification, by alternatively optimizing the classifier and the mixup sample generator. AdAutomixup comprises two modules, a mixed example generator, and a target classifier. The mixed sample generator aims to produce hard mixed examples to challenge the target classifier, while the target classifier's aim is to learn robust features from hard mixed examples to improve generalization. To prevent the collapse of the inherent meanings of images, we further introduce an exponential moving average (EMA) teacher and cosine similarity to train AdAutomixup in an end-to-end way. Extensive experiments on seven image benchmarks consistently prove that our approach outperforms the state of the art in various classification scenarios. The source code is available at https://github.com/JinXins/Adversarial-AutoMixup.

Image-to-image (I2I) translation is vital in computer vision tasks like style transfer and domain adaptation. While recent advances in GAN have enabled high-quality sample generation, real-world challenges such as noise and distortion remain significant obstacles. Although Gaussian noise injection during training has been utilized, its theoretical underpinnings have been unclear. This work provides a robust theoretical framework elucidating the role of Gaussian noise injection in I2I translation models. We address critical questions on the influence of noise variance on distribution divergence, resilience to unseen noise types, and optimal noise intensity selection. Our contributions include connecting $f$-divergence and score matching, unveiling insights into the impact of Gaussian noise on aligning probability distributions, and demonstrating generalized robustness implications. We also explore choosing an optimal training noise level for consistent performance in noisy environments. Extensive experiments validate our theoretical findings, showing substantial improvements over various I2I baseline models in noisy settings. Our research rigorously grounds Gaussian noise injection for I2I translation, offering a sophisticated theoretical understanding beyond heuristic applications.

In this paper, we propose a novel framework, Modality-Free Violence Detection (MFVD), which captures the causal relationships among multimodal cues and ensures stable performance even in the absence of audio information. Specifically, we design a novel Cross-Modal Causal Attention mechanism (CCA) to deal with modality asynchrony by utilizing relative temporal distance and semantic correlation to obtain causal attention between audio and visual information instead of merely calculating correlation scores between audio and visual features. Moreover, to ensure our framework can work well when the audio modality is missing, we design a Cross-Modal Feature Distillation module (CFD), leveraging the common parts of the fused features obtained from CCA to guide the enhancement of visual features. Experimental results on the XD-Violence dataset demonstrate the superior performance of the proposed method in both vision-only and audio-visual modalities, surpassing state-of-the-art methods for both tasks.

In this paper, we propose Dual Space Embedding Learning (DSEL) for weakly supervised audio-visual violence detection, which excavates violence information deeply in both Euclidean and Hyperbolic spaces to distinguish violence from non-violence semantically and alleviate the asynchronous issue of violent cues in audio-visual patterns. Specifically, we first design a dual space visual feature interaction module (DSVFI) to deeply investigate the violence information in visual modality, which contains richer information compared to audio counterpart. Then, considering the modality asynchrony between the two modalities, we employ a late modality fusion method and design an asynchrony-aware audio-visual fusion module (AAF), in which visual features receive the violent prompt from the audio features after interacting among snippets and learning the violence information from each other. Experimental results show that our method achieves state-of-the-art performance on XD-Violence.

With the rise of large-scale model applications, leveraging these models as the base network for efficient transfer learning has garnered increasing attention. Currently, parameter-efficient transfer learning methods have made significant improvements in reducing the number of trainable parameters but introduce latency during inference. In this study, we propose an enhanced adaptation of the adapter using a reparameterization technique, revamping the activating layers into linear layers. This modification retains the high-dimensional fine-tuning capability of the adapter for visual tasks while avoiding additional inference latency. We name this plug-and-play module the Re3adapter, which optimizes the model with only 0.26% of the parameters and introduces no inference latency. Experimental results demonstrate its clear advantages in traditional classification and medical tasks.

Point Cloud Registration (PCR) is fundamental for the automatic perception of our space. With the rapid development of deep neural network, the community has swiftly adapted to this data-driven technique, and achieved promising performances. However, most existing learning-based methods attempt to conduct PCR within specific ideal experimental settings, in which the ground truth transformations are accessible and most of the data points have one-to-one correspondences. But in real-world scenarios, the GT transformations are often unknown, and point clouds may only share partially overlapped regions. It leads us to a challenging yet practical issue: How to perform Partial-to-Partial (PtP) Point Cloud Registration without pre-acquired supervisions? In this paper, we aim to tackle both challenges under a unified framework. To achieve this, we propose a novel Focal Anchor Generator to emulate the human perceptual process, particularly focusing on the mutual cloud parts. On top of it, a set of Multi-Consistency constraints are introduced to equip our model with the unsupervised learning ability, which is highly applicable. Extensive experiments have demonstrated the distinctive quality of our proposed framework. We believe this work will broaden the scope of PCR research and enhance the applicative potential of PCR algorithms. (The project code has been released on github.com/chengxiaojin/FGMC-UPCR).

Semi-supervised image semantic segmentation, a vital but challenging task in multimedia applications, aims to accurately classify pixels with limited labeled data. Traditional approaches in this domain often grapple with the confirmation bias problem, where models, influenced by their own predictions, become prone to replicating errors. To address this critical issue, our research introduces a cross-network-crossview consistency learning framework. This novel paradigm significantly reduce the confirmation bias through diversifying the learning perspectives. Integral to our approach are two components: a pseudo-label validation and filtering mechanism, and a cross-contrastive learning module within the feature domain. These elements work in synergy to not only amplify the accuracy of the model but also its robustness against varied data scenarios. Extensive evaluations, conducted across multiple datasets, clearly demonstrate the effectiveness of our method. In comparison to existing state-of-the-art models, our approach exhibits marked improvements, especially in the challenging contexts of semisupervised image semantic segmentation. The code is available at https://github.com/Sstar2orchid/C3T.

Continual learning (CL) aims to learn from sequentially arriving tasks without catastrophic forgetting (CF). By partitioning the network into two parts based on the Lottery Ticket Hypothesis—one for holding the knowledge of the old tasks while the other for learning the knowledge of the new task—the recent progress has achieved forget-free CL. Although addressing the CF issue well, such methods would encounter serious under-fitting in long-term CL, in which the learning process will continue for a long time and the number of new tasks involved will be much higher. To solve this problem, this paper partitions the network into three parts—with a new part for exploring the knowledge sharing between the old and new tasks. With the shared knowledge, this part of network can be learnt to simultaneously consolidate the old tasks and fit to the new task. To achieve this goal, we propose a task-aware Orthogonal Sparse Network (OSN), which contains shared knowledge induced network partition and sharpness-aware orthogonal sparse network learning. The former partitions the network to select shared parameters, while the latter guides the exploration of shared knowledge through shared parameters. Qualitative and quantitative analyses, show that the proposed OSN induces minimum to no interference with past tasks, i.e., approximately no forgetting, while greatly improves the model plasticity and capacity, and finally achieves the state-of-the-art performances.

Cloth-Changing Person Re-Identification (CC-ReID) aims to accurately identify a target person in the more realistic surveillance scenario where clothes of the pedestrian may change drastically, which is critical in public security systems for tracking down disguised criminal suspects. Existing methods mainly transform the CC-ReID problem into cross-modality feature alignment from the data-driven perspective, without modelling the interference factors such as clothes and camera view changes meticulously. This may lead to over-consideration or under-consideration of the influence of these factors on the extraction of robust and discriminative identity features. This paper proposes a novel algorithm for thoroughly disentangling identity features from interference factors brought by clothes and camera view changes while ensuring the robustness and discriminability. It adopts a dual-stream identity feature learning framework consisting of a raw image stream and a cloth-erasing stream, to explore discriminative and cloth-irrelevant identity feature representations. Specifically, an adaptive cloth-irrelevant contrastive objective is introduced to contrast features extracted by the two streams, aiming to suppress the fluctuation caused by clothes textures in the identity feature space. Moreover, we innovatively mitigate the influence of the interference factors through a generative adversarial interference factor decoupling network. This network is targeted at capturing identity-related information residing in the interference factors and disentangling the identity features from such information. Extensive experimental results demonstrate the effectiveness of the proposed method, achieving superior performances to state-of-the-art methods.

One of the serious impacts brought by artificial intelligence is the abuse of deepfake techniques. Despite the proliferation of deepfake detection methods aimed at safeguarding the authenticity of media across the Internet, they mainly consider the improvement of detector architecture or the synthesis of forgery samples. The forgery perceptions, including the feature responses and prediction scores for forgery samples, have not been well considered. As a result, the generalization across multiple deepfake techniques always comes with complicated detector structures and expensive training costs. In this paper, we shift the focus to real-time perception analysis in the training process and generalize deepfake detectors through an efficient method dubbed Forgery Perception Guidance (FPG). In particular, after investigating the deficiencies of forgery perceptions, FPG adopts a sample refinement strategy to pertinently train the detector, thereby elevating the generalization efficiently. Moreover, FPG introduces more sample information as explicit optimizations, which makes the detector further adapt the sample diversities. Experiments demonstrate that FPG improves the generality of deepfake detectors with small training costs, minor detector modifications, and the acquirement of real data only. In particular, our approach not only outperforms the state-of-the-art on both the cross-dataset and cross-manipulation evaluation but also surpasses the baseline that needs more than 3× training time.

Snapshot spectral compressive imaging can capture spectral information across multiple wavelengths in one imaging. The coded aperture snapshot spectral imaging (CASSI) method, aims to recover 3D spectral cubes from 2D measurements. Most existing approaches employ a deep unfolding framework based on Transformer, which alternately address a data subproblem and a prior subproblem. However, these frameworks lack flexibility regarding the sensing matrix and inter-stage interactions. In addition, the quadratic computational complexity of global Transformer and the restricted receptive field of local Transformer impact reconstruction efficiency and accuracy. In this paper, we propose a dynamic deep unfolding network with mamba for compressive spectral imaging, called VmambaSCI. We integrate spatial-spectral information from the sensing matrix into the data module and utilizes spatial adaptive operations in the stage interaction of the prior module. Furthermore, recognizing that the imaging process causes aliasing of spatial and spectral information, we develop a dual-domain scanning mamba (DSMamba), featuring a novel spatial-channel scanning method for enhanced efficiency and accuracy. To our knowledge, VmambaSCI is the first Mamba-based model for compressive spectral imaging. Experimental results on the public databases, CAVE and KAIST, demonstrate the superiority of the proposed VmambaSCI over the state-of-the-art approaches.

Recent advancements in deep learning have greatly advanced the field of infrared small object detection (IRSTD). Despite their remarkable success, a notable gap persists between these IRSTD methods and generic segmentation approaches in natural image domains. This gap primarily arises from the significant modality differences and the limited availability of infrared data. In this study, we aim to bridge this divergence by investigating the adaptation of generic segmentation models, such as the Segment Anything Model (SAM), to IRSTD tasks. Our investigation reveals that many generic segmentation models can achieve comparable performance to state-of-the-art IRSTD methods. However, their full potential in IRSTD remains untapped. To address this, we propose a simple, lightweight, yet effective baseline model for segmenting small infrared objects. Through appropriate distillation strategies, we empower smaller student models to outperform state-of-the-art methods, even surpassing fine-tuned teacher results. Furthermore, we enhance the model's performance by introducing a novel query design comprising dense and sparse queries to effectively encode multi-scale features. Through extensive experimentation across four popular IRSTD datasets, our model demonstrates significantly improved performance in both accuracy and throughput compared to existing approaches, surpassing SAM and Semantic-SAM by over 14 IoU on NUDT and 4 IoU on IRSTD1k. The source code and models will be released at https://github.com/O937-blip/SimIR.

The proliferation of 2D foundation models has sparked research into adapting them for open-world 3D instance segmentation. Recent methods introduce a paradigm that leverages superpoints as geometric primitives and incorporates 2D multi-view masks from Segment Anything model (SAM) as merging guidance, achieving outstanding zero-shot instance segmentation results. However, the limited use of 3D priors restricts the segmentation performance. Previous methods calculate the 3D superpoints solely based on estimated normal from spatial coordinates, resulting in under-segmentation for instances with similar geometry. Besides, the heavy reliance on SAM and hand-crafted algorithms in 2D space suffers from over-segmentation due to SAM's inherent part-level segmentation tendency. To address these issues, we propose SA3DIP, a novel method for Segmenting Any 3D Instances via exploiting potential 3D Priors. Specifically, on one hand, we generate complementary 3D primitives based on both geometric and textural priors, which reduces the initial errors that accumulate in subsequent procedures. On the other hand, we introduce supplemental constraints from the 3D space by using a 3D detector to guide a further merging process. Furthermore, we notice a considerable portion of low-quality ground truth annotations in ScanNetV2 benchmark, which affect the fair evaluations. Thus, we present ScanNetV2-INS with complete ground truth labels and supplement additional instances for 3D class-agnostic instance segmentation. Experimental evaluations on various 2D-3D datasets demonstrate the effectiveness and robustness of our approach. Our code and proposed ScanNetV2-INS dataset are available HERE.

Visible-infrared person re-identification (VIReID) is widely used in fields such as video surveillance and intelligent transportation, imposing higher demands on model security. In practice, the adversarial attacks based on VIReID aim to disrupt output ranking and quantify the security risks of models. Although numerous studies have been emerged on adversarial attacks and defenses in fields such as face recognition, person re-identification, and pedestrian detection, there is currently a lack of research on the security of VIReID systems. To this end, we propose to explore the vulnerabilities of VIReID systems and prevent potential serious losses due to insecurity. Compared to research on single-modality ReID, adversarial feature alignment and modality differences need to be particularly emphasized. Thus, we advocate for feature-level adversarial attacks to disrupt the output rankings of VIReID systems. To obtain adversarial features, we introduce \textit{Universal Adversarial Perturbations} (UAP) to simulate common disturbances in real-world environments. Additionally, we employ a \textit{Frequency-Spatial Attention Module} (FSAM), integrating frequency information extraction and spatial focusing mechanisms, and further emphasize important regional features from different domains on the shared features. This ensures that adversarial features maintain consistency within the feature space. Finally, we employ an \textit{Auxiliary Quadruple Adversarial Loss} to amplify the differences between modalities, thereby improving the distinction and recognition of features between visible and infrared images, which causes the system to output incorrect rankings. Extensive experiments on two VIReID benchmarks (i.e., SYSU-MM01, RegDB) and different systems validate the effectiveness of our method.

While numerous Video Violence Detection (VVD) methods have focused on representation learning in Euclidean space, they struggle to learn sufficiently discriminative features, leading to weaknesses in recognizing normal events that are visually similar to violent events (i.e., ambiguous violence). In contrast, hyperbolic representation learning, renowned for its ability to model hierarchical and complex relationships between events, has the potential to amplify the discrimination between visually similar events. Inspired by these, we develop a novel Dual-Space Representation Learning (DSRL) method for weakly supervised VVD to utilize the strength of both Euclidean and hyperbolic geometries, capturing the visual features of events while also exploring the intrinsic relations between events, thereby enhancing the discriminative capacity of the features. DSRL employs a novel information aggregation strategy to progressively learn event context in hyperbolic spaces, which selects aggregation nodes through layer-sensitive hyperbolic association degrees constrained by hyperbolic Dirichlet energy. Furthermore, DSRL attempts to break the cyber-balkanization of different spaces, utilizing cross-space attention to facilitate information interactions between Euclidean and hyperbolic space to capture better discriminative features for final violence detection. Comprehensive experiments demonstrate the effectiveness of our proposed DSRL.

Unsupervised out-of-distribution (OOD) detection aims to identify out-of-domain data by learning only from unlabeled In-Distribution (ID) training samples, which is crucial for developing a safe real-world machine learning system. Current reconstruction-based method provides a good alternative approach, by measuring the reconstruction error between the input and its corresponding generative counterpart in the pixel/feature space. However, such generative methods face the key dilemma, $i.e.$, improving the reconstruction power of the generative model, while keeping compact representation of the ID data. To address this issue, we propose the diffusion-based layer-wise semantic reconstruction approach for unsupervised OOD detection. The innovation of our approach is that we leverage the diffusion model's intrinsic data reconstruction ability to distinguish ID samples from OOD samples in the latent feature space. Moreover, to set up a comprehensive and discriminative feature representation, we devise a multi-layer semantic feature extraction strategy. Through distorting the extracted features with Gaussian noises and applying the diffusion model for feature reconstruction, the separation of ID and OOD samples is implemented according to the reconstruction errors. Extensive experimental results on multiple benchmarks built upon various datasets demonstrate that our method achieves state-of-the-art performance in terms of detection accuracy and speed.

Image quality assessment (IQA) aims to build a quality prediction model to assess image quality automatically rather than artificially. Due to a lack of reference images, blind image quality assessment (BIQA) has become an attractive yet challenging research topic. Inspired by the hierarchical perception mechanism in the human visual system , some existing BIQA methods aggregate multi-stage features of a convolutional neural network (CNN). However, they are regardless of the latent dependencies. To solve this problem, we propose a novel BIQA method based on hierarchical dependency learning and quality aggregation (HDLaQA). The proposed method includes multi-stage feature extraction, hierarchical dependency learning, and quality aggregation. In multi-stage feature extraction, a CNN is used as the feature extractor and multi-stage features are output for further learning. In hierarchical dependency learning, spatial and channel dependencies among the multi-stage features are modeled. To this end, a dual-head spatial dependency (DSD) module is designed to harvest the spatial dependencies between the adjacent-stage features and deliver these dependencies to the next stage. Moreover, exponential bilinear pooling (EBP) is presented to learn the channel dependencies, which is more stable than commonly used BP. In quality aggregation, multiple quality scores are predicted based on the learned dependencies, and multiple learnable weights are used to measure the importance of the predicted scores for final quality evaluation. Experimental results on seven IQA databases demonstrate the competitiveness of the proposed method on both synthetic and authentic distortions.

Deep-learning (DL)-based methods have been increasingly used in hyperspectral unmixing (HU), especially the recent trend of unsupervised autoencoder (AE) networks, which have achieved excellent performances. Although some existing unmixing methods take spatial information into account, the utilization of spatial structure is not sufficient and effective. In this letter, we present a deep attention-guided spatial–spectral network for hyperspectral image (HSI) unmixing called DASS-Net, which adopts a parallel dual-stream structure. We design a neighborhood spatial attention (NSA) module, where the abundance features of the central pixel are dynamically weighted by the coarse-grained features of the neighborhood pixels. In addition, a dual-gated mechanism is introduced to further integrate and express the spatial and spectral information. Experimental results show that the proposed DASS-Net performs particularly well in endmember extraction and outperforms all compared methods.

In the real world, data distributions often exhibit multiple granularities. However, the majority of existing neighbor-based machine-learning methods rely on manually setting a single-granularity for neighbor relationships. These methods typically handle each data point using a single-granularity approach, which severely affects their accuracy and efficiency. This paper adopts a dual-pronged approach: it constructs a multi-granularity representation of the data using the granular-ball computing model, thereby boosting the algorithm's time efficiency. It leverages the multi-granularity representation of the data to create tailored, multi-granularity neighborhood relationships for different task scenarios, resulting in improved algorithmic accuracy. The experimental results convincingly demonstrate that the proposed multi-granularity neighbor relationship effectively enhances KNN classification and clustering methods.

Despite providing high-performance solutions for computer vision tasks, the deep neural network (DNN) model has been proved to be extremely vulnerable to adversarial attacks. Current defense mainly focuses on the known attacks, but the adversarial robustness to the unknown attacks is seriously overlooked. Besides, commonly used adaptive learning and fine-tuning technique is unsuitable for adversarial defense since it is essentially a zero-shot problem when deployed. Thus, to tackle this challenge, we propose an attack-agnostic defense method named Meta Invariance Defense (MID). Specifically, various combinations of adversarial attacks are randomly sampled from a manually constructed Attacker Pool to constitute different defense tasks against unknown attacks, in which a student encoder is supervised by multi-consistency distillation to learn the attack-invariant features via a meta principle. The proposed MID has two merits: 1) Full distillation from pixel-, feature- and prediction-level between benign and adversarial samples facilitates the discovery of attack-invariance. 2) The model simultaneously achieves robustness to the imperceptible adversarial perturbations in high-level image classification and attack-suppression in low-level robust image regeneration. Theoretical and empirical studies on numerous benchmarks such as ImageNet verify the generalizable robustness and superiority of MID under various attacks.

This letter presents a novel approach to enhance image-text matching by incorporating word sense disambiguation (WSD) within the text encoder. Our method explicitly models the senses of potentially ambiguous words, refining the semantic understanding between images and text. We introduce a sense-aware mechanism for image-text alignment by integrating a lightweight WSD component into the matching framework, optimizing both tasks simultaneously. Our WSD module operates on extensive word contexts, leveraging the power of graph attention networks (GAT), and distills knowledge from a substantially larger pre-trained WSD model through multi-task learning. Our experiments demonstrate the effectiveness of augmenting original word embeddings with sense representations derived from our WSD approach. We systematically evaluate our method against several baselines and state-of-the-art approaches on two widely-used image-text matching benchmarks: MS-COCO and Flickr30K. The results illustrate significant improvements in matching accuracy, highlighting the efficacy of our proposed approach.

Non-local block (NLB) is a breakthrough technology in computer vision. It greatly boosts the capability of deep convolutional neural networks (CNNs) to capture long-range dependencies. As the critical component of NLB, non-local operation can be considered a network-based implementation of the well-known non-local means filter (NLM). Drawing on the solid theoretical foundation of NLM, we provide an innovative interpretation of the non-local operation. Specifically, it is formulated as an optimization problem regularized by Shannon entropy with a fixed parameter. Building on this insight, we further introduce an adaptive regularization strategy to enhance NLB and get a novel non-local block named ARNLB. Preliminary experiments on semantic segmentation demonstrate its effectiveness.

Most unconstrained facial expression recognition (FER) methods take original facial images as inputs to learn discriminative features by well-designed loss functions, which cannot reflect important visual information in faces. Although existing methods have explored the visual information of constrained facial expressions, there is no explicit modeling of what visual information is important for unconstrained FER. To find out valuable information of unconstrained facial expressions, we pose a new problem of no-reference de-elements learning: we decompose any unconstrained facial image into the facial expression element and a neutral face without the reference of corresponding neutral faces. Importantly, the element provides visualization results to understand important facial expression information and improves the discriminative power of features. Moreover, we propose a simple yet effectiveDe-ElementsNetwork (DENet) to learn the element and introduce appropriate constraints to overcome no ground truth of corresponding neutral faces during the de-elements learning. We extensively evaluate the proposed method on in-the-wild FER datasets including RAF-DB, AffectNet, SFEW and FERPlus. The comparable results show that our method is promising to improve classification performance and achieves equivalent performance compared with state-of-the-art methods. Also, we demonstrate the strong generalization performance on realistic occlusion and pose variation datasets and the cross-dataset evaluation.

This paper presents a fresh paradigm for protecting facial privacy via an invertible image obfuscation framework that incorporates multiple characteristics including anonymity, diversity, reversibility, security, and lightweight all at once. We name the framework PRO-Face S, an acronym for Privacy-preserving Reversible Obfuscation of Face images via Secure flow. The core of the proposed framework is a flow-based generative model (or invertible neural network), which takes as input a face image along with its pre-obfuscated form, and outputs the privacy-protected image that visually mirrors the pre-obfuscated one. The pre-obfuscation applied can be in various forms with different types and strengths. The invertibility of the flow-based model ensures that the original image can be easily recovered from the protected image in high fidelity. An elaborate secret key mechanism is devised to securely guide the mutual transformations of privacy protection and image recovery, such that the correct recovery is only possible upon the availability of the correct secret, pre-specified by the user in the protection stage. Two modes of wrong recovery are investigated to deal with malicious recovery attempts in different scenarios. Finally, extensive experiments conducted on multiple image datasets demonstrate the superiority of the proposed framework over state-of-the-art methods.

Deep learning systems typically suffer from catastrophic forgetting of old knowledge when learning from new data continually. Recently, various class incremental learning (CIL) methods have been proposed to address this issue, and some approaches achieve promising performances by relying on rehearsing the training data of previous tasks. However, storing data from previous tasks would encounter data privacy and memory issues in real-world applications. In this paper, we propose a statistical sampling adaptation method for efficient Exemplar-Free Class-Incremental Learning (EFCIL). Here, instead of preserving the images/features themselves of previous tasks/classes, we store image feature statistics from previous classes to maintain the decision boundary, which is memory-efficient and much semantic-representative. When utilizing the old-class feature statistics, we build a statistical feature adaptation network (SFAN) with a manifold consistency regularization and then train it in a transductive learning paradigm, which can map the outdated statistics onto the current feature space to facilitate a compatible and balanced classifier training subsequently. In this way, the final classifier can be jointly optimized with all the old-class features projected by SFAN and current new-class features, thus alleviating the classification bias problem in EFCIL. Experimental results greatly demonstrate the effectiveness of the proposed method, achieving superior performances than state-of-the-art approaches. Our source code is released inhttps://github.com/yxzhcv/ESSA-EFCIL.

Privacy protection has become a top priority due to the widespread collection and misuse of personal data. Anonymization and visual identity information hiding are two crucial tasks in face privacy protection, both striving to alter identifying characteristics from face images to prevent privacy information leakage. However, the goals of the two are not entirely the same. Consequently, training a model to simultaneously perform both tasks proves challenging. In this paper, we propose Diff-Privacy, a novel face privacy protection method based on diffusion models that unifies the task of anonymization and visual identity information hiding. Specifically, we present a Multi-Scale image Inversion module (MSI) that, through training, generates a set of Stable Diffusion (SD) format conditional embeddings for the original image. With these conditional embeddings, we design corresponding embedding scheduling strategies and formulate distinct energy functions during the inference process to achieve anonymization and visual identity information hiding, respectively. Extensive experiments demonstrate the effectiveness of the proposed method in protecting face privacy.

Few-shot font generation (FFG) aims to preserve the underlying global structure of the original character while generating target fonts by referring to a few samples. It has been applied to font library creation, a personalized signature, and other scenarios. Existing FFG methods explicitly disentangle content and style of reference glyphs universally or component-wisely. However, they ignore the difference between glyphs in different styles and the similarity of glyphs in the same style, which results in artifacts such as local distortions and style inconsistency. To address this issue, we propose a novel font generation approach by learning the Difference between different styles and the Similarity of the same style (DS-Font). We introduce contrastive learning to consider the positive and negative relationship between styles. Specifically, we propose a multi-layer style projector (MSP) for style encoding and realize a distinctive style representation via our proposed Cluster-level Contrastive Style (CCS) loss. The MSP module is employed to assist the generator during training to enhance the style consistency between the generated glyph and the reference glyphs. In addition, we design a glyph-independent patch discriminator, which comprehensively considers different areas of the image and ensures that each style can be distinguished independently. We conduct qualitative and quantitative evaluations comprehensively to demonstrate that our approach achieves significantly better results than state-of-the-art methods.

Accurate repetitive action counting has crucial applications in the era of AI-assisted universal fitness. Existing methods are prone to large errors in spatially fine-grained action counting scenarios. In this study, we propose a joint-wise temporal self-similarity periodic selection network (JTSPS-Net) with a human skeleton as its input. Periodic knowledge is embedded in skeleton joint units and selected in a coarse-to-fine manner to focus on the temporal repetition that occurs in the local space. The proposed JTSPS-Net adopts a temporal multiscale fusion strategy to better handle videos with various lengths. To maintain the interpretability of the model, we design an impulse map regression module that uses one random frame per action unit as its labels. Furthermore, to fill the action counting gap in real physical fitness scenarios and to scale up the current repetition count dataset, we construct a high-quality dataset named FitnessRep, which consists of 2,110 fitness videos collected in realistic scenarios. Experiments demonstrate that the proposed JTSPS-Net outperforms the state-of-the-art approach on our dataset and two other public datasets, especially on fine-grained action samples. In addition, it has a good ability to generalize to repetitive actions belonging to unseen categories.

Recently, deep-learning-based super-resolution methods have achieved excellent performances, but mainly focus on training a single generalized deep network by feeding numerous samples. Yet intuitively, each image has its specific representation, and is expected to acquire an adaptive model. For this issue, we propose a novel convolution modulation (CoMo) mechanism to build image-specific deep networks, by exploiting the principal information of the feature to generate a modulation weight, and thereby adaptively modulating the kernel weights of convolution without any additional parameters, which outperforms the vanilla convolution and several existing attention mechanisms when embedding into the state-of-the-art architectures. To optimize the modulated convolutions in mini-batch training, we introduce an image-specific optimization (IsO) algorithm, which tackles the infeasibility of the conventional optimization algorithms on this issue. Furthermore, we investigate the effect of CoMo on state-of-the-art architectures and design a new CoMoNet architecture by employing the U-style residual learning and hourglass dense block learning, which is an appropriate architecture to utmost improve the effectiveness of CoMo theoretically. Extensive experiments on benchmarks show that the proposed methods achieve superior performances and higher flexibility against the state-of-the-art SISR and blind SR methods. The code is available at github.com/YuanfeiHuang/CoMoNet.

CNNs are widely used in remote sensing image classification because of its outstanding feature extraction ability. However, the classification performance is limited by the complexity of remote sensing scenes and the large inter-class similarity. Furthermore, the existing methods usually distinguish multiple classes of complex targets at the same time, which brings great difficulties to the classification model. To alleviate the above problems, we propose a coarse-to-fine cell division (CFCD) approach to improve HRSIs classification. The algorithm divides the limited labeled samples into two subclasses through continuous decomposition, which reduces the similarity between the ground object classes from the data level. We employ the ℓ12-norm to depict the specific distribution of the target for only two subclasses rather than multiple classes of ground objects, so that the exclusive features of targets can be selected more accurately. Moreover, we propose an optimization process of multi-level training, which not only significantly reduces the difficulty of distinguishing multi-class targets, but also improves the utilization of training samples. Experimental results show that the CFCD algorithm outperforms the state-of-the-art methods with limited training samples on three publicly available HRSIs datasets.

Self-supervised learning (SSL) has been successfully applied to remote sensing image classification by designing pretext tasks to extract valuable feature representations of targets. However, existing SSL methodologies overlook the edge information integral to ground objects, culminating in frequent misclassifications at target boundaries. Additionally, the scarcity of training samples often restricts the full utilization of the knowledge encapsulated in the pre-training model. To address these issues, we propose a novel self-supervised edge perception learning framework (SEPLF) to improve the classification performance of high-resolution remote sensing images (HRSI). The framework comprises self-supervised edge perception learning (SEPL) and training sample augmentation (TSA) algorithms. On the one hand, the SEPL approach leverages morphological data enhancement strategies to render the extracted invariant features more robust. It also effectively mines the potential information concealed at target edges, augmenting ground objects’s edge separability. On the other hand, the TSA algorithm not only obtains a large number of training samples but also enhances the intra-class diversity of the samples by considering different spectral features of the same category of ground objects. Experimental results validate that our proposed method outperforms state-of-the-art algorithms, particularly with limited labeled samples.

Recent CNN-driven face super-resolution (FSR) technologies have achieved excellent breakthroughs by incorporating facial prior knowledge. However, most of them suffer from some obvious limitations. They always estimate facial priors from input low-resolution (LR) faces or coarsely enhanced LR faces, obtaining unfaithful priors that cannot be adequately exploited. This may bring noticeable artifacts to the target results, especially for large scaling factors, deteriorating the fidelity and naturalness and generating suboptimal reconstructed results. In this paper, we propose a two-stage prior-guided FSR approach to learn facial prior knowledge from the optimal SR results of stage one and explore the complementarity between priors to further guide more accurate reconstruction in stage two. Specifically, we develop an efficient local and global interactive hybrid network incorporating facial semantic and geometric priors for more discriminative results. To reach this, we devise a multiscale interconnected symmetric encoder-decoder architecture composed of Prior Interaction-Integration Modules (PIIMs), the Coarse-to-fine Feature Refinement Module (CFRM), and Feature Aggregation Modulation Modules (FAMMs). The encoder concentrates on hierarchically extracting multiscale features. The CFRM is devised to explore the potential correlations between the encoder and the decoder and further guide the refinement and reinforcement of the encoded features. The decoder aims to take full advantage of informative multiscale encoded features to reconstruct high-quality SR representations. Comprehensive evaluation and visualization results on four benchmark datasets demonstrate the superiority of the proposed PLGNet over current state-of-the-art methods. The source code of PLGNet will be available at https://github.com/lil808/PLGNet.git.

Cross-resolution person re-identification (ReID) is a challenging task that addresses the issue of matching individuals across different resolution conditions. Traditional person ReID methods often assume that images have sufficiently high resolution and overlook the practical scenarios involving low-resolution or blurry images. Existing cross-resolution ReID approaches either utilize image super-resolution techniques to improve the quality of low-resolution images or extract and learn resolution invariant features for person representation. Although multi-task learning has been applied in ReID to integrate auxiliary tasks including attribute recognition, image super-resolution, and so on, how to incorporate the vital resolution learning task into cross-resolution ReID has rarely explored before. Therefore, we propose a novel multi-task resolution learning based ReID network named MRLReID. Our approach treats ross-resolution person ReID as the primary task and the resolution estimation as an auxiliary task. Our network simultaneously learns the resolution information and person identity information of images, aiming to improve cross-resolution person ReID performance. Considering that existing similuated cross-resolution datasets are too simple to mimic unconstrained scenario, we further employ image degradation technique to simulate more realistic cross-resolution ReID datasets. We evaluate our method on two real-world cross-resolution datasets and two newly simulated cross-resolution datasets, and both intra-dataset and cross-dataset evaluations demonstrate the effectiveness and superiority of our method in cross-resolution person ReID. The codes and datasets are available at https://github.com/amateurbo/MRLReID.

Multi-focus image fusion (MFIF) creates an image from different source images with various sensors or optical settings as the devices can’t focus all objects at different distances. Most of the MFIF methods have several limitations in encoder enough features from the images and the result are not robust. To overcome the primary issue, we present a robust fusion algorithm based on the Frequency mask and the Hyperdimensional computing. We propose the Frequency Mask Filter (FMF) to get the narrow-band signals by encoding the frequency domain vector through the mask filter in the frequency domain. The Hyperdimensional encoder uses monogenic mapping, in which the multi-modulation features (MMF) such as the frequency, phase and amplitude are dynamically selected to obtain robust focus maps. Generated by multiscale monogenic representations of each image, the narrow-band image are mapped to hypervector encoding. Hyperdimensional encoder shows the energetic and structural information and leads to robust fusion results. Our proposed method is far superior to the existing MFIF method in terms of both objective evaluation metrics and visual effects on three publicly available datasets.Additionally, our proposed method requires only 0.88 seconds and has a parameter count of 0.13 million for multi-focus image fusion.

Palm-vein identification is a highly secure pattern biometrics that has become an active research area in recent years. Despite the recent progress in deep neural networks (DNNs) for vein identification, existing solutions for feature representation continue to lack robustness due to the limited training samples. To address this limitation, data augmentation approaches, including Generative Adversarial Networks (GANs), have been investigated, but these schemes suffer from the following issues. First, it is practically unfeasible to use all the generated samples for classifier training due to the limited storage space and computation resources. Further, some of these generated samples may be non-representative or ineffective, seriously compromising models’ generalization capabilities. Second, the augmented dataset is fed to the target classifier repeatedly, resulting in overfitting after substantial training epochs. To tackle the above problems, we propose AdveinAU, an Adversarial vein AUtomatic AUgmentation approach that generates challenging samples to train a more robust vein classifier for palm-vein identification by alternatively optimizing the vein classifier and a set of latent variables. First, we consider a conditional deep convolution generative adversarial net (cDCGAN) to learn the distribution of real data and the generated data, and then a latent variable from the latent variable space is mapped to the sample space. Second, we combine the trained generator with the vein classifier to constitute AdveinAU, where the input sets of the generator and the classifier are alternatively updated by adversarial training. Specifically, a latent variable set is learned to increase the training loss of a target network through generating adversarial samples, while the classifier learns more robust features from harder examples to improve the generalization. To avoid collapsing inherent meanings of images, an exponential moving average (EMA) teacher andcosinesimilarity are employed for regularization to reduce the search space. Unlike previous works where GANs synthesize new realistic images, our model aims to search a latent variable set, based on which the generator can produce challenging samples along with the training process to improve the classifier’s performance. Finally, we conduct extensive experiments on three public palm-vein datasets to evaluate the performance of AdveinAU, and the experimental results demonstrate that the proposed AdveinAU is capable of generating harder samples to improve the performance of the vein classifier.

The parallel branches with independent optimized classification and localization capabilities are widely used in single-stage object detection. Defects such as feature conflicts, low level of information interaction, and empirical sample allocation scheme lead to weak spatial consistency of the outputs from different branches. In this work, we propose a Progressive Decoupled Task Alignment (PDTA) that enhances the information interaction between tasks while reducing the degree of feature coupling, and adopts a strategy based on sample screening and learning to achieve task alignment. First, we design the Discrepant Feature Decoupling Module (DFDM) embedded with the novel Oriented Decoupling Convolution (ODC) for the coupled features of the shared input, and the features extracted by ODC are utilized for disentanglement through the feed-in scheme with differences. Second, the Probabilistic Mapping Interaction Head (PMI-Head) utilizes the probabilistic mapping method to enhance task-specific semantics by information interaction. Finally, the network’s common attention to the content and position of the target is enhanced through the metric in the proposed Relevance-Guided Adaptive Task Alignment (RATA), in which an exponentially decaying manner is used to preserve the training samples that are more efficient for both tasks. During training, task-aligned learning is performed by Relevance-Guided Loss. Experiments on MS COCO and DIOR datasets demonstrate the effectiveness of our method, PDTA achieves better performance for object detection.

The Intersection over Union (IoU) has been widely employed in various stages of object detection owing to its ability to quantify the similarity between boxes objectively. However, in densely packed scenes full of crowded and small-sized objects, adjacent positive boxes often exhibit high levels of overlap. This overlap interference compromises the consistency between quality evaluation and confidence, leading to ambiguous box prediction within the previous IoU-based models. To address this issue, we design a novel learning paradigm tailored for Dense scenes based on IoU, called DeIoU. This approach effectively suppresses unnecessary overlap between predicted boxes and thereby enhances representation learning for non-salient objects. Specifically, it consists of a dense box regression loss${\mathcal {L}}_{DeIoU}$and a one-to-many (O2M) label matching strategy guided by DeIoU. These components focus on calibrating the position and shape prediction quality during the model training, learning distinguishable object features by penalizing overlap interference between neighboring boxes. Extensive experiments on four object detection datasets including SKU-110K, CrowdHuman, MS COCO 2017, and DIOR, demonstrate that our DeIoU-based learning strategy outperforms other state-of-the-art methods. Notably, the proposed method delivers a substantial improvement (average$1.3~{AP}$and$1.8~MR^{-2}$) across popular detectors on SKU-110K and CrowdHuman while exhibiting distinct competitiveness on small objects within natural scenes.

Person search aims to locate target pedestrians from scene images, involving detection and re-identification. The former seeks to separate the background and focus on the commonality between pedestrians, while the latter aims to identify the target and focus on the difference between pedestrians. To address the paradox of detection and re-identification in search tasks, we propose an efficient Tri-Hybrid person search model utilizing the feature hierarchy design. Our model introduces three feature hybrid models for various feature levels. Before the RoI-Align, we present “Spatial-Channel Hybrid” (SCH) and “Token-Channel Hybrid” (TCH). SCH perceives the boundary frame of pedestrians at multiple scales, thereby enhancing the information disparity between pedestrians and the background and refining the accuracy of the detection frame. TCH uses multi-layer perceptrons (MLP) and blends token and channel features, emphasizing detecting fine-grained semantic information for pedestrians. The interaction of multi-scale perception and fine-grained semantic information enhances the details of detected pedestrians, making them more suitable for similarity measurement in pedestrian matching. After the RoI-Align, we design the “CNN-Transformer Hybrid” to amalgamate global and local features to extract more comprehensive detailed features. Extensive experimental results on CUHK-SYSU and PRW demonstrate the effectiveness of the proposed method over the state-of-the-art performance. Specifically, our method achieves comparable performance on two benchmark datasets, CUHK-SYSU and PRW, with mAP scores of 94.62% and 57.84%, respectively.

Two-stage detectors, which consist of the multi-scale feature representations and the prediction of region proposal boxes, have been recognized as an effective paradigm for tiny object detection in Unmanned Aerial Vehicle (UAV) images. Although most previous methods primarily concentrated on developing efficient feature fusion strategies within the feature pyramid network (FPN), few studies elaborated on improving the performance of region proposal network (RPN). Conventional RPNs exhibit two key weaknesses in the majority of existing two-stage object detection approaches. Firstly, the quality of proposal boxes generated by the RPN is heavily reliant on rich feature representations extracted from the FPN backbone. Secondly, the fixed number of generated proposal boxes limits adaptability to the distribution of tiny person objects. To mitigate the aforementioned problems, in this paper we propose a novel adaptive region proposal network (ARPN) to improve the quality of the proposal boxes and generate particularly compact yet accurate proposal boxes. On one hand, a progressive attention mechanism is devised to make the ARPN focus more on prospective object regions, where a series of multi-scale front attention modules (FAM) are applied to coarsely filter out most of irrelevant background areas and a group of top-to-bottom back attention modules (BAM) aid the ARPN to finely pinpoint tiny objects of interest in a coarse-to-fine manner. On the other hand, a mini-density map, which is inspired by the philosophy of crowd counting, is elaborately designed to adaptively determine the number of region proposal boxes. This approach significantly reduces redundancy while maintaining high-quality proposal boxes. Extensive experiments verify the superiority of proposed ARPN and show obvious improvement over other competitors in terms of two performance indicators of average precision (AP) and average recall (AR). The code will be available at https://github.com/kbzhang0505/ARPN.

In the field of object detection, detecting small objects is an important and challenging task. However, most existing methods tend to focus on designing complex network structures, lack attention to global representation, and ignore redundant noise and dense distribution of small objects in complex networks. To address the above problems, this paper proposes a small object detection method based on global multi-level perception and dynamic region aggregation. The method achieves accurate detection by dynamically aggregating effective features within a region while fully perceiving the features. This method mainly consists of two modules: global multi-level perception module and dynamic region aggregation module. In the global multi-level perception module, self-attention is used to perceive the global region, and its linear transformation is mapped through a convolutional network to increase the local details of global perception, thereby obtaining more refined global information. The dynamic region aggregation module, devised with a sparse strategy in mind, selectively interacts with relevant features. This design allows aggregation of key features of individual instances, effectively mitigating noise interference. Consequently, this approach addresses the challenges associated with densely distributed targets and enhances the model’s ability to discriminate on a fine-grained level. This proposed method was evaluated on two popular datasets. Experimental results show that this method outperforms state-of-the-art methods in small object detection tasks, demonstrating good performance and potential applications.

Reconstructing a high-resolution hyperspectral image (HSI) from a low-resolution HSI is significant for many applications, such as remote sensing and aerospace. Most deep learning-based HSI super-resolution methods pay more attention to developing novel network structures but rarely study the HSI super-resolution problem from the perspective of image dynamic evolution. In this article, we propose that the HSI pixel motion during the super-resolution reconstruction process can be analogized to the particle movement in the smoothed particle hydrodynamics (SPH) field. To this end, we design an SPH network (SPH-Net) for HSI super-resolution in light of the SPH theory. Specifically, we construct a smooth function based on SPH and design a smooth convolution in multiscales to exploit spectral correlation and preserve the spectral information in the super-resolved image. In addition, we apply the SPH approximation method to discretize the Navier-Stokes motion equation into SPH equation form, which can guide the HSI pixel motion in the desired direction during super-resolution reconstruction, thereby producing clear edges in the spatial domain. Experiments on three public hyperspectral datasets demonstrate that the proposed SPH-Net outperforms the state-of-the-art methods in terms of objective metrics and visual quality.

Hyperspectral anomaly detection (HAD) aims to label each hyperspectral image (HSI) pixel as background or anomaly, in a totally unsupervised manner. Thus, a fine background representation is vital to obtain good HAD performance. This article introduces background endmember representation and proposes a novel HAD method termed learnable background endmember with subspace representation (LEBSR). First, the HSI is unmixed to simultaneously obtain the background endmembers and their abundances. The three constraints of$p$-norm, sum-to-one, and nonnegativity work together to promote a more meaningful and accurate background endmember representation. In addition, a mapping matrix with orthogonality is jointly optimized to transform the priori backgrounds into the background endmember subspace, and then the mapped priori backgrounds are approximated to the low-rank representation (LRR) with the background endmembers. With the methodology, the backgrounds can be well reconstructed under the guidance of the priori background information to accurately detect anomalous pixels with the reconstructed residuals. The experimental results on several HSI datasets verify the superior performance of LEBSR than the state-of-the-art methods.https://github.com/HalongL/HAD-LEBSR

Infrared Small Target Detection (IRSTD) is a challenging task of identifying small targets with low signal-to-noise ratios in complex backgrounds. Traditional methods in the complex background of IRSTD lead to a large number of false alarms and missdetections. Although CNN-based methods have made progress in IRSTD, how to extract more effective information and fully utilize inter-layer information remains an unresolved issue. Therefore, this paper proposed a Dual-Encoder Multi-stage Feature Fusion Network (DMFNet). Specifically, we designed a dual-encoder with different inputs to capture more effective small target feature information. We then designed a Receptive Field Expansion Attention Module (REAM) to incorporate non-local contextual information. In the decoding phase, the Triple Cross-layer Fusion Module (TCFM) was developed to exchange the low-level spatial details and the high-level semantic information for preserving more small target information in deeper layers. Finally, by concatenating multi-scale features from various layers of the decoder, more discriminative feature maps were generated to clearly describe the infrared small targets. Experimental results on the NUDT-SIRST, NUAA-SIRST, and IRSTD-1k datasets demonstrated that DMFNet outperforms some other state-of-the-art methods, achieving superior detection performance. Codes: https://github.com/BJZHOU2000/DMFNet.

Transformers has gained widespread adoption in remote sensing image (RSI) segmentation. However, RSI has densely overlapping terrain and significant shadow, making it challenging to segment the blended boundaries of terrains that are the hard classes. Currently, most transformer-based methods construct the self-attention with a sliding window, which influences the feature receptive fields to conquer the intersecting and overlapping objects. Additionally, they often rarely focus specifically on the representation of these hard segmentation objects. To overcome these challenges, we propose a novel Enhancing Semantic Mask Segmentation Network (ESMS-Net) framework including a local-global joint encoder, an auxiliary enhanced encoder, and a multiscale dense decoder. In the local-global joint encoder, we construct a Pyramid Pooling AtrousFormer (PPAFormer) that performs the self-attention with a pyramid-structured atrous sliding window, which enhances the range of receptive fields and the global representation performance. Meanwhile, we construct the dual-feature fusion module (DFFM) and multilevel feature weighted fusion (MFWF) in the multiscale dense decoder to reduce information loss and facilitate the interaction of deep semantic information. For the auxiliary enhanced encoder, we develop a semantic mask based on the predicted results to maintain the hard segmentation classes, and then use the same structure as the first two stages of the local-global joint encoder to learn the hard regions again. Extensive experiments demonstrate the proposed ESMS-Net can achieve significant improvements for segmentation performance compared with the state-of-the-art methods on the ISPRS-Vaihingen and Potsdam datasets. The code will be available athttps://github.com/Wzysaber/ESMS-Net.

Multi-temporal Hyperspectral images (HSIs) have wide applications in change detection (CD) of different land-covers for their rich spectral features and image details. Traditional supervised learning-based HSI CD algorithms often rely on a substantial number of labeled samples. However, it requires a significant cost in sample annotation. In this paper, we propose a diff-feature contrast enhancement network (DCENet) for semi-supervised HSI CD, which leverages a limited number of labeled samples to guide the training process and a large number of unlabeled samples to improve the confidence of change detection. To achieve this, a differential fusion attention (DFA) sub-network is constructed to extract temporal features from the initial input HSI patches. The dual-branch siamese enhancement module (SEM) is utilized to enhance the generalization of differential features in the feature maps. Herein, multi-scale Kullback-Leibler divergence and feature-enhanced probabilistic contrast loss are designed to constrain the SEM. The proposed method excels at detecting subtle changes in bi-temporal HSIs simultaneously improving the generalization performance of networks. The visual and quantitative experimental results on four HSI datasets show that the proposed DCENet outperforms the compared state-of-the-art methods for HSI CD. Codes: https://github.com/Zhoutya/ChangeDetection-DCENet.

Recently, remote sensing super-resolution (SR) tasks have been widely studied and achieved remarkable performance. However, due to the complex texture and serious image degeneration, the conventional methods (e.g. CNN-based, GAN-based) cannot reconstruct high-resolution (HR) remote sensing images with a large SR factor (≥ ×8). In this paper, we model the large-factor super-resolution (LFSR) task as a referenced diffusion process and explore how to embed pixel-wise constraint into the popular diffusion model. Following this motivation, we propose the first diffusion-based LFSR method named texture consistency diffusion model (TCDM) for remote sensing images. Specifically, we build a novel conditional truncated noise generator (CTNG) in TCDM to simultaneously generate the expectation of posterior probabilityp(xt-1|xt) and the truncated noise image. With the predicted truncated noise image, sampling an SR image using CTNG saves nearly 90% processing time compared to the naive diffusion model. Additionally, we design a new denoising process named texture consistency diffusion (TC-diffusion) to explicitly embed pixel-wise constraints into the LFSR diffusion model during the training stage. Universal experiments on five commonly used remote sensing datasets demonstrate that the proposed TCDM surpasses the latest SR methods by a large margin and reports new SOTA results on several evaluation metrics. Additionally, the proposed method demonstrates impressive visual quality on reconstructed remote sensing image texture and details.

Single-frame infrared small target detection (SIRSTD) is in urgent demand for many practical tasks, such as fire rescue and urban management systems, benefiting from the excellent performance of infrared (IR) imaging in harsh climates and low-light environments. SIRSTD strives to segment small targets from the background as accurately as possible. However, in a real-world application, complex background environments with high brightness and strong edges have similar physical characteristics to small IR targets, which makes it extremely difficult to separate small targets. To address this challenge, we propose a novel Gaussian Curvature Inspired Network (GCI-Net). Inspired by the well-known Gaussian curvature, we develop a Gaussian curvature-based branch (GCB) to eliminate the smoothing noise and preserve the target structure texture information. In addition, we design a complementary patch-group attention (PGA) module that relies on the complementary relationship between low-level and high-level features to provide accurate guidance for GCB. The curvature information generated by the GCB is continuously optimized under the constraint of the curvature information of the ground truth. The proposed GCI-Net provides a reliable guarantee for accurate separation of small targets from the background. We conduct extensive experiments on the public IRSTD-1k and SIRST datasets. The experimental results demonstrate that the proposed GCI-Net outperforms the state-of-the-art (SOTA) methods.

Unsupervised person re-identification (ReID) aims at learning discriminative identity features for person retrieval without any annotations. Recent advances accomplish this task by leveraging clustering-based pseudo labels, but these pseudo labels are inevitably noisy, which deteriorates model performance. In this paper, we propose a Neighbour Consistency guided Pseudo Label Refinement (NCPLR) framework, which can be regarded as a transductive form of label propagation under the assumption that the prediction of each example should be similar to its nearest neighbours’. Specifically, the refined label for each training instance can be obtained from the original clustering result and a weighted ensemble of its neighbours’ predictions, with weights determined according to their similarities in the feature space. Furthermore, we also explore building a unified global-local NCPLR mechanism through a global-local label interaction module to achieve mutual label refinement. Such a strategy promotes efficient complementary learning while mitigating some unreliable information, finally improving the quality of the refined pseudo labels for each global-local region. Extensive experimental results demonstrate the effectiveness of the proposed method, showing superior performance to state-of-the-art methods by a large margin. Our source code is released inhttps://github.com/haichuntai/NCPLR-ReID.

Heterogeneous face analysis is an important and challenge problem in face recognition community, because of the large modality discrepancy between heterogeneous face images. Existing methods either focus on transforming heterogeneous faces into the same style via face synthesis process, or intend to directly recognize heterogeneous face via modality invariant descriptors. However, the tasks of cross modality face synthesis and face recognition share a common purpose, which is to disentangle an inherent explainable representation. To this end, we propose a novel universal heterogenous face analysis method via multi-domain feature disentanglement, which does not need any face domain label. The proposed method explores to disentangle factors of variations of cross modality faces in an unsupervised manner. Then we could translate cross modality faces through modifying semantic factors, and the extracted inherent explainable representation still maintains being discriminative for heterogeneous face recognition. Experimental results on multiple cross modality face databases demonstrate the effectiveness of the proposed method. These experimental results also inspire us that the unsupervised disentangled module could help to analyze the interpretability of heterogenous face representation.

With the continuous development of generative models on face generation, how to distinguish the real and fake face has become an important problem for security. Because of the continuous improvement on the detection accuracy by facial physiological signals, video face forgery detection based on facial physiological signal analysis has received more and more attention, which has become an important research branch in the field of face forgery detection. Currently, most of the research on forgery detection based on physiological signal analysis use biometric features such as blinking patterns, head swings, heart rate signals, and lip movements. However, there hasn’t been much exploration on the usage of gaze features in face forgery detection. Through the analysis of gaze directions in face videos, we have observed differences in the distribution of gaze direction pattern between the real and forged videos. Specifically, real videos tend to have more concentrated gaze distribution within a short period of time, while forged videos have more dispersed gaze distributions. In this paper, we present a novel Deepfake gaze analysis method named DFGaze, to explore spatial-temporal gaze inconsistency for video face forgery detection. Our method uses the gaze analysis model (GAM) to analyze the gaze features of face video frames, and then applies a spatial-temporal feature aggregator to realize authenticity classification based on gaze features. In order to better mine the authenticity clues in the videos, we further use the texture analysis model (TAM) and attribute analysis model (AAM) to improve the representation ability of spatial-temporal feature differences between real and forged faces. Extensive experiments show that our method can achieve state-of-the-art performance with the help of gaze analysis. The source code is available at https://github.com/ziminMIAO/DFGaze.

Heterogeneous pedestrian images are captured by visible and infrared cameras with different spectrums, which play an important role in night-time video surveillance. However, visible infrared person re-identification (VI-REID) is still a challenging problem due to the considerable cross-modality discrepancies. To extract modality-invariant features which are discriminative for the person identity, recent studies are inclined to regard modality-specific features as noise and discard them. Actually, the modality-specific characteristics containing background and color information are indispensable for learning modality-shared features. In this paper, we propose a novel Dual-Adversarial Representation Disentanglement (DARD) model to separate modality-specific features from tangled pedestrian representations and effectively learn the robust modality-invariant representations. Specifically, our method employs dual-adversarial learning, incorporating image-level channel exchange and feature-level magnitude change to introduce variations in modality-specific representations. This deliberate perturbation raises the learning difficulty for the model to learn modality-shared features. Simultaneously, to control the changing scope of modality-specific features, bi-constrained noise alleviation is introduced during adversarial learning, keeping the balance of feature generation and adversary. The proposed dual-adversarial learning methodology enhances the robustness against cross-modality visual discrepancy and strengthens the discriminative power of the learned modality-shared representations without introducing additional network parameters. This improvement further elevates the retrieval performance of VI-REID. Extensive experiments with insightful analysis on two cross-modality re-identification datasets verify the effectiveness and superiority of the proposed DARD method.

Advanced manipulation techniques have provided criminals with opportunities to make social panic or gain illicit profits through the generation of deceptive media, such as forgery face images. In response, various deepfake detection methods have been proposed to assess image authenticity. Sequential deepfake detection, which is an extension of deepfake detection, aims to identify forged facial regions with the correct sequence for recovery. Nonetheless, due to the different combinations of spatial and sequential manipulations, forgery face images exhibit substantial discrepancies that severely impact detection performance. Additionally, the recovery of forged images requires knowledge of the manipulation model to implement inverse transformations, which is difficult to ascertain as relevant techniques are often concealed by attackers. To address these issues, we propose Multi-Collaboration and Multi-Supervision Network (MMNet) that handles various spatial scales and sequential permutations in forgery face images and achieve recovery without requiring knowledge of the corresponding manipulation method. Furthermore, existing evaluation metrics only consider detection accuracy at a single inferring step, without accounting for the matching degree with ground-truth under continuous multiple steps. To overcome this limitation, we propose a novel evaluation metric called Complete Sequence Matching (CSM), which considers the detection accuracy at multiple inferring steps, reflecting the ability to detect integrally forged sequences. Extensive experiments on several typical datasets demonstrate that MMNet achieves state-of-the-art detection performance and independent recovery performance. Code will be available at https://github.com/xarryon/MMNet.

Overcoming image acquisition perspectives and face pose variations is a key problem in unconstrained face recognition tasks. One of the practical approaches is by reconstructing the face with extreme pose into a version that is more easily recognized by the discriminator, such as a frontal face. Often, existing methods attempt to balance the accuracy of downstream tasks with human visual perception, but ignore the differences in propensity between the two. Besides, large-scale datasets of profile-frontal paired face images are absent, which further hinders the training of models. In this work, we investigate a variety of face reconstruction approaches and propose a very simple, but very effective method to match face images across different scenes, named facial representation learning (FRL). The core idea of FRL is to introduce a representation generator in front of a pre-trained face recognition model, which can extract face representations from arbitrary faces that are more suitable for recognition model discrimination. In particular, the representation generator reconstructs the facial representation by minimising identity differences from the frontal face and adds pixel-level and adversarial constraints to cater for discriminator preferences. Extensive benchmark experiments show that the proposed method not only achieves better performance than state-of-the-art methods, but also can further squeeze the inference potential of existing face recognition models.

The advancement of face recognition technology has delivered substantial societal advantages. However, it has also raised global privacy concerns due to the ubiquitous collection and potential misuse of individuals’ facial data. This presents a notable paradox: while there is a societal demand for a robust face recognition ecosystem to ensure public security and convenience, an increasing number of individuals are hesitant to release their facial data. Numerous studies have endeavored to find such a utility-privacy trade-off, yet many struggle with the dilemma of prioritizing one at the expense of the other. In response to this challenge, this paper proposes PRO-Face C, a novel paradigm for privacy-preserving recognition of obfuscated faces via a dedicated feature compensation mechanism, aimed at optimizing the equilibrium between privacy preservation and utility maximization. The proposed approach is characterized by a specialized client-server architecture: the client transmits only obfuscated images to the server, which then performs identity recognition using a pre-trained model in conjunction with a suite of privacy-free complementary features. This framework facilitates accurate face identification while safeguarding the original facial appearance from explicit disclosure. Furthermore, the obfuscated image retains its visualization capability, crucial for image preview functionalities. To ensure the desired properties, we have developed an identity-guided feature compensation mechanism, complemented by several privacy-enhancing techniques. Extensive experiments conducted across multiple face datasets underscore the effectiveness of the proposed approach in diverse scenarios.

The key to multi-object tracking is its stability and the retention of identity information. A common problem with most detection-based approaches is trusting and using all the detector outputs for the association. However, some settings of detectors can affect stable long-range tracking. Based on the principle of reducing the association noise in the detection processing step, we propose a new framework, the Box application Pattern Mining Tracker (BPMTrack), to address this issue. Specifically, we worked on three main aspects: output threshold, association strategy, and motion model. Due to the problem of inconsistency between classification scores and localization accuracy, we propose the Box Quality Estimation Network (BQENet) to predict the localization quality scores of all detections in the current frame, reserving high-quality boxes for the tracker. In addition, based on observations of intensive scenarios, we propose a simple and effective data association method, the Non-Maximum Suppression Integration (NMSI) matching strategy. It recovers the Non-Maximum Suppression (NMS) detection, inputs them into BQENet, and then performs hierarchical matching with reasonable control of box priority to alleviate the problem of absent objects caused by occlusion. Finally, we propose an improved Measurement Correct and Noise Scale (MCNS) Kalman algorithm to improve the prediction accuracy of object positions and, thus, the association quality. We performed an extensive ablation evaluation of the proposed framework to prove its effectiveness. Moreover, the three tracking benchmarks show our method's accuracy and long-distance performance.

Due to the sparse single-frame annotations, current Single-Frame Temporal Action Localization (SF-TAL) methods generally employ threshold-based pseudo-label generation strategies. However, these approaches suffer from inefficient data utilization, as only parts of unlabeled frames with confidence scores surpassing a predefined threshold are selected for training. Moreover, the variability of single-frame annotations and unreliable model predictions introduce pseudo-label noise. To address these challenges, we propose two strategies by using the relationship of the video segments with their neighbors': 1) temporal neighbor-guided soft pseudo-label generation (TNPG); and 2) semantic neighbor-guided pseudo-label refinement (SNPR). TNPG utilizes a local-global self-attention mechanism in a transformer encoder to capture temporal neighbor information while focusing on the whole video. Then the generated self-attention map is multiplied by the network predictions to propagate information between labeled and unlabeled frames, and produce soft pseudo-label for all segments. Despite this, label noise persists due to unreliable model predictions. To mitigate this, SNPR refines pseudo-labels based on the assumption that predictions should resemble their semantic nearest neighbors'. Specifically, we search for semantic nearest neighbors of each video segment by cosine similarity in the feature space. Then the refined soft pseudo-labels can be obtained by a weight combination of the original pseudo-label and the semantic nearest neighbors'. Finally, the model can be trained with the refined pseudo-labels, and the performance has been greatly improved. Comprehensive experimental results on different benchmarks show that we achieve state-of-the-art performances on THUMOS14, ActivityNet1.2, and ActivityNet1.3 datasets.

Nowadays, data in the real world often comes from multiple sources, but most existing multi-view ${K}$ -Means perform poorly on linearly non-separable data and require initializing the cluster centers and calculating the mean, which causes the results to be unstable and sensitive to outliers. This paper proposes an efficient multi-view ${K}$ -Means to solve the above-mentioned issues. Specifically, our model avoids the initialization and computation of clusters centroid of data. Additionally, our model use the Butterworth filters function to transform the adjacency matrix into a distance matrix, which makes the model is capable of handling linearly inseparable data and insensitive to outliers. To exploit the consistency and complementarity across multiple views, our model constructs a third tensor composed of discrete index matrices of different views and minimizes the tensor's rank by tensor Schatten ${p}$ -norm. Experiments on two artificial datasets verify the superiority of our model on linearly inseparable data, and experiments on several benchmark datasets illustrate the performance.

The true label plays an important role in semi-supervised medical image segmentation (SSMIS) because it can provide the most accurate supervision information when the label is limited. The popular SSMIS method trains labeled and unlabeled data separately, and the unlabeled data cannot be directly supervised by the true label. This limits the contribution of labels to model training. Is there an interactive mechanism that can break the separation between two types of data training to maximize the utilization of true labels? Inspired by this, we propose a novel consistency learning framework based on the non-parametric distance metric of boundary-aware prototypes to alleviate this problem. This method combines CNN-based linear classification and nearest neighbor-based non-parametric classification into one framework, encouraging the two segmentation paradigms to have similar predictions for the same input. More importantly, the prototype can be clustered from both labeled and unlabeled data features so that it can be seen as a bridge for interactive training between labeled and unlabeled data. When the prototype-based prediction is supervised by the true label, the supervisory signal can simultaneously affect the feature extraction process of both data. In addition, boundary-aware prototypes can explicitly model the differences in boundaries and centers of adjacent categories, so pixel-prototype contrastive learning is introduced to further improve the discriminability of features and make them more suitable for non-parametric distance measurement. Experiments show that although our method uses a modified lightweight UNet as the backbone, it outperforms the comparison method using a 3D VNet with more parameters.

Visible infrared person re-identification (VI-ReID) exposes considerable challenges because of the modality gaps between the person images captured by daytime visible cameras and nighttime infrared cameras. Several fully-supervised VI-ReID methods have improved the performance with extensive labeled heterogeneous images. However, the identity of the person is difficult to obtain in real-world situations, especially at night. Limited known identities and large modality discrepancies impede the effectiveness of the model to a great extent. In this paper, we propose a novel Semi-Supervised Learning framework with Heterogeneous Distribution Consistency (HDC-SSL) for VI-ReID. Specifically, through investigating the confidence distribution of heterogeneous images, we introduce a Gaussian Mixture Model-based Pseudo Labeling (GMM-PL) method, which adaptively adjusts different thresholds for each modality to label the identity. Moreover, to facilitate the representation learning of unutilized data whose prediction is lower than the threshold, Modality Consistency Regularization (MCR) is proposed to ensure the prediction consistency of the cross-modality pedestrian images and handle the modality variance. Extensive experiments with different label settings on two VI-ReID datasets demonstrate the effectiveness of our method. Particularly, HDC-SSL achieves competitive performance with state-of-the-art fully-supervised VI-ReID methods on RegDB dataset with only 1 visible label and 1 infrared label per class.

The emergence of face forgery has raised global concerns on social security, thereby facilitating the research on automatic forgery detection. Although current forgery detectors have demonstrated promising performance in determining authenticity, their susceptibility to adversarial perturbations remains insufficiently addressed. Given the nuanced discrepancies between real and fake instances are essential in forgery detection, previous defensive paradigms based on input processing and adversarial training tend to disrupt these discrepancies. For the detectors, the learning difficulty is thus increased, and the natural accuracy is dramatically decreased. To achieve adversarial defense without changing the instances as well as the detectors, a novel defensive paradigm called Inspector is designed specifically for face forgery detectors. Specifically, Inspector defends against adversarial attacks in a coarse-to-fine manner. In the coarse defense stage, adversarial instances with evident perturbations are directly identified and filtered out. Subsequently, in the fine defense stage, the threats from adversarial instances with imperceptible perturbations are further detected and eliminated. Experimental results across different types of face forgery datasets and detectors demonstrate that our method achieves state-of-the-art performances against various types of adversarial perturbations while better preserving natural accuracy. Code is available on https://github.com/xarryon/Inspector.

Most few-shot learning methods employ either adaptive approaches or parameter amortization techniques. However, their reliance on pre-trained models presents a significant vulnerability. When an attacker's trigger activates a hidden backdoor, it may result in the misclassification of images, profoundly affecting the model's performance. In our research, we explore adaptive defenses against backdoor attacks for few-shot learning. We introduce a specialized stochastic process tailored to task characteristics that safeguards the classification model against attack-induced incorrect feature extraction. This process functions during forward propagation and is thus termed an "in-process defense." Our method employs an adaptive strategy, effectively generating task-level representations, enabling rapid adaptation to pre-trained models, and proving effective in few-shot classification scenarios for countering backdoor attacks. We apply latent stochastic processes to approximate task distributions and derive task-level representations from the support set. This task-level representation guides feature extraction, leading to backdoor trigger mismatching and forming the foundation of our parameter defense strategy. Benchmark tests on Meta-Dataset reveal that our approach not only withstands backdoor attacks but also shows an improved adaptation in addressing few-shot classification tasks.

The potential vulnerability of deep neural networks and the complexity of pedestrian images, greatly limits the application of person re-identification techniques in the field of smart security. Current attack methods often focus on generating carefully crafted adversarial samples or only disrupting the metric distances between targets and similar pedestrians. However, both aspects are crucial for evaluating the security of methods adapted for person re-identification tasks. For this reason, we propose an image-level adaptive adversarial ranking method that comprehensively considers two aspects to adapt to changes in pedestrians in the real world and effectively evaluate the robustness of models in adversarial environments. To generate more refined adversarial samples, our image representation enhancement module leverages channel-wise information entropy, assigning varying weights to different channels to produce images with richer information content, along with a generative adversarial network to create adversarial samples. Subsequently, for adaptive perturbation of ranking, the adaptive weight confusion ranking loss is presented to calculate the weights of distances between positive or negative samples and query samples. It endeavors to push positive samples away from query samples and bring negative samples closer, thereby interfering with the ranking of system. Notably, this method requires no additional hyperparameter tuning or extra data training, making it an adaptive attack strategy. Experimental results on large-scale datasets such as Market1501, CUHK03, and DukeMTMC demonstrate the effectiveness of our method in attacking ReID systems.

Due to many unmarked data, there has been tremendous interest in developing unsupervised feature selection methods, among which graph-guided feature selection is one of the most representative techniques. However, the existing feature selection methods have the following limitations: (1) All of them only remove redundant features shared by all classes and neglect the class-specific properties; thus, the selected features cannot well characterize the discriminative structure of the data. (2) The existing methods only consider the relationship between the data and the corresponding neighbor points by Euclidean distance while neglecting the differences with other samples. Thus, existing methods cannot encode discriminative information well. (3) They adaptively learn the graph in the original or embedding space. Thus, the learned graph cannot characterize the data’s cluster structure. To solve these limitations, we present a novel unsupervised discriminative feature selection via contrastive graph learning, which integrates feature selection and graph learning into a uniform framework. Specifically, our model adaptively learns the affinity matrix, which helps characterize the data’s intrinsic and cluster structures in the original space and the contrastive learning. We minimize ℓ1,2-norm regularization on the projection matrix to preserve class-specific features and remove redundant features shared by all classes. Thus, the selected features encode discriminative information well and characterize the discriminative structure of the data. Generous experiments indicate that our proposed model has state-of-the-art performance.

Magnetic resonance imaging (MRI)-based deep neural networks (DNN) have been widely developed to perform prostate cancer (PCa) classification. However, in real-world clinical situations, prostate MRIs can be easily impacted by rectal artifacts, which have been found to lead to incorrect PCa classification. Existing DNN-based methods typically do not consider the interference of rectal artifacts on PCa classification, and do not design specific strategy to address this problem. In this study, we proposed a novel Targeted adversarial training with Proprietary Adversarial Samples (TPAS) strategy to defend the PCa classification model against the influence of rectal artifacts. Specifically, based on clinical prior knowledge, we generated proprietary adversarial samples with rectal artifact-pattern adversarial noise, which can severely mislead PCa classification models optimized by the ordinary training strategy. We then jointly exploited the generated proprietary adversarial samples and original samples to train the models. To demonstrate the effectiveness of our strategy, we conducted analytical experiments on multiple PCa classification models. Compared with ordinary training strategy, TPAS can effectively improve the single- and multi-parametric PCa classification at patient, slice and lesion level, and bring substantial gains to recent advanced models. In conclusion, TPAS strategy can be identified as a valuable way to mitigate the influence of rectal artifacts on deep learning models for PCa classification.

Semi-supervised learning methods have been explored to mitigate the scarcity of pixel-level annotation in medical image segmentation tasks. Consistency learning, serving as a mainstream method in semi-supervised training, suffers from low efficiency and poor stability due to inaccurate supervision and insufficient feature representation. Prototypical learning is one potential and plausible way to handle this problem due to the nature of feature aggregation in prototype calculation. However, the previous works have not fully studied how to enhance the supervision quality and feature representation using prototypical learning under the semi-supervised condition. To address this issue, we propose an implicit-explicit alignment (IEPAlign) framework to foster semi-supervised consistency training. In specific, we develop an implicit prototype alignment method based on dynamic multiple prototypes on-the-fly. And then, we design a multiple prediction voting strategy for reliable unlabeled mask generation and prototype calculation to improve the supervision quality. Afterward, to boost the intra-class consistency and inter-class separability of pixel-wise features in semi-supervised segmentation, we construct a region-aware hierarchical prototype alignment, which transmits information from labeled to unlabeled and from certain regions to uncertain regions. We evaluate IEPAlign on three medical image segmentation tasks. The extensive experimental results demonstrate that the proposed method outperforms other popular semi-supervised segmentation methods and achieves comparable performance with fully-supervised training methods.

Health disparities among marginalized populations with lower socioeconomic status significantly impact the fairness and effectiveness of healthcare delivery. The increasing integration of artificial intelligence (AI) into healthcare presents an opportunity to address these inequalities, provided that AI models are free from bias. This paper aims to address the bias challenges by population disparities within healthcare systems, existing in the presentation of and development of algorithms, leading to inequitable medical implementation for conditions such as pulmonary embolism (PE) prognosis. In this study, we explore the diverse bias in healthcare systems, which highlights the demand for a holistic framework to reducing bias by complementary aggregation. By leveraging de-biasing deep survival prediction models, we propose a framework that disentangles identifiable information from images, text reports, and clinical variables to mitigate potential biases within multimodal datasets. Our study offers several advantages over traditional clinical-based survival prediction methods, including richer survival-related characteristics and bias-complementary predicted results. By improving the robustness of survival analysis through this framework, we aim to benefit patients, clinicians, and researchers by enhancing fairness and accuracy in healthcare AI systems.

This article proposes a finite-time based sliding-mode controller (FTSMC) and disturbance observer (FTDO) for connected vehicle (CV) platoon with uncertain dynamics. In particular, a recursive structure consisting of first-level and second-level sliding mode surfaces (SMSs) is developed for the chattering of the conventional SMC. Herein, the first-level SMS is designed based on a proportional-integral-derivative SMS considering the spacing error and interactive behaviors of vehicles, and the second-level SMS is based on an integral terminal SMS. Simultaneously, the disturbances suffered from the ego-vehicle uncertainty and nonlinearity are estimated by the FTDO. The FTSMC and FTDO are proposed to regulate the CV platoon in finite time. Meanwhile, the CV platoon can ensure the finite stability and string stability via rigorous analysis. Finally, the feasibility of the proposed controller is verified by extensive simulations and co-simulations, and small-scaled experiments.

Traditional fuzzy set methods, designed around the finest granularity of inputs-individual points and their membership degrees-often struggle with inefficiencies and label noise. To overcome these challenges, we introduce granular-ball computing into the fuzzy set, creating the new granular-ball fuzzy set framework. This approach uses granular-ball inputs rather than single points, significantly reducing the number of entities and minimizing susceptibility to the noise affecting individual sample points. As a result, our framework enhances both efficiency and robustness compared to traditional methods and is applicable across various domains of fuzzy data processing. Furthermore, we apply this framework to fuzzy support vector machines (FSVMs), developing the Granular-ball Fuzzy Support Vector Machine (GBFSVM). Experimental tests on UCI benchmark datasets show that GBFSVM surpasses traditional models in efficiency and robustness.

Image dehazing is a pivotal preliminary step in the advancement of robust intelligent surveillance system. However, it is an extremely challenging ill-posed problem, as it faces severe information degradation when accurately restoring the clean image from its haze-polluted counterpart. This paper proposes a novel Progressive Negative Enhancing (PNE) contrastive learning mechanism to fully exploit various types of negative information, thereby facilitating the traditional positive-oriented objective function for image dehazing. The proposed method can progressively update the negative samples during model training, to steadily squeeze the restored image towards its desired clean target from various directions. Furthermore, considering the image dehazing task as a many-to-one feature mapping problem, we also make an early effort to enhance the robustness of the dehazing model under variational haze densities. Specifically, a novel density-variational dehazing network is proposed to be optimized under the consistency-regularized framework using the proposed PNE learning mechanism. The consistency regularization ensures consistent output given multi-level degraded hazy images, thereby significantly enhancing the robustness of the model in dealing with various hazy scenarios. Extensive experiments demonstrate that the proposed method exhibits superior performance over existing state-of-the-art methods. It achieves average PSNR boosts of 0.60dB, 0.28dB and 0.82dB on dehazing, deraining and desnowing tasks, respectively. The source code is available athttps://github.com/YanLi-LY/PNE-Net.

Object detection aims to classify interest objects within an image and pinpoint their positions using predicted rectangular bounding boxes. However, classification and localization tasks are heterogeneous, not only spatially misaligned but also differing in properties and feature requirements. Modern detectors commonly share the spatial region and detection head for both tasks, making them challenging to achieve optimal performance altogether, resulting in inconsistent accuracy. Specifically, the predicted bounding box may have higher classification confidence but lower localization quality, or vice versa. To tackle this issue, the spatial decoupling mechanism via general deformable RoI pooling is first proposed. This mechanism separately pursues the favorable regions for classification and localization, and subsequently extracts the corresponding features. Then, the semi-decoupled head is designed. Compared to the decoupled head that utilizes independent classification and localization networks, potentially leading to excessive decoupling and compromised detection performance, the semi-decoupled head enables the networks to mutually enhance each other while concentrating on their respective tasks. In addition, the semi-decoupled head also introduces a redundancy suppression module to filter out redundant task-irrelevant information of features extracted by separate networks and reinforce task-related information. By combining the spatial decoupling mechanism with the semi-decoupled head, the proposed detector achieves an impressive 43.7 AP in Faster R-CNN framework with ResNet-101 as backbone network. Without bells and whistles, extensive experimental results on the popular MS COCO dataset demonstrate that the proposed detector suppresses the baseline by a significant margin and outperforms some state-of-the-art detectors. Code is available athttps://github.com/HB-X/gdpool_semi_dehead.

With the assistance of Convolutional Neural Networks (CNNs), Image Quality Assessment (IQA) models have made great progress in evaluating both simulated distortion and authentic distortion. However, most of the existing IQA models only learn the features of distorted images, and thus do not make full use of the available feature representation of other domains. Furthermore, the common multi-scale fusion strategies are relatively simple, such as downsampling and concatenating, which further limits the prediction performance. To this end, we propose a novel blind image quality index with cross-domain interaction and cross-scale integration, which is designed based on the combination of CNN and Transformer. First, the hierarchical spatial-domain and gradient-domain representations are obtained through a typical CNN architecture. Then, based on the proposed gradient-query cross-attention, these two types of features are fully interacted in the Cross-Domain Interaction (CDI) module. To represent the distortion information more comprehensively, the Cross-Scale Integration (CSI) module is proposed to combine the information between different scales progressively. Finally, the quality score is obtained through a simple regression module. The experimental results on five public IQA databases of both simulated and authentic scenes show that the proposed model outperforms the compared state-of-the-art metrics. In addition, cross-database experiments show that the proposed model has strong generalization performance.

Domain adaptive object detection (DAOD) aims to develop a detector trained on labeled source domains to identify objects in unlabeled target domains. A primary challenge in DAOD is the domain shift problem. Most existing methods learn domain-invariant features within single domain embedding space, often resulting in heavy model biases due to the intrinsic data properties of source domains. To mitigate the model biases, this paper proposes VLDadaptor, a domain adaptive object detector based on vision-language models (VLMs) distillation. Firstly, the proposed method integrates domain-mixed contrastive knowledge distillation between the visual encoder of CLIP and the detector by transferring category-level instance features, which guarantees the detector can extract domain-invariant visual instance features across domains. Then, VLDadaptor employs domain-mixed consistency distillation between the text encoder of CLIP and detector by aligning text prompt embeddings with visual instance features, which helps to maintain the category-level feature consistency among the detector, text encoder and the visual encoder of VLMs. Finally, the proposed method further promotes the adaptation ability by adopting a prompt-based memory bank to generate semantic-complete features for graph matching. These contributions enable VLDadaptor to extract visual features into the visual-language embedding space without any evident model bias towards specific domains. Extensive experimental results demonstrate that the proposed method achieves state-of-the-art performance on Pascal VOC to Clipart adaptation tasks and exhibits high accuracy on driving scenario tasks with significantly less training time.

Research in psychology demonstrates that visual features and semantic content can convey various emotions. Furthermore, studies have proved that image emotion and aesthetics are inextricably linked. During the image aesthetic assessment process (IAA), images elicit emotional responses from individuals, leading to emotional resonance and influencing the evaluation of images. This article proposes an image aesthetics assessment method based on hypernetwork of emotion fusion (HNEF). Our method incorporates the emotions depicted in images into the process of IAA. To accomplish this, we extract both aesthetic and emotional features from the images. Additionally, we employed the self-attention mechanism of the transformer to comprehensively investigate the intimate connection between aesthetics and emotion. Additionally, the hypernetwork is designed to establish perception rules governing the high-level semantic information in images. The experimental results validate the strong correlation between emotion and aesthetics. Furthermore, the proposed method exhibits a significantly competitive advantage when compared to existing methods on the Aesthetic Visual Analysis (AVA) dataset.

Recent studies have shown impressive performance in object detection. However, most current detectors only explore the appearance feature to locate and classify objects but disregard or underestimate the valuable contextual information in the image, which limits the detection performance for those hard objects, such as small objects, occluded objects, blurred objects, etc. In this article, we instead seek to build a novel context modeling framework and conduct more effective context reasoning for object detection. Specifically, we design a Context-guided Reasoning Network (CRNet) to explore the relationships between objects and use easy detected objects to help understand hard ones. In our CRNet, an image is modeled as a graph and local features of objects are viewed as nodes of the graph to learn the relationships between objects. By passing contextual information in the built graph, the features of hard objects can be updated to discriminative features. To this end, we first develop a cascaded center prediction module built upon CenterNet to produce a set of high-quality proposals viewed as nodes of the graph. In addition, to maximize the value of global context information, we present a multi-granularity feature fusion network to encode the whole scene information which is also viewed as nodes of the graph. Then, the spatial and semantic relationships between objects are learned to initialize edges of the graph. Finally, context reasoning is conducted to update the node states iteratively. Extensive experiments are conducted on MS COCO and Pascal VOC to demonstrate the effectiveness of the proposed CRNet. Experimental results show that the proposed CRNet greatly improves the detection performance over existing context-based detectors, and it is comparable with state-of-the-art detectors.

Due to the excellent interpretability of non-negative matrix factorization (NMF), NMF-based multi-view clustering has attracted much attention for multi-media data analysis and processing. However, the existing clustering methods leverage NMF to cluster data matrix, resulting in high computational complexity. Moreover, they are sub-optimal to exploit the complementary information between views because they all measure the between-views error pixel by pixel. To tackle this problem, inspired by orthogonal NMF and anchor graph, we present an efficient anchor graph factorization model with orthogonal, non-negative, and tensor low-rank constraints. We use an anchor graph instead of a data matrix to get an indicator matrix without post-processing, which remarkably reduces the computational complexity. To exploit the between-views complementary information well, we introduce tensor Schatten$p$-norm regularization on the third tensor, composed of soft label matrices of views. The solution can be obtained by iteratively optimizing four decoupled sub-problems, which can be solved more efficiently with good convergence. Through experimental results on the six multi-view datasets, our approach ensures the enhancement of clustering performance while improving efficiency.

Prompt learning has emerged as a thriving parameter-efficient fine-tuning technique for adapting pre-trained vision-language models (VLMs) to various downstream tasks. However, existing prompt learning approaches still exhibit limited capability for adapting foundational VLMs to specific domains that require specialized and expert-level knowledge. Since this kind of specific knowledge is primarily embedded in the pre-defined text labels, we infer that foundational VLMs cannot directly interpret semantic meaningful information from these specific text labels, which causes the above limitation. From this perspective, this paper additionally models text labels with learnable tokens and casts this operation into traditional prompt learning framework. By optimizing label tokens, semantic meaningful text labels are automatically learned for each class. Nevertheless, directly optimizing text label still remains two critical problems, i.e., insufficient optimization and biased optimization. We further address these problems by proposing Modality Interaction Text Label Optimization (MITLOp) and Color-based Consistency Augmentation (CCAug) respectively, thereby effectively improving the quality of the optimized text labels. Extensive experiments indicate that our proposed method achieves significant improvements in VLM adaptation on specific domains.

Face forgery detection plays an important role in personal privacy and social security. With the development of adversarial generative models, high-quality forgery images become more and more indistinguishable from real to humans. Existing methods always regard as forgery detection task as the common binary or multi-label classification, and ignore exploring diverse multi-modality forgery image types, e.g. visible light spectrum and near-infrared scenarios. In this article, we propose a novelHierarchicalForgeryClassifier forMulti-modalityFaceForgeryDetection(HFC-MFFD), which could effectively learn robust patches-based hybrid domain representation to enhance forgery authentication in multiple modality scenarios. The local hybrid domain representation is designed to explore strong discriminative forgery clues both in the image and frequency domain with the intra-attention mechanism. Furthermore, the specific hierarchical face forgery classifier is designed through the authenticity feedback strategy to integrate diverse discriminative clues. Experimental results on representative multi-modality face forgery datasets demonstrate the superior performance of the proposed HFC-MFFD compared with state-of-the-art algorithms.

Heterogeneous face recognition (HFR), which refers to matching face images with different modalities, is essential to public safety. Although HFR has made promising progress in recent years, disguised faces in HFR scenarios still remain a major challenge for the following reasons. First, most existing HFR methods focus on traditional scenarios without disguised accessories, and the performance degrades when dealing directly with disguised faces. Second, there is a need for disguised heterogeneous face datasets, which is essential for developing the related research community. Third, colorful accessories are distinct from heterogeneous face images in terms of their modalities, and their direct combination results in style inconsistency and poor quality. Therefore, we propose a disguised heterogeneous face generation method based on an iterative-adversarial style unification framework. Our approach aims to gradually learn frame textures to detail textures in multiple confrontation iterations, resulting in style unification for disguised accessories and heterogeneous faces. We also construct a disguised heterogeneous face dataset, which contains a disguised NIR-VIS subset and a disguised sketch-photo subset. Moreover, we provide benchmark evaluations conducted on our proposed dataset with face recognition and image quality assessment, demonstrating the superiority of our method over direct addition and two representative disguised face generation techniques.

After several years of development, deep synthesis technology has made significant progress in image and video synthesis. Deep forgery represented by Deepfakes has become a research hotspot, which is used as a tool for disinformation attacks. The current strongly discriminative models can have good performance on specific datasets, even close to 100% accuracy. Unfortunately, since a specific discriminative method only fits a specific data distribution, and different forgery methods or datasets have different data distributions. These methods fail to achieve high performance in cross-dataset detection. In response to this problem and focusing on the actual situation, we adjust the strong generalization detection across the dataset to the generalization detection of unseen fake video. We propose Multi-Crise-Cross Attention and StyleGANv2 Generative Adversarial Network (MCS-GAN). Firstly, we built a Generative Adversarial Network (GAN) framework to learn the distribution of real face data and generate corresponding face images. Secondly, to break the high stitch between the fake region and the background, the model needs to have strong enough feature analysis and pixel restoration capabilities. Therefore, we propose a generator consisting of a Multi-Crise-Cross-Attention (MC) encoder and a StyleGANv2 (SG2) decoder. Finally, to avoid the situation where as long as a face is normal or different faces are abnormal, we set a latent space encoding discriminator and increase the ratio of latent space vector, so as to detect anomaly generated by the forgery operation acting on latent space. We conduct some model generalization experiments on videos on the Internet and some popular deepfake databases. The results show that the accuracy of our method is better compared with the best methods.

Point cloud registration suffers from repeated patterns and low geometric structures in indoor scenes. The recent transformer utilises attention mechanism to capture the global correlations in feature space and improves the registration performance. However, for indoor scenarios, global correlation loses its advantages as it cannot distinguish real useful features and noise. To address this problem, we propose an image-geometry-assisted point cloud registration method by integrating image information into point features and selectively fusing the geometric consistency with respect to reliable salient areas. Firstly, an Intra-Image-Geometry fusion module is proposed to integrate the texture and structure information into the point feature space by the cross-attention mechanism. Initial corresponding superpoints are acquired as salient anchors in the source and target. Then, a selective correlation fusion module is designed to embed the correlations between the salient anchors and points. During training, the saliency location and selective correlation fusion modules exchange information iteratively to identify the most reliable salient anchors and achieve effective feature fusion. The obtained distinctive point cloud features allow for accurate correspondence matching, leading to the success of indoor point cloud registration. Extensive experiments are conducted on 3DMatch and 3DLoMatch datasets to demonstrate the outstanding performance of the proposed approach compared to the state-of-the-art, particularly in those geometrically challenging cases such as repetitive patterns and low-geometry regions.

Visible-infrared person re-identification (VI-ReID) is a challenging task because the different imaging principles of visible and infrared images bring about huge modality discrepancy. Existing methods primarily address this issue by generating intermediate images to align modality features and establish connections between the visible and infrared modalities. However, the quality of these generated images is often unstable, limiting the effectiveness of such approaches. To overcome this limitation, we propose a novel method called modality shared-specific features cooperative separation. It consists of two key modules: the saliency response module and the cooperative separation module, aimed at alleviating the modality gap. The saliency response module incorporates a location attention mechanism and local features to construct contextual connections and extract local saliency information. Then, the cooperative separation module employs a more concise dual-MLPs as generator to effectively separate shared-specific features. Additionally, we introduce a shared feature refinement mechanism in both the generator and discriminator. By coordinating the shared-specific features, our method achieves secondary separation and extracts purer modality-shared features without specific information. Extensive experiments conducted on the SYSU-MM01 and RegDB public datasets demonstrate that our proposed method performs excellently in VI-ReID.

The usual active learning is to sample unlabeled set by designing efficient sample information evaluation algorithms. However, information redundancy between candidate sets is often overlooked. This can cause similar data to be labeled repeatedly, producing ineffective gains for the model. In this paper, we proposed an Unsupervised Redundant Feature Elimination Active Learning module (URFEAL), which utilizes the information feature coincidence of the unlabeled set to eliminate information redundant data, thus guaranteeing the validity of each candidate data. URFEAL consists of feature clusterer and eliminator. The feature clusterer computes class boundaries based on feature densities to discretize each class of the candidate set, and the eliminator judges data similarity by overlapping degree to eliminate redundant data features. Furthermore, we propose an anti-noise sampling strategy Outlier Feature Elimination (OFE) in URFEAL to filter mislabeled sets for relabeling in the data sampling stage. We extensively evaluate our method by image classification and perform experimental validation on CIFAR-10, CIFAR-100 and CALTECH-101. The experimental results show that the improvements we make are especially significant for most existing active learning algorithms in the low data stage, which demonstrates the effectiveness and generality of URFEAL.

Visible-infrared Person Re-identification (VI-ReID) aims to retrieve the images of pedestrian with the same identity from different modalities and cameras given a pedestrian image. To reduce modality discrepancy, existing methods often perform hard partitioning to mine more detail. However, these methods employ only uniform partitioning, without considering pedestrian structure, and lose a lot of pedestrian semantic information. To this end, this paper proposes a structural semantic representation reconstruction (SSRR) method to capture pedestrian semantic information by focusing on pedestrian structure. Specifically, based on the fine-grained features obtained by hard partitioning, we carry out structural reconstruction to obtain the reconstructed features containing semantic information. By adopting the direct link reconstruction structure, the reciprocal learning of fine-grained features and semantic features is ensured. Semantic features are reconstructed based on fine-grained features, and semantic information is beneficial to fine-grained features to better capture pedestrian-related details. In addition, local consistency loss is introduced to ensure the consistency of fine-grained features in the same component location, further enhancing the discriminant of the learned reconstructed representation. Extensive experiments confirm the superiority of our method on two public datasets SYSU-MM01 and RegDB.

Video-based person re-identification (Re-ID) is designed to retrieve target pedestrians in video sequences under non-overlapping cameras. At present, mainstream approaches post-process the feature map extracted by the convolutional neural network backbone to obtain a global representation or a fine-grained local representation for higher accuracy. However, they still suffer from challenges, such as information loss for global-based methods and spatio-temporal feature fragmentation for local-based methods. To alleviate these problems, this article proposes a Spatio-Temporal Feature Enhancement (STFE) network from a spatio-temporal comprehensive perspective, combining the advantages of the above methods to obtain more comprehensive information from video tracklets. STFE consists of two main modules: Feature Space Projection Module (FSPM) and Global Low-frequency Enhancement Module (GLEM). FSPM mathematically converts continuous video information into a discrete feature space and selectively retains more useful information, thus avoiding spatio-temporal information loss. Meanwhile, FSPM applies global features instead of dividing feature maps spatially, thereby avoiding spatio-temporal feature fragmentation. In addition, GLEM which is based on transformer, acts as a broadband low-pass filter to mine richer global comprehensive information. Finally, by combining FSPM with GLEM, STFE can obtain spatio-temporal comprehensive video representation. Extensive experiments were conducted on two widely-used video Re-ID datasets. The experimental results verify our idea and demonstrate the effectiveness of the proposed STFE with 95.5% Rank-1 accuracy on MARS benchmarks, which surpasses previousstate-of-the-artsby a large margin of +4%.

Semi-Supervised Object Detection (SSOD) has shown remarkable results by leveraging image pairs with a teacher-student framework. An excellent strong augmentation method can generate richer images and alleviate the influence of noise in pseudo-labels. However, existing data augmentation methods for SSOD do not consider instance-level information, thus, they cannot make full use of unlabeled data. Besides, the current teacher-student framework in SSOD solely relies on pseudo-labeling techniques, which may disregard some uncertain information. In this article, we introduce a new method called Elaborate Teacher which generates and exploits image pairs in a more refined manner. To enrich strongly augmented images, a novel data augmentation method called Information-Aware Mixup Representation (IAMR) is proposed. IAMR utilizes the teacher model's predictions as prior information and considers instance-level information, which can be seamlessly integrated with existing SSOD data augmentation methods. Furthermore, to fully exploit the information in unlabeled data, we propose the Enhanced Scale Consistency Regularization (ESCR), which considers the consistency from both semantic space and feature space. Elaborate Teacher introduces a fresh data augmentation method, complemented by consistency regularization, which boosts the performance of semi-supervised object detectors. Extensive experiments on thePASCAL VOCandMS-COCOdatasets demonstrate the effectiveness of our method in leveraging unlabeled image information. Our method consistently outperforms the baseline method and improves mAP by 11.6% and 9.0% relative to the supervised baseline method when using 5% and 10% of labeled data onMS-COCO, respectively.

Recently, multi-view clustering methods have been widely used in handling multi-media data and have achieved impressive performances. Among the many multi-view clustering methods, anchor graph-based multi-view clustering has been proven to be highly efficient for large-scale data processing. However, most existing anchor graph-based clustering methods necessitate post-processing to obtain clustering labels and are unable to effectively utilize the information within anchor graphs. To address this issue, we draw inspiration from regression and feature selection to proposeAnchorGraph-BasedFeatureSelection forOne-stepMulti-ViewClustering (AGFS-OMVC). Our method combines embedding learning and sparse constraint to perform feature selection, allowing us to remove noisy anchor points and redundant connections in the anchor graph. This results in a clean anchor graph that can be projected into the label space, enabling us to obtain clustering labels in a single step without post-processing. Lastly, we employ the tensor Schatten$p$-norm as a tensor rank approximation function to capture the complementary information between different views, ensuring similarity between cluster assignment matrices. Experimental results on five real-world datasets demonstrate that our proposed method outperforms state-of-the-art approaches.

Cross-domain face translation aims to transfer face images from one domain to another. It can be widely used in practical applications, such as photos/sketches in law enforcement, photos/drawings in digital entertainment, and near-infrared (NIR)/visible (VIS) images in security access control. Restricted by limited cross-domain face image pairs, the existing methods usually yield structural deformation or identity ambiguity, which leads to poor perceptual appearance. To address this challenge, we propose a multi-view knowledge (structural knowledge and identity knowledge) ensemble framework with frequency consistency (MvKE-FC) for cross-domain face translation. Due to the structural consistency of facial components, the multi-view knowledge learned from large-scale data can be appropriately transferred to limited cross-domain image pairs and significantly improve the generative performance. To better fuse multi-view knowledge, we further design an attention-based knowledge aggregation module that integrates useful information, and we also develop a frequency-consistent (FC) loss that constrains the generated images in the frequency domain. The designed FC loss consists of a multidirection Prewitt (mPrewitt) loss for high-frequency consistency and a Gaussian blur loss for low-frequency consistency. Furthermore, our FC loss can be flexibly applied to other generative models to enhance their overall performance. Extensive experiments on multiple cross-domain face datasets demonstrate the superiority of our method over state-of-the-art methods both qualitatively and quantitatively.

Binary neural network (BNN) is an effective method for reducing model computational and memory cost, which has achieved much progress in the super-resolution (SR) field. However, there is still a noticeable performance gap between a binary SR network and its full-precision counterpart. Considering that the information density in quantization features is far lower than full-precision features, we aim to improve the precision of quantization features to produce rich-enough output activations for SR task. First, we make several observations that a multibit value could be approximated by multiple 1-bit values, and the computation power of binary convolution could be improved by approximating the multibit convolution process. Then, we propose a mixed binary representation set to approximate multibit activations, which is effective in compensating the quantization precision loss. Finally, we present a new precision-driven binary convolution (PDBC) module, which increases the convolution precision and protects image detail information without extra computation. Compared with normal binary convolution, our method could largely reduce the information loss caused by binarization. In experiments, our methods consistently show superior performance over the baseline models and can surpass state-of-the-art methods in terms of peak signal to noise ratio (PSNR) and visual quality.

Weakly supervised temporal action localization (WTAL) aims to classify and localize temporal boundaries of actions for the video, given only video-level category labels in the training datasets. Due to the lack of boundary information during training, existing approaches formulate WTAL as a classification problem, i.e., generating the temporal class activation map (T-CAM) for localization. However, with only classification loss, the model would be suboptimized, i.e., the action-related scenes are enough to distinguish different class labels. Regarding other actions in the action-related scene (i.e., the scene same as positive actions) as co-scene actions, this suboptimized model would misclassify the co-scene actions as positive actions. To address this misclassification, we propose a simple yet efficient method, named bidirectional semantic consistency constraint (Bi-SCC), to discriminate the positive actions from co-scene actions. The proposed Bi-SCC first adopts a temporal context augmentation to generate an augmented video that breaks the correlation between positive actions and their co-scene actions in the inter-video. Then, a semantic consistency constraint (SCC) is used to enforce the predictions of the original video and augmented video to be consistent, hence suppressing the co-scene actions. However, we find that this augmented video would destroy the original temporal context. Simply applying the consistency constraint would affect the completeness of localized positive actions. Hence, we boost the SCC in a bidirectional way to suppress co-scene actions while ensuring the integrity of positive actions, by cross-supervising the original and augmented videos. Finally, our proposed Bi-SCC can be applied to current WTAL approaches and improve their performance. Experimental results show that our approach outperforms the state-of-the-art methods on THUMOS14 and ActivityNet. The code is available at https://github.com/lgzlIlIlI/BiSCC.

Domain adaptation person re-identification (Re-ID) is a challenging task, which aims to transfer the knowledge learned from the labeled source domain to the unlabeled target domain. Recently, some clustering-based domain adaptation Re-ID methods have achieved great success. However, these methods ignore the inferior influence on pseudo-label prediction due to the different camera styles. The reliability of the pseudo-label plays a key role in domain adaptation Re-ID, while the different camera styles bring great challenges for pseudo-label prediction. To this end, a novel method is proposed, which bridges the gap of different cameras and extracts more discriminative features from an image. Specifically, an intra-to-intermechanism is introduced, in which samples from their own cameras are first grouped and then aligned at the class level across different cameras followed by our logical relation inference (LRI). Thanks to these strategies, the logical relationship between simple classes and hard classes is justified, preventing sample loss caused by discarding the hard samples. Furthermore, we also present a multiview information interaction (MvII) module that takes features of different images from the same pedestrian as patch tokens, obtaining the global consistency of a pedestrian that contributes to the discriminative feature extraction. Unlike the existing clustering-based methods, our method employs a two-stage framework that generates reliable pseudo-labels from the views of the intracamera and intercamera, respectively, to differentiate the camera styles, subsequently increasing its robustness. Extensive experiments on several benchmark datasets show that the proposed method outperforms a wide range of state-of-the-art methods. The source code has been released at https://github.com/lhf12278/LRIMV.

The success of modern single image super-resolution (SISR) algorithms is inspired by the development of deep convolutional neural networks (CNNs). However, these CNN-based methods require considerable computation and complexity, making it impossible for these methods to perform real-time calculations in edge devices. Thus, lightweight model design has become a development trend in the super-resolution field, including pruning, quantization, and other methods. The 1-bit quantization is an extreme lightweight method which can reduce the calculation amount of the model in an extreme manner and is friendly to hardware such as edge devices. Most existing binary quantization approaches lead to a large information loss during forward propagation, especially in detailed color information (e.g., edge, texture, and contrast). The loss of color information makes modern binary methods unsuitable for SISR tasks. We think the loss occurs because these methods typically utilize a uniform threshold to quantize the weights and activations. Thus, in this article, we thoroughly analyze the difference between normal classification tasks and SISR tasks, and present a binarization scheme based on local means. The proposed method can maintain more detailed information in feature maps using dynamic thresholds during quantization. Specifically, each value in the full precision activations has a corresponding threshold during the quantization process, and those thresholds are determined by the full precision values of the surroundings. In addition, a gradient approximator is introduced to adaptively optimize the gradient for updating binary weights. We then verify the effectiveness of our method for training binary networks on several SISR benchmarks including VDSR and SRResNet. Experimental results show that the proposed method can outperform the state-of-the-art algorithms to obtain binary networks for image super-resolution with better peak signal-to-noise ratio (PSNR) values and visual quality.

In this article, we address the face image translation task, which aims to translate a face image of a source domain to a target domain. Although significant progress has been made by recent studies, face image translation is still a challenging task because it has more strict requirements for texture details: even a few artifacts will greatly affect the impression of generated face images. Targeting to synthesize high-quality face images with admirable visual appearance, we revisit the coarse-to-fine strategy and propose a novel p arallel m ultistage architecture on the basis of g enerative a dversarial n etworks (PMSGAN). More specifically, PMSGAN progressively learns the translation function by disintegrating the general synthesis process into multiple parallel stages that take images with gradually decreasing spatial resolution as inputs. To prompt the information exchange between various stages, a cross-stage atrous spatial pyramid (CSASP) structure is specially designed to receive and fuse the contextual information from other stages. At the end of the parallel model, we introduce a novel attention-based module that leverages multistage decoded outputs as in situ supervised attention to refine the final activations and yield the target image. Extensive experiments on several face image translation benchmarks show that PMSGAN performs considerably better than state-of-the-art approaches.

The performance of zero-shot learning (ZSL) can be improved progressively by learning better features and generating pseudosamples for unseen classes. Existing ZSL works typically learn feature extractors and generators independently, which may shift the unseen samples away from their real distribution and suffers from the domain bias problem. In this article, to tackle this challenge, we propose a variational autoencoder (VAE)-based framework, that is, joint Attentive Region Embedding with Enhanced Semantics (AREES), which is tailored to advance the zero-shot recognition. Specifically, AREES is end-to-end trainable and consists of three network branches: 1) attentive region embedding is used to learn the semantic-guided visual features by the attention mechanism (AM); 2) a decomposition structure and a semantic pivot regularization are used to extract enhanced semantics; and 3) a multimodal VAE (mVAE) with the cross-reconstruction loss and the distribution alignment loss is used to obtain a shared latent embedding space of visual features and semantics. Finally, features' extraction and features' generation are optimized together in AREES to address the domain shift problem to a large extent. The comprehensive evaluations on six benchmarks, including the ImageNet, demonstrate the superiority of the proposed model over its state-of-the-art counterparts.

Multimodal metric learning aims to transform heterogeneous data into a common subspace where cross-modal similarity computing can be directly performed and has received much attention in recent years. Typically, the existing methods are designed for nonhierarchical labeled data. Such methods fail to exploit the intercategory correlations in the label hierarchy and, therefore, cannot achieve optimal performance on hierarchical labeled data. To address this problem, we propose a novel metric learning method for hierarchical labeled multimodal data, named deep hierarchical multimodal metric learning (DHMML). It learns the multilayer representations for each modality by establishing a layer-specific network corresponding to each layer in the label hierarchy. In particular, a multilayer classification mechanism is introduced to enable the layerwise representations to not only preserve the semantic similarities within each layer, but also retain the intercategory correlations across different layers. In addition, an adversarial learning mechanism is proposed to bridge the cross-modality gap by producing indistinguishable features for different modalities. Through integration of the multilayer classification and adversarial learning mechanisms, DHMML can obtain hierarchical discriminative modality-invariant representations for multimodal data. Experiments on two benchmark datasets are used to demonstrate the superiority of the proposed DHMML method over several state-of-the-art methods.

Granular-ball computing (GBC) is an efficient, robust, and scalable learning method for granular computing. The granular ball (GB) generation method is based on GB computing. This article proposes a method for accelerating GB generation using division to replace k -means. It can significantly improve the efficiency of GB generation while ensuring an accuracy similar to that of the existing methods. In addition, a new adaptive method for GB generation is proposed by considering the elimination of the GB overlap and other factors. This makes the GB generation process parameter-free and completely adaptive in the true sense. In addition, this study first provides mathematical models for the GB covering. The experimental results on some real datasets demonstrate that the two proposed GB generation methods have accuracies similar to those of the existing method in most cases, while adaptiveness or acceleration is realized. All the codes were released in the open-source GBC library at https://www.cquptshuyinxia.com/GBC.html or https://github.com/syxiaa/gbc.

Text attribute person search aims to identify the particular pedestrian by textual attribute information. Compared to person re- identification tasks which requires imagery samples as its query, text attribute person search is more useful under the circumstance where only witness is available. Most existing text attribute person search methods focus on improving the matching correlation and alignments by learning better representations of person-attribute instance pairs, with few consideration of the latent correlations between attributes. In this work, we propose a graph convolutional network (GCN) and pseudo-label-based text attribute person search method. Concretely, the model directly constructs the attribute correlations by label co- occurrence probability, in which the nodes are represented by attribute embedding and edges are by the filtered correlation matrix of attribute labels. In order to obtain better representations, we combine the cross-attention module (CAM) and the GCN. Furthermore, to address the unseen attribute relationships, we update the edge information through the instances through testing set with high predicted probability thus to better adapt the attribute distribution. Extensive experiments illustrate that our model outperforms the existing state-of-the-art methods on publicly available person search benchmarks: Market-1501 and PETA.

The world we see is ever-changing and it always changes with people, things, and the environment. Domain is referred to as the state of the world at a certain moment. A research problem is characterized as transfer adaptation learning (TAL) when it needs knowledge correspondence between different moments/domains. TAL aims to build models that can perform tasks of target domain by learning knowledge from a semantic-related but distribution different source domain. It is an energetic research field of increasing influence and importance, which is presenting a blowout publication trend. This article surveys the advances of TAL methodologies in the past decade, and the technical challenges and essential problems of TAL have been observed and discussed with deep insights and new perspectives. Broader solutions of TAL being created by researchers are identified, i.e., instance reweighting adaptation, feature adaptation, classifier adaptation, deep network adaptation, and adversarial adaptation, which are beyond the early semisupervised and unsupervised split. The survey helps researchers rapidly but comprehensively understand and identify the research foundation, research status, theoretical limitations, future challenges, and understudied issues (universality, interpretability, and credibility) to be broken in the field toward generalizable representation in open-world scenarios.

Very high-resolution (VHR) remote sensing (RS) image classification is the fundamental task for RS image analysis and understanding. Recently, Transformer-based models demonstrated outstanding potential for learning high-order contextual relationships from natural images with general resolution ( pixels) and achieved remarkable results on general image classification tasks. However, the complexity of the naive Transformer grows quadratically with the increase in image size, which prevents Transformer-based models from VHR RS image ( pixels) classification and other computationally expensive downstream tasks. To this end, we propose to decompose the expensive self-attention (SA) into real and imaginary parts via discrete Fourier transform (DFT) and, therefore, propose an efficient complex SA (CSA) mechanism. Benefiting from the conjugated symmetric property of DFT, CSA is capable to model the high-order contextual information with less than half computations of naive SA. To overcome the gradient explosion in Fourier complex field, we replace the Softmax function with the carefully designed Logmax function to normalize the attention map of CSA and stabilize the gradient propagation. By stacking various layers of CSA blocks, we propose the Fourier complex Transformer (FCT) model to learn global contextual information from VHR aerial images following the hierarchical manners. Universal experiments conducted on commonly used RS classification datasets demonstrate the effectiveness and efficiency of FCT, especially on VHR RS images. The source code of FCT will be available at https://github.com/Gao-xiyuan/FCT.

Image super-resolution (SR) is a critical image preprocessing task for many applications. How to recover features as accurately as possible is the focus of SR algorithms. Most existing SR methods tend to guide the image reconstruction process with gradient maps, frequency perception modules, etc. and improve the quality of recovered images from the perspective of enhancing edges, but rarely optimize the neural network structure from the system level. In this article, we conduct an in- depth exploration for the inner nature of the SR network structure. In light of the consistency between thermal particles in the thermal field and pixels in the image domain, we propose a novel heat-transfer-inspired network (HTI-Net) for image SR reconstruction based on the theoretical basis of heat transfer. With the finite difference theory, we use a second-order mixed-difference equation to redesign the residual network (ResNet), which can fully integrate multiple information to achieve better feature reuse. In addition, according to the thermal conduction differential equation (TCDE) in the thermal field, the pixel value flow equation (PVFE) in the image domain is derived to mine deep potential feature information. The experimental results on multiple standard databases demonstrate that the proposed HTI-Net has superior edge detail reconstruction effect and parameter performance compared with the existing SR methods. The experimental results on the microscope chip image (MCI) database consisting of realistic low-resolution (LR) and high-resolution (HR) images show that the proposed HTI-Net for image SR reconstruction can improve the effectiveness of the hardware Trojan detection system.

In minimally invasive surgery videos, label-free monocular laparoscopic depth estimation is challenging due to smoke. For this reason, we propose a self-supervised collaborative network-based depth estimation method with smoke-removal for monocular endoscopic video, which is decomposed into two steps of smoke-removal and depth estimation. In the first step, we develop a de-endoscopic smoke for cyclic GAN (DS-cGAN) to mitigate the smoke components at different concentrations. The designed generator network comprises sharpened guide encoding module (SGEM), residual dense bottleneck module (RDBM) and refined upsampling convolution module (RUCM), which restores more detailed organ edges and tissue structures. In the second step, high resolution residual U-Net (HRR-UNet) consisting of a DepthNet and two PoseNets is designed to improve the depth estimation accuracy, and adjacent frames are used for camera self-motion estimation. In particular, the proposed method requires neither manual labeling nor patient computed tomography scans during the training and inference phases. Experimental studies on the laparoscopic data set of the Hamlyn Centre show that our method can effectively achieve accurate depth information after net smoking in real surgical scenes while preserving the blood vessels, contours and textures of the surgical site. The experimental results demonstrate that the proposed method outperforms existing state-of-the-art methods in effectiveness and achieves a frame rate of 94.45fps in real time, making it a promising clinical application.

For image local forgery detection, the existing methods require a large amount of labeled data for training, and most of them cannot detect multiple types of forgery simultaneously. In this paper, we firstly analyzed the JPEG compression traces which are mainly caused by different JPEG compression chains, and designed a trace extractor to learn such traces. Then, we utilized the trace extractor as the backbone and trained self-supervised to strengthen the discrimination ability of learned traces. With its benefits, regions with different JPEG compression chains can easily be distinguished within a forged image. Furthermore, our method does not rely on a large amount of training data, and even does not require any forged images for training. Experiments show that the proposed method can detect image local forgery on different datasets without re-training, and keep stable performance over various types of image local forgery.

Visible-infrared person re-identification (VI-ReID) aims to retrieve the person images of the same identity from the RGB to infrared image space, which is very important for real-world surveillance system. In practice, VI-ReID is more challenging due to the heterogeneous modality discrepancy, which further aggravates the challenges of traditional single-modality person ReID problem, i.e., inter-class confusion and intra-class variations. In this paper, we propose an aggregated memory-based cross-modality deep metric learning framework, which benefits from the increasing number of learned modality-aware and modality-agnostic centroid proxies for cluster contrast and mutual information learning. Furthermore, to suppress the modality discrepancy, the proposed cross-modality alignment objective simultaneously utilizes both historical and up-to-date learned cluster proxies for enhanced cross-modality association. Such training mechanism helps to obtain hard positive references through increased diversity of learned cluster proxies, and finally achieves stronger ``pulling close'' effect between cross-modality image features. Extensive experiment results demonstrate the effectiveness of the proposed method, surpassing state-of-the-art works significantly by a large margin on the commonly used VI-ReID datasets.

Medical image segmentation is a critical task in the field of medical image analysis. Recently, there has been increasing attention on scribble-supervised medical image segmentation due to its simplicity for label generation. However, the performance of a scribble-based task highly relies on the quality of learning from inconsistent annotations and the classification of pixels in high-entropy regions. In this paper, we propose a novel framework called SRPA that combines both ScribbleMatch and Reliable-Guided Pixel Alignment to enhance the performance of the scribble-based task. The ScribbleMatch technique utilizes the pseudo label incorporated from two different weakly perturbed views of the same image to supervise a strongly perturbed view, which assists in boosting the quality of the shape information learning as the scribble-based task lacks the prototypes to consistently capture shape prior to model training. The Reliable-Guided Pixel Alignment technique employs reliable pixels selected by contrasting two weakly perturbed views, which serve as the standard for blurred pixels in strongly perturbed images to maximally align. This ensures the reliable classification of high-entropy pixels. Our method is evaluated on the public ACDC and MSCMRseg datasets, and the results demonstrate that our approach surpasses current scribble-supervised segmentation methods. Code will be available at https://github.com/RheinSXY/SRPA.

Deformable image registration is a basic building block in intelligent bioinformatical analysis and biomedicine systems. With the rapid development of deep learning, the community has witnessed a great leap via this effective data-driven technique. Recently, the Vision Transformer, famous by its long-range modeling ability, has been successfully used in the field of medical image registration. However, the existing ViT based techniques are deemed to have certain limitations. Firstly, these methods often transplanted the transformer module directly into the networks, while did not dive deeper to explore its compatibility to practical registration tasks. Moreover, self-attention’s relatively rigid all-to-all patching strategy may cause undesirable discontinuity effect to the spatial calculation. To address these issues, we propose a novel medical image registration framework based on an efficient image prior learning and fusion mechanism. Unlike the existing prior-based registration methods, our model is capable of learning task-specific saliency priors, without the need of hand-crafted features, or heavy-loaded auxiliary tasks, or pre-acquired expensive annotations. Then, followed by a multi-scale patch embedding module, the self-driven saliency prior is integrated into a ViT block with an active feature fusion mechanism, to further expand our network’s structural learning abilities. Extensive experiments on multiple data sets have demonstrated the superior quality of the proposed framework. We believe this plug-and-play model will bring about more application potentials to the community (Project webpage: https://github.com/raincity212/SPF-Net).

Due to the lack of temporal annotation, current Weakly-supervised Temporal Action Localization (WTAL) methods are generally stuck into over-complete or incomplete localization. In this paper, we aim to leverage the text information to boost WTAL from two aspects, i.e., (a) the discriminative objective to enlarge the inter-class difference, thus reducing the over-complete; (b) the generative objective to enhance the intra-class integrity, thus finding more complete temporal boundaries. For the discriminative objective, we propose a Text-Segment Mining (TSM) mechanism, which constructs a text description based on the action class label, and regards the text as the query to mine all class-related segments. Without the temporal annotation of actions, TSM compares the text query with the entire videos across the dataset to mine the best matching segments while ignoring irrelevant ones. Due to the shared sub-actions in different categories of videos, merely applying TSM is too strict to neglect the semantic-related segments, which results in incomplete localization. We further introduce a generative objective named Video-text Language Completion (VLC), which focuses on all semantic-related segments from videos to complete the text sentence. We achieve the state-of-the-art performance on THUMOS14 and ActivityNetl.3. Surprisingly, we also find our proposed method can be seamlessly applied to existing methods, and improve their performances with a clear margin. The code is available at https://github.com/lgzlIlIlI/Boosting-WTAL.

State-of-the-art 3D object detectors are usually trained on large-scale datasets with high-quality 3D annotations. However, such 3D annotations are often expensive and time-consuming, which may not be practical for real applications. A natural remedy is to adopt semi-supervised learning (SSL) by leveraging a limited amount of labeled samples and abundant unlabeled samples. Current pseudo-labeling-based SSL object detection methods mainly adopt a teacher-student framework, with a single fixed threshold strategy to generate supervision signals, which inevitably brings confused supervision when guiding the student network training. Besides, the data augmentation of the point cloud in the typical teacher-student framework is too weak, and only contains basic down sampling and flip-and-shift (i.e., rotate and scaling), which hinders the effective learning of feature information. Hence, we address these issues by introducing a novel approach of Hierarchical Supervision and Shuffle Data Augmentation (HSSDA), which is a simple yet effective teacher-student framework. The teacher network generates more reasonable supervision for the student network by designing a dynamic dual-threshold strategy. Besides, the shuffle data augmentation strategy is designed to strengthen the feature representation ability of the student network. Extensive experiments show that HSSDA consistently outperforms the recent state-of-the-art methods on different datasets. The code will be released at https://github.com/azhuantou/HSSDA.

Semi-supervised learning is a promising method for medical image segmentation under limited annotation. However, the model cognitive bias impairs the segmentation performance, especially for edge regions. Furthermore, current mainstream semi-supervised medical image segmentation (SSMIS) methods lack designs to handle model bias. The neural network has a strong learning ability, but the cognitive bias will gradually deepen during the training, and it is difficult to correct itself. We propose a novel mutual correction framework (MCF) to explore network bias correction and improve the performance of SSMIS. Inspired by the plain contrast idea, MCF introduces two different subnets to explore and utilize the discrepancies between subnets to correct cognitive bias of the model. More concretely, a contrastive difference review (CDR) module is proposed to find out inconsistent prediction regions and perform a review training. Additionally, a dynamic competitive pseudo-label generation (DCPLG) module is proposed to evaluate the performance of subnets in real-time, dynamically selecting more reliable pseudo-labels. Experimental results on two medical image databases with different modalities (CT and MRI) show that our method achieves superior performance compared to several state-of-the-art methods. The code will be available at https://github.com/WYC-321/MCF.

For learning-based binary descriptors, the binarization process has not been well addressed. The reason is that the binarization blocks gradient back-propagation. Existing learning-based binary descriptors learn real-valued output, and then it is converted to binary descriptors by their proposed binarization processes. Since their binarizaiion processes are not a component of the network, the learning-based binary descriptor cannot fully utilize the advances of deep learning. To solve this issue, we propose a model-agnostic plugin binary transformation layer (BTL), making the network directly generate binary descriptors. Then, we present the first self-supervised, direct-learned binary descriptor, dubbed DLBD. Furthermore, we propose ultra-wide temperature-scaled crossentropy loss to adjust the distribution of learned descriptors in a larger range. Experiments demonstrate that the proposed BTL can substitute the previous binarization process. Our proposed DLBD outperforms SOTA on different tasks such as image retrieval and classification11Our code is available at: https://github.com/CQUPT-CV/DLBD.

Very high-resolution (VHR) aerial image scene classification is an essential task for aerial image understanding. Although transformer-based models have demonstrated strong ability in natural image classification, transformer-based methods on VHR aerial image tasks are still lack of concern because the complexity of self-attention in the transformer grows quadratically with the image resolution. To address this issue, we decompose the self-attention via Fourier Transform and propose a novel Fourier self-attention (FSA) mechanism. Based on FSA, we design a highly efficient network named DecomFormer, which learns contextual relationships in the real part and imaginary part of the Fourier field, respectively. Theoretically, the DecomFormer reduces the complexity of the naive self-attention mechanism from O(n2) to O(nlog(n)). Universal experiments on public VHR aerial image classification benchmarks demonstrated the DecomFormer’s efficiency, especially on images with very high-resolution.

Recent years, deep-learning-based methods achieve remarkable improvements on the super-resolution (SR) task. However, recovering high-quality (HQ) texture from the low-quality (LQ) aerial image is still challenging due to the limited contextual modeling ability of current deep-learning methods as well as the sharp artificial texture of aerial images. In this paper, we rethink aerial image super resolution (AISR) task with the perspective of Fourier analysis. Firstly, we build the Fourier Global Convolution (FGC) inspired by the convolution theorem of the Fourier Transform to extract the shadow features. Then, following the Gabor Transform, a carefully designed oriented Texture Contextual Block (OTCB) is proposed to enhance the oriented texture representation. By stacking FGC and OTCB, we propose a simple but effective straight-forward network named Fourier Consistency Image Reconstruction Model (FCIR) to restore HQ aerial image. Moreover, we design a gradient consistency loss (GC Loss) to enhance the quality of reconstructed high-frequency details. Compared with very recent state-of-the-art super-resolution methods, experimental results demonstrate promising SR performance boosts from FCIR on 3 typical aerial image datasets.

Growing leakage and misuse of visual information raise security and privacy concerns, which promotes the development of information protection. Existing adversarial perturbations-based methods mainly focus on the de-identification against deep learning models. However, the inherent visual information of the data has not been well protected. In this work, inspired by the Type-I adversarial attack, we propose an Adversarial Visual Information Hiding (AVIH) method to protect the visual privacy of data. Specifically, the method generates obfuscating adversarial perturbations to obscure the visual information of the data. Meanwhile, it maintains the hidden objectives to be correctly predicted by models. In addition, our method does not modify the parameters of the applied model, which makes it flexible for different scenarios. Experimental results on the recognition and classification tasks demonstrate that the proposed method can effectively hide visual information and hardly affect the performances of models. The code is available at https://github.com/suzhigangssz/AVIH.

Impressive performance on point cloud semantic segmentation has been achieved by fully-supervised methods with large amounts of labelled data. As it is labour-intensive to acquire large-scale point cloud data with point-wise labels, many attempts have been made to explore learning 3D point cloud segmentation with limited annotations. Active learning is one of the effective strategies to achieve this purpose but is still under-explored. The most recent methods of this kind measure the uncertainty of each pre-divided region for manual labelling but they suffer from redundant information and require additional efforts for region division. This paper aims at addressing this issue by developing a hierarchical point-based active learning strategy. Specifically, we measure the uncertainty for each point by a hierarchical minimum margin uncertainty module which considers the contextual information at multiple levels. Then, a feature-distance suppression strategy is designed to select important and representative points for manual labelling. Besides, to better exploit the unlabelled data, we build a semi-supervised segmentation framework based on our active strategy. Extensive experiments on the S3DIS and ScanNetV2 datasets demonstrate that the proposed framework achieves 96.5% and 100% performance of fully-supervised baseline with only 0.07% and 0.1% training data, respectively, outperforming the state-of-the-art weakly-supervised and active learning methods. The code will be available at https://github.com/SmiletoE/HPAL.

Single hyperspectral image super-resolution (single-HSI-SR) aims to restore a high-resolution hyperspectral image from a low-resolution observation. However, the prevailing CNN-based approaches have shown limitations in building long-range dependencies and capturing interaction information between spectral features. This results in inadequate utilization of spectral information and artifacts after upsampling. To address this issue, we propose ES-SAformer, an ESSA attention-embedded Transformer network for single-HSI-SR with an iterative refining structure. Specifically, we first introduce a robust and spectral-friendly similarity metric, i.e., the spectral correlation coefficient of the spectrum (SCC), to replace the original attention matrix and incorporates inductive biases into the model to facilitate training. Built upon it, we further utilize the kernelizable attention technique with theoretical support to form a novel efficient SCC-kernel-based self-attention (ESSA) and reduce attention computation to linear complexity. ESSA enlarges the receptive field for features after upsampling without bringing much computation and allows the model to effectively utilize spatial-spectral information from different scales, resulting in the generation of more natural high-resolution images. Without the need for pretraining on large-scale datasets, our experiments demonstrate ESSA’s effectiveness in both visual quality and quantitative results. The code will be released at ESSAformer.

Attention-based arbitrary style transfer studies have shown promising performance in synthesizing vivid local style details. They typically use the all-to-all attention mechanism—each position of content features is fully matched to all positions of style features. However, all-to-all attention tends to generate distorted style patterns and has quadratic complexity, limiting the effectiveness and efficiency of arbitrary style transfer. In this paper, we propose a novel all-to-key attention mechanism—each position of content features is matched to stable key positions of style features—that is more in line with the characteristics of style transfer. Specifically, it integrates two newly proposed attention forms: distributed and progressive attention. Distributed attention assigns attention to key style representations that depict the style distribution of local regions; Progressive attention pays attention from coarse-grained regions to fine-grained key positions. The resultant module, dubbed StyA2K, shows extraordinary performance in preserving the semantic structure and rendering consistent style patterns. Qualitative and quantitative comparisons with state-of-the-art methods demonstrate the superior performance of our approach. Codes and models are available on https://github.com/LearningHx/StyA2K.

The previous deep CNN-based single-image dehazing methods are devoted to improving the performance by increasing the network’s depth and width. In this paper, a novel Large Kernel Convolution Dehaze Network (LKD-Net) is proposed to enhance the network’s performance by increasing the size of the convolutional kernel. The main module in LKD-Net is the designed Large Kernel Convolution Dehaze Block (LKD Block), which consists of the Decomposition deep-wise Large Kernel Convolution Block (DLKCB) and the Channel Enhanced Feed-forward Network (CEFN). DLKCB is designed to reduce the massive amount of computational overhead and parameters of the large kernel by splitting the deep-wise large kernel convolution into a smaller depth-wise convolution and a depth-wise dilated convolution. Meanwhile, CEFN is designed to enhance the robustness of the Feed-forward Network by exploiting significant channels. The extensive experiments on RESIDE dataset demonstrate that our LKD-Net outperforms the state-of-the-art with far less computational overhead and parameters.

Semi-supervised learning is a promising approach in reducing the requirement to collect large amounts of dense annotations, especially in medical image segmentation. However, most existing semi-supervised 3D medical image segmentation methods tend to ignore the cross-slice context that contains extensive structural information. We believe cross-slice context can help the model capture semantic information complementary to slice context and achieve robust and more accurate segmentation. Therefore, in this paper, we propose a novel cross-slice context consistency framework for 3D left atrium segmentation named CSC2-Net. Our method can effectively utilize unlabeled data by encouraging consistent results between slice segmentation and cross-slice inference segmentation. To achieve this, we design a bidirectional gated context inference module (Bi-GCM) to model cross-slice context and predict slice segmentation without direct slice features. Experiments on a public left atrium (LA) databases show that our method achieves higher performance and outperforms state-of-the-art methods by imposing cross-slice consistency constraint.

Deep neural networks have been found to be vulnerable to adversarial noise. Recent works show that exploring the impact of adversarial noise on intrinsic components of data can help improve adversarial robustness. However, the pattern closely related to human perception has not been deeply studied. In this paper, inspired by the cognitive science, we investigate the interference of adversarial noise from the perspective of image phase, and find ordinarily-trained models lack enough robustness against phase-level perturbations. Motivated by this, we propose a joint adversarial defense method: a *phase-level adversarial training mechanism* to enhance the adversarial robustness on the phase pattern; an *amplitude-based pre-processing operation* to mitigate the adversarial perturbation in the amplitude pattern. Experimental results show that the proposed method can significantly improve the robust accuracy against multiple attacks and even adaptive attacks. In addition, ablation studies demonstrate the effectiveness of our defense strategy.

Deep neural networks (DNNs) are vulnerable to adversarial noise. Denoising model-based defense is a major protection strategy. However, denoising models may fail and induce negative effects in fully white-box scenarios. In this work, we start from the latent inherent properties of adversarial samples to break the limitations. Unlike solely learning a mapping from adversarial samples to natural samples, we aim to achieve denoising by destroying the spatial characteristics of adversarial noise and preserving the robust features of natural information. Motivated by this, we propose a defense based on information discard and robust representation restoration. Our method utilize complementary masks to disrupt adversarial noise and guided denoising models to restore robust-predictive representations from masked samples. Experimental results show that our method has competitive performance against white-box attacks and effectively reverses the negative effect of denoising models.

As an important research element in the field of hyperspectral remote sensing, the purpose of hyperspectral unmixing is to decompose the mixed pixels in a hyperspectral image into endmembers and abundance. With the development of deep learning, it also has good prospects for application in the field of hyperspectral unmixing. Existing hyperspectral unmixing networks tend to focus only on the spectral information in the image which neglecting the spatial information. Therefore, a network based on dual-stream convolutional feature fusion for hyperspectral unmixing is proposed to make full use of the correlation between neighboring pixels in the paper, which includes three parts: dual-stream feature extraction, feature fusion and abundance estimation. The unmixing performance of the network is verified on two real datasets and exhibits better performance compared with other state-of-the-art methods.

Unsupervised learning visible-infrared person re-identification (USL-VI-ReID) aims at learning modality-invariant features from unlabeled cross-modality dataset, which is crucial for practical applications in video surveillance systems. The key to essentially address the USL-VI-ReID task is to solve the cross-modality data association problem for further heterogeneous joint learning. To address this issue, we propose a Dual Optimal Transport Label Assignment (DOTLA) framework to simultaneously assign the generated labels from one modality to its counterpart modality. The proposed DOTLA mechanism formulates a mutual reinforcement and efficient solution to cross-modality data association, which could effectively reduce the side-effects of some insufficient and noisy label associations. Besides, we further propose a cross-modality neighbor consistency guided label refinement and regularization module, to eliminate the negative effects brought by the inaccurate supervised signals, under the assumption that the prediction or label distribution of each example should be similar to its nearest neighbors'. Extensive experimental results on the public SYSU-MM01 and RegDB datasets demonstrate the effectiveness of the proposed method, surpassing existing state-of-the-art approach by a large margin of 7.76% mAP on average, which even surpasses some supervised VI-ReID methods.

Unsupervised visible-infrared person re-identification (USL-VI-ReID) aims to match pedestrian images of the same identity from different modalities without annotations. Existing works mainly focus on alleviating the modality gap by aligning instance-level features of the unlabeled samples. However, the relationships between cross-modality clusters are not well explored. To this end, we propose a novel bilateral cluster matching-based learning framework to reduce the modality gap by matching cross-modality clusters. Specifically, we design a Many-to-many Bilateral Cross-Modality Cluster Matching (MBCCM) algorithm through optimizing the maximum matching problem in a bipartite graph. Then, the matched pairwise clusters utilize shared visible and infrared pseudo-labels during the model training. Under such a supervisory signal, a Modality-Specific and Modality-Agnostic (MSMA) contrastive learning framework is proposed to align features jointly at a cluster-level. Meanwhile, the cross-modality Consistency Constraint (CC) is proposed to explicitly reduce the large modality discrepancy. Extensive experiments on the public SYSU-MM01 and RegDB datasets demonstrate the effectiveness of the proposed method, surpassing state-of-the-art approaches by a large margin of 8.76% mAP on average.

Current face sketch-photo synthesis researches generally embrace an image-to-image (I2I) translation pipeline. However, these methods ignore the one-to-many mapping problem (i.e., multiple plausible photo results can correspond to a single input sketch) in sketch-to-photo synthesis task, resulting in significant performance degradation on diverse datasets. Besides, generating high-quality images on limited data is also a challenge for this task. To address these challenges, we propose a dual-path framework that introduces generative priors to better perform cross-domain reconstruction on limited data. The coarse path uses a layer-swapped pre-trained generator to achieve coarse cross-domain reconstruction, and the refinement path further improves the structure and texture details. To align the feature maps between the two paths, we introduce a spatial feature calibration module. Despite this, our framework still struggles to handle diverse datasets. Thanks to the flexibility of generative priors, we can extend the framework to achieve exemplar-guided I2I translation by incorporating an exemplar with style mixing and a proposed semantic-aware style refinement strategy, which addresses the one-to-many mapping problem in sketch-to-photo synthesis task. Furthermore, our framework can perform cross-domain editing by employing off-the-shelf editing methods based on the latent space, achieving fine-grained control. Extensive experiments on diverse datasets demonstrate the superiority of our framework over other state-of-the-art methods.

DEtection TRansformer (DETR) and its variants (DETRs) achieved impressive performance in general object detection. However, in crowded pedestrian detection, the performance of DETRs is still unsatisfactory due to the inappropriate sample selection method which results in more false positives. To settle the issue, we propose a simple but effective sample selection method for DETRs, Sample Selection for Crowded Pedestrians (SSCP), which consists of the constraint-guided label assignment scheme (CGLA) and the utilizability-aware focal loss (UAFL). Our core idea is to select learnable samples for DETRs and adaptively regulate the loss weights of samples based on their utilizability. Specifically, in CGLA, we proposed a new cost function to ensure that only learnable positive training samples are retained and the rest are negative training samples. Further, considering the utilizability of samples, we designed UAFL to adaptively assign different loss weights to learnable positive samples depending on their gradient ratio and IoU. Experimental results show that the proposed SSCP effectively improves the baselines without introducing any overhead in inference. Especially, Iter Deformable DETR is improved to 39.7(-2.0)% MR on Crowdhuman and 31.8(-0.4)% MR on Citypersons.

Heterogeneous face recognition (HFR) aims to match input face identity across different image modalities. Due to the existing large modality gap and the limited number of training data, HFR is still a challenging problem in biometrics and draws more and more attention. Existing researchers always extract modality invariant features or generate homogeneous images to decrease the modality gap, lacking abundant labeled data to avoid the overfitting problem. In this paper, we proposed a novel Modality-Agnostic Augmented Multi-Collaboration representation for Heterogeneous Face Recognition (MAMCO-HFR) in a semi-supervised manner. The modality-agnostic augmentation strategy is proposed to generate adversarial perturbations to map unlabeled faces into the modality-agnostic domain. The multi-collaboration feature constraint is designed to mine the inherent relationships between diverse layers for discriminative representation. Experiments on several large-scale heterogeneous face datasets (CASIA NIR-VIS 2.0, LAMP-HQ and Tufts Face dataset) prove the proposed algorithm can achieve superior performance compared with state-of-the-art methods. The source code is available at https://github.com/xiyin11/Semi-HFR.

Image aesthetic assessment (IAA) has drawn wide attention in recent years as more and more users post images and texts on the Internet to share their views. The intense subjectivity and complexity of IAA make it extremely challenging. Text triggers the subjective expression of human aesthetic experience based on human implicit memory, so incorporating the textual information and identifying the relationship with the image is of great importance for IAA. However, IAA with the image as input fails to fully consider subjectivity, while existing multimodal IAA ignores the interrelationship among modalities. To this end, we propose a brain-inspired multimodal interaction network (BMI-Net) that simulates how the association area of the cerebral cortex processes sensory stimuli. In particular, the knowledge integration LSTM (KI-LSTM) is proposed to learn the image-text interaction relation. The proposed scalable multimodal fusion (SMF) based on low-rank decomposition fuses image, text and interaction modalities to predict the aesthetic distribution. Extensive experiments show that the proposed BMI-Net outperforms existing state-of-the-art methods on three IAA tasks.