Without manual annotations, unsupervised cross-modal hashing (UCMH) aims to achieve efficient clustering and retrieval by leveraging data interrelationships. However, the retrieval accuracy is constrained by two main aspects: 1) insufficient exploration of data relationships; 2) existing knowledge mining strategies are not well aligned with the architectural properties of multilayer perceptrons. Through summary and error analysis, the human brain is able to achieve fast learning through experience and minimal data. Inspired by this cognitive process, we propose a novel Error Notebook strategy, named ENHash, to more effectively capture similarity information between multi-modal data for fine-grained unsupervised clustering. Firstly, simulating the human process of summarizing experiences, ENHash gradually integrates the information from each batch into a global clustering representation. Secondly, drawing upon human error analysis capabilities, ENHash utilizes the summarized experiences to identify and record incorrectly predicted hash codes. Finally, by leveraging the knowledge derived from this analysis, ENHash guides the hash function to learn fine-grained patterns from the errors. To the best of our knowledge, ENHash represents the first attempt at integrating cognitively-inspired mechanisms into fine-grained UCMH optimization paradigms. We evaluate the proposed ENHash against eight state-of-the-art methods on three widely used datasets and one fine-grained cross-modal dataset. Experimental results show that ENHash achieves substantial improvements over existing approaches.

Unsupervised cross-modal hashing (UCMH) has attracted considerable attention owing to its minimal reliance on manual annotations and low retrieval latency. However, existing UCMH methods based on contrastive learning frameworks combined with multilayer perceptrons (MLPs) often suffer from two key limitations: the inherent challenge in constructing reliable positive-negative sample pairs under unsupervised settings, and the restricted representational capacity of static network architectures. Inspired by cognitive mechanisms observed in the human brain, where specialized regions support distinct roles in knowledge acquisition and error-driven learning, we propose a Brain-inspired Region-Aligned Interaction Network for unsupervised cross-modal Hashing (BRAINHash), which is a novel brain-inspired memory-based temporal modeling strategy. BRAINHash emulates several functional components of the brain through modular design choices: 1) diverse encoders approximate feature encoding akin to occipital lobe processing, 2) an error-aware optimization strategy models initial cross-modal association construction analogous to prefrontal cortex activity, 3) a teacher network serves as hippocampal-like memory storage capturing learned representations as persistent memory points, 4) we incorporate soft-target contrastive objectives alongside spiking neural network-based temporal modeling to simulate higher-order reasoning typically attributed to prefrontal decision-making circuits. To the best of our knowledge, BRAINHash represents the first integration of biologically inspired architectural principles with temporal dynamics into an unsupervised cross-modal hashing paradigm. Extensive evaluations conducted on five widely-used datasets demonstrate that our method outperforms fifteen state-of-the-art approaches. The implementation code is publicly available at https://github.com/YSU-ISU-Lab/BRAIN.

This work focuses on AIGC detection to develop universal detectors capable of identifying various types of forgery images. Recent studies have found large pre-trained models, such as CLIP, are effective for generalizable deepfake detection along with linear classifiers. However, two critical issues remain unresolved: 1) understanding why CLIP features are effective on deepfake detection through a linear classifier; and 2) exploring the detection potential of CLIP. In this study, we delve into the underlying mechanisms of CLIP's detection capabilities by decoding its detection features into text and performing word frequency analysis. Our finding indicates that CLIP detects deepfakes by recognizing similar concepts. Building on this insight, we introduce Category Common Prompt CLIP, called C2P-CLIP, which integrates the category common prompt into the text encoder to inject category-related concepts into the image encoder, thereby enhancing detection performance. Our method achieves a 12.4% improvement in detection accuracy compared to the original CLIP.

Online cross-modal hashing utilizing a progressive update strategy has attracted considerable interest due to its effectiveness and scalability in similarity search for large-scale multimodal data retrieval across various extensive multimedia datasets. Nevertheless, existing online cross-modal hashing approaches face several limitations. One major challenge lies in effectively capturing the intrinsic linkages among different heterogeneous modalities. Additionally, relying on relaxation-based strategies to solve the discrete constraint problem often introduces significant quantization errors, resulting in suboptimal solutions. To mitigate these limitations, we present a novel online hashing approach named AAOH. This method preserves the information integrity of multimodal data by decomposing the input into a common latent representation and transformation matrices, guided by adaptive weighting and nuclear norm minimization. Next, the common latent representation is aligned with the semantic label matrix, and an asymmetric hashing framework is used to enhance the model's discriminative ability, producing more compact hash codes. Finally, an iterative discrete optimization algorithm is proposed to efficiently solve the non-convex multi-variable optimization problem. A series of thorough evaluations on three well-established benchmark datasets are carried out to showcase the superior performance of the proposed AAOH method.

Recently, fine-tuning vision-language models like CLIP has led to impressive performance on downstream tasks, with approaches such as prompt tuning and adapter-based methods. However, these methods primarily rely on limited labeled data, while the potential of unlabeled data remains largely underexplored. In this paper, we propose a two-stage learning framework inspired by children's cognitive development, integrating supervised learning and self-evolving adaptation. Firstly, our DASPL framework fully integrates the complementary strengths of CLIP and MoCoV3 through a Dual-Adapter architecture, ensuring a more comprehensive feature representation in the supervised learning stage. By leveraging CLIP's high-level semantic alignment and MoCoV3's fine-grained visual distinctions, our approach captures both global and local feature information. Secondly, in the self-evolving adaptation stage, we employ a three-branch ensemble along with the classification max prediction margin criterion to refine pseudo-label selection. This strategy not only improves the overall pseudo-label quality but also effectively distinguishes between visually similar and easily confused categories, ensuring that only the most reliable samples contribute to model adaptation. Notably, our approach achieves strong performance with just a single iteration of pseudo-labeling. Experimental results demonstrate that DASPL consistently outperforms existing methods, achieving higher classification accuracy across multiple downstream tasks and out-of-distribution benchmarks. These findings highlight the effectiveness of our approach and provide new insights for few-shot learning.

Cross-Modal Hashing (CMH) has gained significant attention for its ability to learn semantic category discrimination and enable efficient retrieval. However, in practical applications, the massive amounts of multi-modal data collected from the internet often contain coarse annotations, which inevitably introduce noisy labels and degrade retrieval performance. To address this challenge, this paper proposes a dynamic optimization-based training framework, namely Dynamic Optimization Noisy Cross-Modal Hashing (DONCMH). Firstly, to alleviate the issue of overfitting to noisy labels during training, we propose a novel regularization-based noise-robust strategy that updates the target distribution with momentum to optimize clustering learning, thus avoiding over-emphasizing noisy samples. Secondly, to more accurately select high-quality training samples, we introduce ClusterOT, a novel Optimal Transport formulation explicitly tailored for Noisy Cross-Modal Hashing (NCMH), which integrates center representation learning and cross-modal alignment into a unified structure. By leveraging the spatial distribution of samples, ClusterOT effectively mitigates distribution imbalances inherent in center representation learning, thereby significantly improving the model's robustness to noisy label predictions. Finally, a robust feature learning module is employed to enhance the extraction of informative and discriminative representations from both modalities. Extensive experiments conducted on four widely used benchmark datasets demonstrate that the proposed method effectively mitigates the impact of noisy labels and significantly improves cross-modal retrieval performance.

This research addresses the challenge of developing a universal deepfake detector that can effectively identify unseen deepfake images despite limited training data. Existing frequency-based paradigms have relied on frequency-level artifacts introduced during the up-sampling in GAN pipelines to detect forgeries. However, the rapid advancements in synthesis technology have led to specific artifacts for each generation model. Consequently, these detectors have exhibited a lack of proficiency in learning the frequency domain and tend to overfit to the artifacts present in the training data, leading to suboptimal performance on unseen sources. To address this issue, we introduce a novel frequency-aware approach called FreqNet, centered around frequency domain learning, specifically designed to enhance the generalizability of deepfake detectors. Our method forces the detector to continuously focus on high-frequency information, exploiting high-frequency representation of features across spatial and channel dimensions. Additionally, we incorporate a straightforward frequency domain learning module to learn source-agnostic features. It involves convolutional layers applied to both the phase spectrum and amplitude spectrum between the Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (iFFT). Extensive experimentation involving 17 GANs demonstrates the effectiveness of our proposed method, showcasing state-of-the-art performance (+9.8\%) while requiring fewer parameters. The code is available at https://github.com/chuangchuangtan/FreqNet-DeepfakeDetection.

Recently, the proliferation of highly realistic synthetic images, facilitated through a variety of GANs and Diffusions, has significantly heightened the susceptibility to misuse. While the primary focus of deepfake detection has traditionally centered on the design of detection algorithms, an investigative inquiry into the generator architectures has remained conspicuously absent in recent years. This paper contributes to this lacuna by rethinking the architectures of CNN-based generator, thereby establishing a generalized representation of synthetic artifacts. Our findings illuminate that the up-sampling operator can, beyond frequency-based artifacts, produce generalized forgery artifacts. In particular, the local interdependence among image pixels caused by upsampling operators is significantly demon-strated in synthetic images generated by GAN or diffusion. Building upon this observation, we introduce the concept of Neighboring Pixel Relationships(NPR) as a means to capture and characterize the generalized structural artifacts stemming from up-sampling operations. A comprehensive analysis is conducted on an open-world dataset, comprising samples generated by 28 distinct generative models. This analysis culminates in the establishment of a novel state-of-the-art performance, showcasing a remarkable 12.8% im-provement over existing methods. The code is available at https://github.com/chuangchuangtan/NPR-DeepfakeDetection.

Video camouflaged object detection aims to identify objects that are visually concealed within the surroundings in a video. Most of the existing methods fall into analyzing the implicit inter-frame motion to capture the camouflaged object. However, due to a lack of exploring the prior explicit motion of the camouflaged object, these works generally encounter difficulty in capturing the complete camouflaged object. To address this issue, we propose to integrate implicit and explicit motion learning into a unified framework, namelyImplicit-Explicit Motion Learning network (IMEX), for video camouflaged object detection. Specifically, to promote the identifiability of the camouflaged object, a cross-scale representation fusion was proposed for global inter-frame alignment. By establishing cross-scale temporal-spatial association and aggregating the temporal-spatial attentive representations, it also achieves an elimination of the implicit motion of inter-frame to some extent. Moreover, to further improve the discriminability of boundary regions of the detected object, an explicit motion-induced consistency preserving of camouflaged objects is proposed, in which the prior boundary-aware explicit motion field is leveraged to supervise the consistency of camouflaged objects in consecutive frames. Extensive experiments show that our proposed IMEX achieves substantial performance improvements by a large margin.

Recently, there has been a significant advancement in image generation technology, known as GAN. It can easily generate realistic fake images, leading to an increased risk of abuse. However, most image detectors suffer from sharp performance drops in unseen domains. The key of fake image detection is to develop a generalized representation to describe the artifacts produced by generation models. In this work, we introduce a novel detection framework, named Learning on Gradients (LGrad), designed for identifying GAN-generated images, with the aim of constructing a generalized detector with cross-model and cross-data. Specifically, a pretrained CNN model is employed as a transformation model to convert images into gradients. Subsequently, we leverage these gradients to present the generalized artifacts, which are fed into the classifier to ascertain the authenticity of the images. In our framework, we turn the data-dependent problem into a transformation-model-dependent problem. To the best of our knowledge, this is the first study to utilize gradients as the representation of artifacts in GAN-generated images. Extensive experiments demonstrate the effectiveness and robustness of gradients as generalized artifact representations. Our detector achieves a new state-of-the-art performance with a remarkable gain of 11.4%. The code is released at https://github.com/chuangchuangtan/LGrad.

Deep hashing methods have achieved tremendous success in cross-modal retrieval, due to its low storage consumption and fast retrieval speed. Supervised cross-modal hashing methods have achieved substantial advancement by incorporating semantic information. However, to a great extent, supervised methods rely on large-scale labeled cross-modal training data which are laborious to obtain. Moreover, most cross-modal hashing methods only handle two modalities of image and text, without taking the scene of multiple modalities into consideration. In this paper, we propose a novel semi-supervised approach called semi-supervised knowledge distillation for cross-modal hashing (SKDCH) to overcome the above-mentioned challenges, which enables guiding a supervised method using outputs produced by a semi-supervised method for multimodality retrieval. Specifically, we utilize teacher-student optimization to propagate knowledge. Furthermore, we improves triplet ranking loss to better mitigate the heterogeneity gap, which increases the discriminability of our proposed approach. Extensive experiments executed on two benchmark datasets validate that the proposed SKDCH surpasses the state-of-the-art methods.

This article focuses on generating object locations in a given image while only using image-level annotations. Towards this end, we present a simple and effective training-free framework, named Dual-Gradients Localization (DGL) framework. The key idea of the proposed DGL framework is to leverage two kinds of gradients to achieve precise localization on any convolutional layer of a classification model during the testing stage. Concretely, the DGL framework is developed based on two branches: 1) Pixel-level Class Selection, leveraging gradients of the target class to identify the correlation ratio of pixels to the target class within any convolutional feature maps, and 2) Class-aware Enhanced Maps, utilizing linear relationship in gradients of the classification loss function to mine entire target object regions. To further polish the details of objects, we apply the skip-layer connections to the classification model, which concatenates the high- and low-level layers to achieve classification. In such a case, DGL with Skip-layer Connections (DGL-SC) can capture more edge information on the high-level layer. In addition, we propose a Localization Maps Selection method to evaluate the quality of the localization map and provide a way for automatically selecting localization maps produced on different layers. Extensive experiments on public ILSVRC and CUB-200-2011 datasets show the effectiveness of the proposed DGL framework. Especially, our DGL-SC obtains a new state-of-the-art gt-known localization error of 27.35% on the ILSVRC benchmark.

Weakly supervised object localization exploits the last convolutional feature maps of classification model and the weights of Fully-Connected (FC) layer to achieves localization. However, high-level feature maps for localization lack edge features. Additionally, the weights are specific to classification task, causing only discriminative regions to be discovered. In order to fuse edge features and adjust the attention distribution for feature map channels, we propose an efficient method called Attention-based Dual-Branches Localization (ADBL) Network, in which dual-branches structure and attention mechanism are adopted to mine edge features and non-discriminative features for locating more target areas. Specifically, dual-branches structure cascades low-level feature maps to mine target object edge regions. Additionally, during inference stage, attention mechanism assigns appropriate attention for different features to preserve non-discriminative areas. Extensive experiments on both ILSVRC and CUB-200-2011 datasets show that the ADBL method achieves substantial performance improvements.

The feature representation of images is the key to bridge the semantic gap and make computer understand images in the tasks of image retrieval. Semantic regions and salient targets are irreplaceable parts of cognitive images in image recognition. This paper proposes an unsupervised image retrieval method based on multi-semantic region weighting and multi-scale flatness weighting. Firstly, we divide the semantic regions by using the Fully Convolutional Network (FCN) and calculate the multi-Semantic weight map (S-mask) to obtain the global features. Secondly, we introduce a flatness-weighted strategy to weight feature maps and aggregate the multi-scale features to obtain the local features. Finally, we cascade the global features and the local features to construct the final image representation. Experimental results on two widely-used databases demonstrate that the proposed method is effective and significantly outperforms the state-of-the-art retrieval methods.

Weakly Supervised Object Localization (WSOL) aims to learn object locations in a given image while only using image-level annotations. For highlighting the whole object regions instead of the discriminative parts, previous works often attempt to train classification model for both classification and localization tasks. However, it is hard to achieve a good tradeoff between the two tasks, if only classification labels are employed for training on a single classification model. In addition, all of recent works just perform localization based on the last convolutional layer of classification model, ignoring the localization ability of other layers. In this work, we propose an offline framework to achieve precise localization on any convolutional layer of a classification model by exploiting two kinds of gradients, called Dual-Gradients Localization (DGL) framework. DGL framework is developed based on two branches: 1) Pixel-level Class Selection, leveraging gradients of the target class to identify the correlation ratio of pixels to the target class within any convolutional feature maps, and 2) Class-aware Enhanced Maps, utilizing gradients of classification loss function to mine entire target object regions, which would not damage classification performance. Extensive experiments on public ILSVRC and CUB-200-2011 datasets show the effectiveness of the proposed DGL framework. Especially, our DGL obtains a new state-of-the-art Top-1 localization error of 43.55% on the ILSVRC benchmark.

Generally distance-based similarity estimation between two images is not always reliable due to the limitations in both image understanding techniques and distance measure methods. This paper presents a novel approach for improving the similarity estimation through introducing the distribution information of similarity scores. The key idea is based on an underlying assumption that the distributions of similarity scores are similar for true-relevant images when they query an independent database. By representing each distribution with the area under the corresponding similarity score curve, the difference between different distributions can be easily calculated and employed to update the original distance measure. Experiments on three public datasets with various feature representations show that the enhanced similarity estimation remarkably outperforms the original distance measure and the proposed approach also keeps a good generalization ability on various datasets and feature representations.