Traditional referring expression comprehension (REC) aims to locate the target referent in an image guided by a text query. Several previous methods have studied on the Counterfactual problem in REC (C-REC) where the objects for a given query cannot be found in the image. However, these methods focus on the overall image-text or specific attribute mismatch only. In this paper, we address the C-REC problem from a deep perspective of fine- grained attributes. To this aim, we first propose a fine-grained counterfactual sample generation method to construct C-REC datasets. Specifically, we leverage pre-trained language model such as BERT to modify the attribute words in the queries, obtaining the corresponding counterfactual samples. Furthermore, we propose a C-REC framework. We first adopt three encoders to extract image, text and attribute features. Then, our dual-branch attentive fusion module fuses these cross-modal features with two branches by an attention mechanism. At last, two prediction heads generate a bounding box and a counterfactual label, respectively. In addition, we incorporate contrastive learning with the generated counterfactual samples as negatives to enhance the counterfactual perception. Extensive experiments show that our framework achieves promising performance on both public REC datasets RefCOCO/+lg and our constructed C-REC datasets C-RefCOCO/+lg. The code and data are available at https://github.com/Glacier0012/CREC.

Previous works on the task of weakly supervised phrase grounding (WSG) rely heavily on object detectors providing RoIs for the localization. However, such methods cannot be applied effectively to real-world scenarios largely because that the detectors are trained with limited categories. In this paper, we propose a refinement-based approach to WSG through fine-tuning a detector-free phrase grounding model with a visual prompt. This visual prompt is extracted from the text-related representations in CLIP. Furthermore, we combine the visual prompt with learnable features and then fine-tune the grounding network. Our experimental results significantly outperform state-of-the-art methods on the WSG task and shows the effectiveness of our method.

The malicious use of deepfake technologies poses a significant threat to social security, emphasizing the urgent necessity to advance deepfake detection. Existing detection models tend to overfit specific forgery traces within the training set, resulting in a weak generalization performance on unseen data. Considering that the feature distribution of real images is limited compared to the diverse forgery patterns, capturing the real feature space and treating features outside the distribution as potential forgery traces can mitigate overfitting and enhance the detector's generalization ability. Furthermore, since facial manipulation generates artifacts in pixels and disrupts the global consistency of the data, potential forgery traces at both pixel and feature levels can contribute to detection. In this paper, we propose a novel two-stage deepfake detection model named CodeDetector, which utilizes a codebook to capture the feature space of real faces and obtain potential forgery traces by facial reconstruction for detection. In the codebook learning stage, a codebook is used to capture the real feature distribution. In the detector training stage, we obtain pixel-level and feature-level residuals as potential forgery traces through facial reconstruction on real and fake faces to guide the model's attention to forgery clues. Specifically, we propose a Quantized Residual-Guided Attention Module and a Dual Residual Attention Module, calculating residuals and utilizing the attention mechanism to enhance global feature representation. Additionally, an Indices Prediction Module is introduced to ensure the accuracy of residual guidance that enhances the robustness of reconstruction during detection. Extensive experiments have demonstrated that CodeDetector outperforms state-of-the-art in deepfake detection cross-dataset benchmark.

Phrase Grounding, i.e., PG aims to locate objects referred by noun phrases. Recently, PG under weak supervision (i.e., grounding without region-level annotations) and zero-shot PG (i.e., grounding from seen categories to unseen ones) are proposed, respectively. However, for real-world applications these two approaches are limited due to slight annotations and numerable categories during training. In this paper, we propose a framework of zero-shot PG under weak supervision. Specifically, our PG framework is built on triple alignment strategies. Firstly, we propose a region-text alignment (RTA) strategy to build region-level attribute associations via CLIP. Secondly, we propose a domain alignment (DomA) strategy by minimizing the difference between distributions of seen classes in the training and those of the pre-training. Thirdly, we propose a category alignment (CatA) strategy by considering both category semantics and region-category relations. Extensive experimental results show that our proposed PG framework outperforms previous zero-shot methods and weakly-supervised methods. Our code is available at https://github.com/LinPengyue/ZS-WSG.