The goal of image change captioning is to capture the content differences between two images and describe them in natural language. The key is how to learn stable content changes from noise such as viewpoint and image structure. However, current work mostly focuses on identifying changes, and the influence of global noise leads to unstable recognition of global features. In order to tackle this problem, we propose a Self-supervised Global-Part Alignment (SSGPA) network and revisit the image change captioning task by enhancing the construction process of overall image global features, enabling the model to integrate global changes such as viewpoint into local changes, and to detect and describe changes in the image through alignment. Concretely, we first design a Global-Part Transport Alignment mechanism to enhance global features and learn stable content changes through a self-supervised method of optimal transport. Further, we design a Change Fusion Adapter with pre-trained vision-language model to enhance the similar parts features of paired images, thereby enhancing global features, and expanding content changes. Extensive experiments show our method achieves the state-of-the-art results on four datasets.

Large language models (LLMs) increasingly rely on lengthy prompts to achieve complex tasks. However, longer prompts not only increase inference costs but pose significant challenges for limited context windows of LLMs. Prompt compression aims to accelerate inference in long-text scenarios while preserving performance in various tasks. Conventional extractive compression methods often lead to semantic incoherence and loss of primitive key information. To this end, we propose a Generative PrompT Compression (GPT-C) paradigm. In the absence of appropriate datasets, we design a Collaborative Ordered Agent (Co-Agent) framework to distill high-quality datasets. We then develop a task-agnostic, length-adaptive reinforcement learning strategy that controls compression length and enhances the generalization across diverse tasks. Extensive experiments demonstrate that GPT-C achieves performance comparable to the original prompts while utilizing only 20% of the tokens.

The goal of the image change captioning task is to capture the differences between two similar images and describe them in natural language. In this paper, we decompose this task into two sub-problems, i.e., fine-grained change feature learning and discrimination of changed regions. Compared with existing methods which only focus on change feature learning, we propose a novel change captioning learning paradigm, Find and Perceive (F&P). Our proposed F&P consists of two main ideas, i.e., the Fine-Grained Semantic Change Perception (FGSCP) module for improving the model's perception ability of subtle changes and the Weakly-Supervised Discriminator (WSD) of changed regions for improving the model's sensitivity of localising the important regions. Specifically, the FGSCP deploys a two-step manner, firstly introducing the fine-grained categorisation and then enhancing the interaction of the two paired images. And the WSD adopts the contributions of each image region for final generated captions, accurately indicating which regions are important for change captions without any extra annotations. Finally, we conduct extensive experiments on four change captioning datasets, and experimental results show that our proposed method F&P outperforms existing change caption methods and achieves new state-of-the-art performance.

Abstract Image captioning aims to automatically generate a natural language description of a given image, and most state‐of‐the‐art models have adopted an encoder–decoder transformer framework. Such transformer structures, however, show two main limitations in the task of image captioning. Firstly, the traditional transformer obtains high‐level fusion features to decode while ignoring other‐level features, resulting in losses of image content. Secondly, the transformer is weak in modelling the natural order characteristics of language. To address theseissues, the authors propose a HI erarchical and S equential T ransformer ( HIST ) structure, which forces each layer of the encoder and decoder to focus on features of different granularities, and strengthen the sequentially semantic information. Specifically, to capture the details of different levels of features in the image, the authors combine the visual features of multiple regions and divide them into multiple levels differently. In addition, to enhance the sequential information, the sequential enhancement module in each decoder layer block extracts different levels of features for sequentially semantic extraction and expression. Extensive experiments on the public datasets MS‐COCO and Flickr30k have demonstrated the effectiveness of our proposed method, and show that the authors’ method outperforms most of previous state of the arts.

Fine-Grained Visual Classification (FGVC) is a challenging task, due to the small variation of visual representations from different categories. An effective solution is utilizing the bounding boxes centering the object parts to extract the discriminative representations. However, regular rectangles contains the background when the shape of the part is irregular, which may interfere with the classification. In this paper, we propose a weighted focus-attention deep network (FA-Net) to address the problem of background interference in fine-grained classification. In our FA-Net, a focus-attention module is proposed to identify the foreground region from the class activation map and remove the background. Two branches are employed to obtain the primary and secondary attention regions with focus-attention module, and a weighted layer is utilized to integrate the attention regions. Experiment results on three challenging fine-grained classification datasets (e.g., CUB-200-2011, Stanford Dogs and FGVC Aircraft) show that our FA-Net obtains state-of-the-art results and outperforms the other fine-grained algorithms.