Change captioning is a task that describes changes in image pairs using natural language. This task is more complex than single-image captioning as it requires a comprehensive understanding of each image and the ability to recognize and describe the semantic changes in image pairs. The key challenge lies in making the network generate an accurate and stable change representation under the interference of viewpoint shift. In this paper, we propose a cross-view and multi-step interaction network to generate robust change representation to resist pseudo-change. Specifically, in the intra-image representation learning stage, a cross-view interaction encoder is designed to enhance internal relationships by cross-referencing in image pairs. In the change feature learning stage, a multi-step change perceptron is employed to capture the change semantics from coarse to fine progressively. Then, a fusion module dynamically combines them as a fine-grained change representation. Besides, we propose a backward representation reconstruction module that facilitates the capture of semantic changes, thus improving the quality of captions in a self-supervised manner. Extensive experiments have shown that the method effectively captures real semantic changes under the interference of viewpoint shift and achieves state-of-the-art performance on five public datasets. The code is available at https://github.com/TTXiann/CVMSI

Remote Sensing Image Change Captioning (RSICC) plays a critical role in automated environmental monitoring by generating natural language descriptions that provide intuitive interpretations of changes between bi-temporal remote sensing images. Despite recent advancements, existing methods suffer from two fundamental limitations: (1) static encoding architectures fail to account for scene complexity and semantic diversity during feature extraction, leading to inflexible representation learning; and (2) symmetric computational structures are inherently unsuitable for modeling the asymmetric temporal dependencies inherent in “before-to-after” image pairs. To address these challenges, we propose a Dynamic Asymmetric Encoder (DAE), which introduces two key innovations. First, we design a difference-guided dynamic convolution module that adaptively adjusts convolutional parameters using input-driven scaling factors and offsets, thereby enabling scene-aware intra-image feature enhancement. Second, we develop a Multi-expert Temporal Interaction (METI) module that establishes an asymmetric computational topology: the “after” image branch actively perceives change information relative to the “before” image through three heterogeneous interaction experts, followed by feature fusion. This design allocates greater computational capacity to the “after” image branch while preserving temporal coherence. Furthermore, we introduce a Multi-level Feature Aggregator (MFA) that enhances the representation of salient changed regions across multiple scales via an iterative reinforcement mechanism. Experimental results validate the effectiveness of the proposed method, demonstrating state-of-the-art performance on two benchmark RSICC datasets. We publicly release our code repository at https://github.com/TTXiann/Dynamic-Asymmetric to facilitate future research.

Internal relationship of image objects has contributed significantly to the development of image captioning, especially when combined with Transformer architecture. Most of these methods only calculate the relationship between entities and ignore the information between entities and background. Besides, the way of exploring the relational information inside the image can also be extended. In this paper, we continually explore the relationship between objects from both internal and external perspectives, and embed the vital image global information into the internal relationship module. To validate the effectiveness of our model, we conduct extensive experiments on the most popular MSCOCO dataset, and achieve state-of-the-art performance on both online and offline test sets.

In recent years, image caption methods based on the encoder-decoder framework have made promising achievements, but most of them lack the exploitation of global information. In general, visual global information can provide more fine-grain details for recognizing small objects. On the other hand, the textual global information provides a coarse understanding of the visual scene. In this paper, we propose Dual Global Enhanced Transformer (DGET) to explicitly utilize both visual and textual global information. In encoding stages, we complement two visual features with different properties to obtain a global enhanced visual representation by a novel Global Enhanced Encoder (GEE). During decoding, we proposed Global Enhanced Decoder (GED) to utilize the textual global information explicitly. To validate our model, we conduct extensive experiments on the COCO image captioning dataset and achieve superior performance over many state-of-the-art methods.

Transformer-based architectures have shown great success in image captioning. However, the calculation process of the model is predefined, which is heavily dependent on expert experience and lacks flexibility. In this paper, we propose a Dynamic Transformer to change the encoding procedure from sequential to adaptive, i.e., data-dependent computing paths. Specifically, we design three different types of feature extraction blocks, each containing a calculation unit to perform the corresponding operation and a routing gate to select the direction to pass the signal based on the input data. Three blocks are deployed in parallel at each layer to build a multi-layer routing space in a fully connected manner. In addition, we use a path regularization constraint model to learn similar paths for semantically similar images. We conduct extensive experiments on the COCO image captioning dataset and achieve new state-of-the-art performance, i.e., 133.2% CIDEr on Karpathy test split.

Image captioning is a challenging task, i.e., given an image machine automatically generates natural language that matches its semantic content and has attracted much attention in recent years. However, most existing models are designed manually, and their performance depends heavily on the expert experience of the designer. In addition, the computational flow of the model is predefined, and hard and easy samples will share the same coding path and easily interfere with each other, thus confusing the learning of the model. In this paper, we propose a Dynamic Transformer to change the encoding procedure from sequential to adaptive, i.e., data-dependent computing paths. Specifically, we design three different types of visual feature extraction blocks and deploy them in parallel at each layer to construct a multi-layer routing space in a fully connected manner. Each block contains a calculation unit that performs the corresponding operations and a routing gate that learns to adaptively select the direction to pass the signal based on the input image. Thus, our model can achieve a robust visual representation by exploring potential visual feature extraction paths. We evaluate our method quantitatively and qualitatively using a benchmark MSCOCO image caption dataset and perform extensive ablation studies to investigate the reasons behind its effectiveness. The experimental results show that our method is significantly superior to previous state-of-the-art methods.

Image captioning methods with attention mechanism are leading this field, especially models with global and local attention. But there are few conventional models to integrate the relationship information between various regions of the image. In this paper, this kind of relationship features are embedded into the fused attention mechanism to explore the internal visual and semantic relations between different object regions. Besides, to alleviate the exposure bias problem and make the training process more efficient, we combine Generative Adversarial Network with Reinforcement Learning and employ the greedy decoding method to generate a dynamic baseline reward for self-critical training. Finally, experiments on MSCOCO datasets show that the model can generate more accurate and vivid image captioning sentences and perform better in multiple prevailing metrics than the previous advanced models.