Large-scale pre-trained Vision-Language Models (VLMs) have become essential for transfer learning across diverse tasks. However, adapting these models with limited few-shot data often leads to overfitting, diminishing their performance on new tasks. To tackle this issue, we propose a novel Multi-Modal Representation Learning (MMRL) framework that introduces a shared, learnable, and modality-agnostic representation space. MMRL projects the space tokens to text and image representation tokens, facilitating more effective multi-modal interactions. Unlike previous approaches that solely optimize class token features, MMRL integrates representation tokens at higher layers of the encoders—where dataset-specific features are more prominent—while preserving generalized knowledge in the lower layers. During training, both representation and class features are optimized, with trainable projection layer applied to the representation tokens, whereas the class token projection layer remains frozen to retain pre-trained knowledge. Furthermore, a regularization term is introduced to align the class features and text features with the zero-shot features from the frozen VLM, thereby safeguarding the model’s generalization capacity. For inference, a decoupling strategy is employed, wherein both representation and class features are utilized for base classes, while only the class features, which retain more generalized knowledge, are used for new tasks. Extensive experiments across 15 datasets demonstrate that MMRL outperforms state-of-the-art methods, achieving a balanced trade-off between task-specific adaptation and generalization. Code is available at https://github.com/yunncheng/MMRL.

In this paper, we address the task of change captioning, which aims to generate sentences describing fine-grained differences between similar images while resisting distractors like viewpoint variations. To overcome the challenges of learning robust difference representations and avoiding over-reliance on partial features, we propose two key innovations. First, the Hybrid Visual Adapter (HVA) enhances the Vision Transformer (ViT) backbone by explicitly modeling cross-image interactions through attention mechanisms. By attending to counterpart regions during feature extraction, HVA suppresses irrelevant distractors and learns viewpoint-invariant representations, significantly improving robustness against spatial misalignment. Second, we introduce drop-view training, a strategy that randomly masks one of the three input features (before, after, or difference features) during caption generation. This forces the decoder to fully explore complementary visual cues. Extensive experimental results show that our method achieves superior performance on three public datasets with different change types.

In recent years, the research of employing knowledge graphs (KGs) in sequential recommendation (SR) has received a lot of attention, since the side information extracted from KGs, especially the information of the correlations between items, indeed helps the SR models achieve better performance. However, many previous KG-based SR models tend to introduce some noise information when learning item embeddings, or insufficiently fuse item–item correlations into their sequential modeling, thus limiting their performance improvements . In this paper, we propose a D istance- A ware K nowledge-based S equential R ecommendation model ( DAKSR ), which exploits the explicit item–item correlations from KGs to achieve enhanced SR. Specifically, as one critical component in our DAKSR, the distance score matrix (DSM) is first obtained to indicate the correlations between items, and then leveraged in the following three major modules of DAKSR. First, in the Item-Set Embedding layer (ISE) all item embeddings are learned based on DSM, in which the noise information is eliminated effectively. Meanwhile, the Knowledge-Infused Transformer (KIT) incorporates DSM into its attention mechanism to improve the feature extraction. Furthermore, the Knowledge Contrastive Learning module (KCL) also leverages the item–item correlations presented in DSM to generate two credible sequence views, which are used to refine sample representations through a contrastive learning strategy, and thus improve the model’s robustness. Our extensive experiments on three SR benchmarks obviously demonstrate our DAKSR’s superior performance over the state-of-the-art (SOTA) KG-based recommendation models. The implementation of our DAKSR is available at https://github.com/Easonsi/DAKSR for reproducing our experiment results conveniently.

Visual object tracking is a challenging task that aims to accurately estimate the scale and position of a designated target. Recently, segmentation networks have proven effective in visual tracking, producing outstanding results for target scale estimation. However, segmentation-based trackers still lack robustness due to the presence of similar distractors. To mitigate this issue, we propose an Attention-based Gating Network (AGNet) that produces gating weights to diminish the impact of feature maps linked to similar distractors. Subsequently, we incorporate the AGNet into the segmentation-based tracking paradigm to achieve accurate and robust tracking. Specifically, the AGNet utilizes three cascading Multi-Head Cross-Attention (MHCA) modules to generate gating weights that govern the generation of feature maps in the baseline tracker. The proficiency of the MHCA in modeling global semantic information effectively suppresses feature maps associated with similar distractors. Additionally, we introduce a distractor-aware training strategy that leverages distractor masks to train our model. To alleviate the issue of partial occlusion, we introduce a box refinement module to enhance the accuracy of the predicted target box. Comprehensive experiments conducted on 11 challenging tracking benchmarks show that our approach significantly surpasses the baseline tracker across all metrics and achieves excellent results on multiple tracking benchmarks.

Cross-modal retrieval aims at retrieving highly semantic relevant information among multi-modalities. Existing cross-modal retrieval methods mainly explore the semantic consistency between image and text while rarely consider the rankings of positive instances in the retrieval results. Moreover, these methods seldom take into account the cross-interaction between image and text, which leads to the deficiency of learning their semantic relations. In this paper, we propose a Unified framework with Ranking Learning (URL) for cross-modal retrieval. The unified framework consists of three sub-networks, visual network, textual network, and interaction network. Visual network and textual network project the image feature and text feature into their corresponding hidden spaces respectively. Then, the interaction network forces the target image-text representation to align in the common space. For unifying both semantics and rankings, we propose a new optimization paradigm including pre-alignment for semantic knowledge transfer and ranking learning for final retrieval, which can decouple semantic alignment and ranking learning. The former focuses on the semantic pre-alignment optimized by semantic classification and the latter revolves around the retrieval rankings. For the ranking learning, we introduce a cross-AP loss which can directly optimize the retrieval metric average precision for cross-modal retrieval. We conduct experiments on four widely-used benchmarks, including Wikipedia dataset, Pascal Sentence dataset, NUS-WIDE-10k dataset, and PKU XMediaNet dataset respectively. Extensive experimental results show that the proposed method can obtain higher retrieval precision.

Visual tracking plays a pivotal role in intelligent transportation systems and has a wide range of practical applications such as autonomous driving and traffic counting. Recently, the attention mechanism in Transformers has been successfully applied to the field of visual tracking, leading to a significant improvement in tracking performance. However, Transformer-based trackers directly flatten two-dimensional image features into one-dimensional vectors to compute attention scores. This process unavoidably results in the omission of crucial position distribution information necessary for precise target localization. To address this issue, we propose a novel cross-attention based tracking-by-segmentation framework, called Dynamical Position Embedding based Tracking framework (DPET). DPET incorporates an additional network for modeling position information to complement the cross-attention module. To be specific, a dynamical position embedding network is introduced to adaptively encode position information. This network is then integrated into the cross-attention based feature fusion network to compensate for the loss of position distribution information. As a result, the fused feature incorporates abundant contextual semantic cues for target classification and precise position information for target localization simultaneously. To overcome the constraints imposed by bounding-boxes, a segmentation network that takes the fused feature as input is designed to achieve accurate pixel-wise tracking. Extensive experiments on eight challenging tracking benchmarks show that our DPET tracker enables real-time operations and achieves promising tracking performance on the GOT-10K benchmark. Especially, DPET tracker achieves the top accuracy scores on VOT2016, VOT2018 and VOT2019 benchmarks.

The effective aggregation of spatiotemporal information to accommodate real-world complex scenes is a fundamental issue in video saliency detection. In this paper, we propose a Triplet Spatiotemporal Aggregation Network (TSAN) to address it from the aggregation of spatiotemporal interaction, spatiotemporal information distribution, and multi-level spatiotemporal features. Firstly, we propose an interactive aggregation gate (IAG) module to model spatial and temporal global context information and perform inter-modal information transfer. Secondly, we employ an information distribution consistency (IDC) module to enhance the consistency of spatiotemporal representation by maximizing the correlation of spatiotemporal high-level features. Finally, we design a multi-level spatiotemporal feature aggregation (MSF) framework to merge cross-level and cross-modal features. These three modules are combined into a unified framework to jointly optimize spatiotemporal information for more precise results. Experimental results on five prevailing datasets show that TSAN outperforms previous competitors.

Segmentation-based tracking paradigm has been successfully applied to tracking field and significantly improves the tracking performance. Although segmentation-based trackers are effective for target scale estimation, it makes the trackers have high requirements for the extracted target features due to the need for pixel-level segmentation. Therefore, in this article, we propose a novel multi feature representation and aggregation network and introduce it into tracking-by-segmentation framework to extract and integrate rich features for segmentation-based tracking. To be specific, the proposed approach firstly models three complementary feature representations through cross-attention, cross-correlation and dilated involution mechanisms respectively and employ a simple feature aggregation network to fuse these features. And then feeding those fusion features into a segmentation network obtains the accurate target state estimation. In addition, we introduce a bounding box refinement module to further refine the target box to alleviate the issues of partial occlusion and surrounding distractors. The extensive experimental results show that the proposed tracker achieves very promising tracking performance on seven challenging visual tracking benchmarks. Code and models are available at https://github.com/Yang428/FEAST.

Correlation operation and attention mechanism are two popular feature fusion approaches which play an important role in visual object tracking. However, the correlation-based tracking networks are sensitive to location information but loss some context semantics, while the attention-based tracking networks can make full use of rich semantic information but ignore the position distribution of the tracked object. Therefore, in this paper, we propose a novel tracking framework based on joint correlation and attention networks, termed as JCAT, which can effectively combine the advantages of these two complementary feature fusion approaches. Concretely, the proposed JCAT approach adopts parallel correlation and attention branches to generate position and semantic features. Then the fusion features are obtained by directly adding the location feature and semantic feature. Finally, the fused features are fed into the segmentation network to generate the pixel-wise state estimation of the object. Furthermore, we develop a segmentation memory bank and an online sample filtering mechanism for robust segmentation and tracking. The extensive experimental results on eight challenging visual tracking benchmarks show that the proposed JCAT tracker achieves very promising tracking performance and sets a new state-of-the-art on the VOT2018 benchmark.

Text-to-image synthesis is an attractive but challenging task that aims to generate a photo-realistic and semantic consistent image from a specific text description. The images synthesized by off-the-shelf models usually contain limited components compared with the corresponding image and text description, which decreases the image quality and the textual-visual consistency. To address this issue, we propose a novel Vision-Language Matching strategy for text-to-image synthesis, named VLMGAN*, which introduces a dual vision-language matching mechanism to strengthen the image quality and semantic consistency. The dual vision-language matching mechanism considers textual-visual matching between the generated image and the corresponding text description, and visual-visual consistent constraints between the synthesized image and the real image. Given a specific text description, VLMGAN* firstly encodes it into textual features and then feeds them to a dual vision-language matching-based generative model to synthesize a photo-realistic and textual semantic consistent image. Besides, the popular evaluation metrics for text-to-image synthesis are borrowed from simple image generation, which mainly evaluate the reality and diversity of the synthesized images. Therefore, we introduce a metric named Vision-Language Matching Score (VLMS) to evaluate the performance of text-to-image synthesis which can consider both the image quality and the semantic consistency between the synthesized image and the description. The proposed dual multi-level vision-language matching strategy can be applied to other text-to-image synthesis methods. We implement this strategy on two popular baselines, which are marked with${\text{VLMGAN}_{+\text{AttnGAN}}}$and${\text{VLMGAN}_{+\text{DFGAN}}}$. The experimental results on two widely-used datasets show that the model achieves significant improvements over other state-of-the-art methods.

Attention based methods for image-text generation often focus on visual features individually, while ignoring relationship information among image features that provides important guidance for generating sentences. To alleviate this issue, in this work we propose the Joint Relationship Attention Network (JRAN) that novelly explores the relationships among the features. Specifically, different from the previous relationship based approaches that only explore the single relationship in the image, our JRAN can effectively learn two relationships, the visual relationships among region features and the visual-semantic relationships between region features and semantic features, and further make a dynamic trade-off between them during outputting the relationship representation. Moreover, we devise a new relationship based attention, which can adaptively focus on the output relationship representation when predicting different words. Extensive experiments on large-scale MSCOCO and small-scale Flickr30k datasets show that JRAN achieves state-of-the-art performance. More remarkably, JRAN achieves new 28.3% and 58.2% performance in terms of BLEU4 and CIDEr metric on Flickr30k dataset.

Visual Dialog aims to answer multi-round, interactive questions based on the dialog history and image content. Existing methods either consider answer ranking and generating individually or only weakly capture the relation across the two tasks implicitly by two separate models. The research on a universal framework that jointly learns to rank and generate answers in a single model is seldom explored. In this paper, we propose a contrastive learning-based framework UTC to unify and facilitate both discriminative and generative tasks in visual dialog with a single model. Specifically, considering the inherent limitation of the previous learning paradigm, we devise two inter-task contrastive losses i.e., context contrastive loss and answer contrastive loss to make the discriminative and generative tasks mutually reinforce each other. These two com-plementary contrastive losses exploit dialog context and target answer as anchor points to provide representation learning signals from different perspectives. We evaluate our proposed UTC on the VisDial v1.0 dataset, where our method outperforms the state-of-the-art on both discriminative and generative tasks and surpasses previous state-of-the-art generative methods by more than 2 absolute points on Recall@1.

Co-saliency detection (CoSOD) aims at discovering the repetitive salient objects from multiple images. Two primary challenges are group semantics extraction and noise object suppression. In this paper, we present a unified Two-stage grOup semantics PropagatIon and Contrastive learning NETwork (TopicNet) for CoSOD. TopicNet can be decomposed into two substructures, including a two-stage group semantics propagation module (TGSP) to address the first challenge and a contrastive learning module (CLM) to address the second challenge. Concretely, for TGSP, we design an image-to-group propagation module (IGP) to capture the consensus representation of intra-group similar features and a group-to-pixel propagation module (GPP) to build the relevancy of consensus representation. For CLM, with the design of positive samples, the semantic consistency is enhanced. With the design of negative samples, the noise objects are suppressed. Experimental results on three prevailing benchmarks reveal that TopicNet outperforms other competitors in terms of various evaluation metrics.

In recent years, as an active topic in the field of vision and language, image captioning has made great progress. Previous approaches have demonstrated the superiority of spatial and channel attentions in image captioning task. However, such attention-based approaches ignore the difference between function words (e.g., “to”, “for” and “out”) and notional words (e.g., “girl”, “teddy” and “bear”). To address above issue, in this paper we propose a dynamical balancing attention model (BAM) based on attention variation for image captioning, which uses attention variation to fuse channel attention and region attention. Generating function and notional words, it effectively balances the contribution of image channel feature and that of image region feature. Further, the proposed approach dynamically focuses on the most relevant attention features in word prediction. Extensive experimental results on typical datasets show our approach outperforms the attention based approaches and achieves competitive performance over existing end-to-end leading approaches.

Image-Text Matching is one major task in cross-modal information processing. The main challenge is to learn the unified visual and textual representations. Previous methods that perform well on this task primarily focus on not only the alignment between region features in images and the corresponding words in sentences, but also the alignment between relations of regions and relational words. However, the lack of joint learning of regional features and global features will cause the regional features to lose contact with the global context, leading to the mismatch with those non-object words which have global meanings in some sentences. In this work, in order to alleviate this issue, it is necessary to enhance the relations between regions and the relations between regional and global concepts to obtain a more accurate visual representation so as to be better correlated to the corresponding text. Thus, a novel multi-level semantic relations enhancement approach namedDual Semantic Relations Attention Network(DSRAN)is proposed which mainly consists of two modules, separate semantic relations module and the joint semantic relations module. DSRAN performs graph attention in both modules respectively for region-level relations enhancement and regional-global relations enhancement at the same time. With these two modules, different hierarchies of semantic relations are learned simultaneously, thus promoting the image-text matching process by providing more information for the final visual representation. Quantitative experimental results have been performed on MS-COCO and Flickr30K and our method outperforms previous approaches by a large margin due to the effectiveness of the dual semantic relations learning scheme.

Recent great progress has been made on Salient Object Detection (SOD) by deep Convolutional Neural Networks (CNNs). However, most SOD methods still suffer from scale imbalance issue, which pays more attention on large salient areas but ignores small salient areas though they belong to the same object. To address this issue, this paper proposes a Scale Balance Network (SBN) to accurately locate large salient areas and recognize small salient areas. Firstly, a backbone network specifically designed for object detection is adopted in this paper, which captures larger resolution with more spatial features in deeper layers. Secondly, to focus on the balance between the large salient areas and the small salient areas, this paper proposes a novel Connective Feature Pyramid Module (CFPM) for sufficiently leveraging the multi-scale features and the multi-level features, which includes Feature Coherence Enhancement (FCE) and Feature Progressive Extraction (FPE). FCE is designed to enhance the coherence between high-level and low-level features, and FPE is designed to extract the progressive features in different convolutional layers. Finally, an Edge Enhancement Architecture with Various Kernels (EEAVK) is proposed to refine the edge features. Experimental results on five benchmark datasets show that the proposed method outperforms or achieves consistently superior performance in comparison with other methods under different evaluation metrics.

Image-Text Matching is one major task in cross-modal information processing. The main challenge is to learn the unified vision and language representations. Previous methods that perform well on this task primarily focus on the region features in images corresponding to the words in sentences. However, this will cause the regional features to lose contact with the global context, leading to the mismatch with those non-object words in some sentences. In this work, in order to alleviate this problem, a novel Dual Semantic Relationship Attention Network is proposed which mainly consists of two modules, separate semantic relationship module and the joint semantic relationship module. With these two modules, different hierarchies of semantic relationships are learned simultaneously, thus promoting the image-text matching process. Quantitative experiments have been performed on MS-COCO and Flickr-30K and our method outperforms previous approaches by a large margin due to the effectiveness of the dual semantic relationship attention scheme.

Pose variation has been one of the challenges of face recognition. To solve this challenge, the authors propose a classification algorithm using supervised subspace learning and non‐local representation (SSLNR). In SSLNR, they first propose a supervised subspace learning algorithm (SSLA). SSLA includes three different terms. The first term is the difference term, which can reduce the intra‐class differences. The second term is the block‐diagonal regularisation term, which promotes the samples to be represented by intra‐class samples. The last one is the noise robust term. Then, the original samples are mapped to the learned subspace by using SSLA. Thus, the intra‐class differences of the samples mapped to the learned subspace are reduced. Finally, those mapped samples are classified by proposed non‐local constraint‐based extended sparse representation classifier. SSLNR is extensively evaluated using four databases, namely Georgia Tech, Label faces in the wild, FEI and CVL. Experimental results show that SSLNR achieves better performance than some state‐of‐the‐art algorithms, such as DARG and RRNN.

Generally, the commonly used sparse‐based methods, such as sparse representation classifier, have achieved a good recognition result in face recognition. However, there exist several problems in those methods. First, those methods think that the importance of each atom is the same in representing other query samples. This is not reasonable because different atoms contain different amounts of information, their importance should be different when they together represent the query samples. Second, those methods cannot meet the real‐time requirement when dealing with large data set. In this study, on the one hand, the authors propose a fast extended sparse‐weighted representation classifier (FESWRC) by considering the different importance of atoms and using primal augmented Lagrangian method as well as principal component analysis. On the other hand, the authors propose a distinctive feature descriptor, named logarithmic‐weighted sum (LWS) feature descriptor. The authors combine FESWRC and LWS and used for face recognition, this method is called face recognition algorithm based on feature descriptor and weighted linear sparse representation (FDWLSR). Experimental results show that FDWLSR can realise real‐time recognition and the recognition rate can achieve 100.0, 100.0, 91.6, 93.4 and 87.4%, respectively, on the Yale, Olivetti Research Laboratory (ORL), faculdade de engenharia industrial (FEI), face recognition technology program (FERET) and labelled face in the wild datasets.

Here, the authors propose a hybrid classification approach using extreme learning machine (ELM) and sparse representation classifier (SRC) with adaptive threshold, which they called ATELMSRC. ATELMSRC can adaptively adjust the threshold, and make more test images correctly classified by ELM compared with ELMSRC, which not only reduces the classification time greatly but also improves the classification accuracy. In addition, primal augmented Lagrangian method is used in ATELMSRC to speed up the solution of ‐norm, which also speeds up the classification process. Experimental results on USPS handwritten digits data set and UMIST face data set show that the total classification time of the authors ATELMSRC is very short for large data sets, only 1/310 of SRC, 1/805 of extended SRC (ESRC), and 1/41 of ELMSRC. Meanwhile, the classification accuracy of the authors’ ATELMSRC is 97.80% on USPS handwritten digits data set, and 99.27% on UMIST face data set, which are higher than those of ELM, SRC, ESRC, ELMSRC etc.

This paper studies Aspect-based Opinion Summarization (AOS) of reviews on particular products. In practice, an AOS system needs to address two core subtasks, aspect extraction and sentiment classification. Most existing approaches to aspect extraction, using linguistic analysis or topic modeling, are general across different products but not precise enough or suitable for particular products. Instead we take a less general but more precise scheme, which directly maps each review sentence into pre-defined aspects. To tackle aspect mapping and sentiment classification, we propose two Convolutional Neural Network (CNN) based methods, cascaded CNN and multitask CNN. Cascaded CNN contains two levels of convolutional networks. Multiple CNNs at level 1 deal with aspect mapping task, and a single CNN at level 2 deals with sentiment classification. Multitask CNN also contains multiple aspect CNNs and a sentiment CNN, but different networks share the same word embeddings. Experimental results show that both cascaded and multitask CNNs with pre-trained word embedding outperform linear classifiers, and multitask CNN generally performs better than cascaded CNN.

Topological properties have important function in human beings visual attention. TPQFT (Topological properties based Phase spectrum of Quaternion Fourier Transform) model is an attention selection model using topological properties expression we have introduced. A new quantification criterion to evaluate every channel's contribution and model's performance is proposed in this paper and used in TPQFT. This paper improves TPQFT model in several aspects and WTPQFT (Weight Adjusted Topological properties based Phase spectrum of Quaternion Fourier Transform) model is introduced. The experimental results show that WTPQFT model reflects the real attention selection more accurately than TPQFT and PQFT (Phase spectrum of Quaternion Fourier Transform) method.

Measurement of visual quality is of fundamental importance to some image processing applications. And the perceived image distortion of any image strongly depends on the local features, such as edges, flats and textures. Since edges often convey much information of an image, we propose a novel algorithm for image quality assessment based on the edge and contrast similarity between the distorted image and the reference(perfect) image. We demonstrate its promise through a set of intuitive examples, as well as validate its performance with subjective ratings. We also compare our method with two other state-of-the-art objective ones, which uses 550 images with different distortion types and BP neural network.

In this paper, we construct a novel HNN energy function, and use the HNN with this energy function to solve anti-aircraft missile deployment, which is a constrained layout problem. A near-optimum solution is obtained when HNN reaches a stable state, i.e, the minimum of the energy function is reached. We have studied the convergence of the network and the relationship between parameters of the network and stability. A group of suitable parameters are obtained by lots of experiences. The simulation results in different scales show that our approach can obtain the near-optimum solution. In addition, our approach can get better solutions in larger scales than other methods such as the divide-and-conquer algorithm which is often used to solve constrained layout problems, and it can also be extended to other constrained layout problems, such as mobile base station planning and integrated circuit layout design.

PCNN (pulse coupled neural network), a phenomenological model exhibiting synchronous pulse bursts, can be used in image processing efficiently by the specified algorithm corresponding to the specified application, but so far there has been no general design approach. This paper describes that the parallel pulse-spreading behavior of unit-linking PCNN for image processing, based on PCNN, is equal to the operation of mathematic morphology. Hereby we propose the general unit-linking PCNN design approach for binary image processing. In the meantime, in order to explain how to apply this new general design approach in the specified application, binary image denoising, binary image edge detection, binary image hole-filter based on unit-linking PCNN are analyzed respectively as examples.

This paper introduces an approach for image shadow removal by using pulse coupled neural network (PCNN), based on the phenomena of synchronous pulse bursts in the animal visual cortexes. Two shadow-removing criteria are proposed. These two criteria decide how to choose the optimal parameter (the linking strength beta). The computer simulation results of shadow removal based on PCNN show that if these two criteria are satisfied, shadows are removed completely and the shadow-removed images are almost as the same as the original nonshadowed images. The shadow removal results are independent of changes of intensities of shadows in some range and variations of the places of shadows. When the first criterion is satisfied, even if the second criterion is not satisfied, as to natural grey images that have abundant grey levels, shadows also can be removed and PCNN shadow-removed images retain the shapes of the objects in original images. These two criteria also can be used for color images by dividing a color image into three channels (R, G, B). For shadows varying drastically, such as the noisy points in images, these two criteria are still right, but difficult to satisfy. Therefore, this approach can efficiently remove shadows that do not include the random noise.