Contrastive vision-language pre-training models such as CLIP have achieved remarkable progress in vision-language understanding. However, a modality gap still exists in the shared embedding space, which refers to the gap between image and text feature clusters, ultimately limiting the performance of downstream tasks.Most existing studies attempt to alleviate this gap by directly pulling embeddings closer within the shared space. However, they often overlook the balance between alignment and distinctiveness, leading to degraded uniformity, which can easily result in representation collapse. Our approach focuses on addressing the modality gap from modality distinctiveness. We propose CCRA (Cross-modal Complementary Representation Alignment), a lightweight post-alignment framework that explicitly models the complementarity between modalities while maintaining semantic consistency, thereby effectively alleviating the modality gap.CCRA consists of two modules: (1) a Feature Complementary Network (FCN), which dynamically extracts and fuses complementary semantic cues and learnable query vectors; (2) a Distillation Enhancement Module (DEM), which transfers similarity distributions from a frozen CLIP teacher model to maintain training stability through a distillation constraint. CCRA achieves consistent improvements over five representative baselines across two cross-modal retrieval benchmarks (COCO2017-Val and Flickr8k/30k). On COCO2017-Val, it shows an improvement of 10.06% for I→T and 6.12% for T→I, reaching the best overall performance. Both quantitative and qualitative analyses further confirm that CCRA effectively narrows the modality gap and produces more uniform feature distributions on the hypersphere.

YifeiLu YifeiLu, Fanghua Ye, Jian Li, Qiang Gao, Cheng Liu, Haibo Luo, Nan Du, Xiaolong Li, Feiliang Ren. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Relation extraction (RE) aims to identify semantic relations between entities in unstructured text.Although recent work extends traditional RE to multimodal scenarios, most approaches still adopt classification-based paradigms with fused multimodal features, representing relations as discrete labels.This paradigm has two significant limitations: (1) it overlooks structural constraints like entity types and positional cues, and (2) it lacks semantic expressiveness for fine-grained relation understanding.We propose Retrieval Over Classification (ROC), a novel framework that reformulates multimodal RE as a retrieval task driven by relation semantics.ROC integrates entity type and positional information through a multimodal encoder, expands relation labels into natural language descriptions using a large language model, and aligns entity-relation pairs via semantic similarity-based contrastive learning.Experiments show that our method achieves state-of-the-art performance on the benchmark datasets MNRE and MORE and exhibits stronger robustness and interpretability.

Role-Playing Language Agents (RPLAs) have emerged as a significant application direction for Large Language Models (LLMs).Existing approaches typically rely on prompt engineering or supervised fine-tuning to enable models to imitate character behaviors in specific scenarios, but often neglect the underlying cognitive mechanisms driving these behaviors.Inspired by cognitive psychology, we introduce CogDual, a novel RPLA adopting a cognize-then-respond reasoning paradigm.By jointly modeling external situational awareness and internal self-awareness, CogDual generates responses with improved character consistency and contextual alignment.To further optimize the performance, we employ reinforcement learning with two general-purpose reward schemes designed for open-domain text generation.Extensive experiments on the CoSER benchmark, as well as Cross-MR and Life-Choice, demonstrate that CogDual consistently outperforms existing baselines and generalizes effectively across diverse role-playing tasks.Our code is available at chengliu01/CogDual.

Although existing machine reading comprehension models are making rapid progress on many datasets, they are far from robust. In this article, we propose an understanding-oriented machine reading comprehension model to address three kinds of robustness issues, which are over-sensitivity, over-stability, and generalization. Specifically, we first use a natural language inference module to help the model understand the accurate semantic meanings of input questions to address the issues of over-sensitivity and over-stability. Then, in the machine reading comprehension module, we propose a memory-guided multi-head attention method that can further well understand the semantic meanings of input questions and passages. Third, we propose a multi-language learning mechanism to address the issue of generalization. Finally, these modules are integrated with a multi-task learning-based method. We evaluate our model on three benchmark datasets that are designed to measure models’ robustness, including DuReader (robust) and two SQuAD-related datasets. Extensive experiments show that our model can well address the mentioned three kinds of robustness issues. And it achieves much better results than the compared state-of-the-art models on all these datasets under different evaluation metrics, even under some extreme and unfair evaluations. The source code of our work is available at https://github.com/neukg/RobustMRC .

Tagging based relational triple extraction methods are attracting growing research attention recently. However, most of these methods take a unidirectional extraction framework that first extracts all subjects and then extracts objects and relations simultaneously based on the subjects extracted. This framework has an obvious deficiency that it is too sensitive to the extraction results of subjects. To overcome this deficiency, we propose a bidirectional extraction framework based method that extracts triples based on the entity pairs extracted from two complementary directions. Concretely, we first extract all possible subject-object pairs from two paralleled directions. These two extraction directions are connected by a shared encoder component, thus the extraction features from one direction can flow to another direction and vice versa. By this way, the extractions of two directions can boost and complement each other. Next, we assign all possible relations for each entity pair by a biaffine model. During training, we observe that the share structure will lead to a convergence rate inconsistency issue which is harmful to performance. So we propose a share-aware learning mechanism to address it. We evaluate the proposed model on multiple benchmark datasets. Extensive experimental results show that the proposed model is very effective and it achieves state-of-the-art results on all of these datasets. Moreover, experiments show that both the proposed bidirectional extraction framework and the share-aware learning mechanism have good adaptability and can be used to improve the performance of other tagging based methods. The source code of our work is available at: https://github.com/neukg/BiRTE.

Most existing multi-document machine reading comprehension models mainly focus on understanding the interactions between the input question and documents, but ignore the following two kinds of understandings. First, to understand the semantic meaning of words in the input question and documents from the perspective of each other. Second, to understand the supporting cues for a correct answer from the perspective of intra-document and inter-documents. Ignoring these two kinds of important understandings would make the models overlook some important information that may be helpful for finding correct answers. To overcome this deficiency, we propose a deep understanding based model for multi-document machine reading comprehension. It has three cascaded deep understanding modules which are designed to understand the accurate semantic meaning of words, the interactions between the input question and documents, and the supporting cues for the correct answer. We evaluate our model on two large scale benchmark datasets, namely TriviaQA Web and DuReader. Extensive experiments show that our model achieves state-of-the-art results on both datasets.

We investigate response selection for multi-turn conversation in retrieval-based chatbots. Existing studies pay more attention to the matching between utterances and responses by calculating the matching score based on learned features, leading to insufficient model reasoning ability. In this paper, we propose a graph- reasoning network (GRN) to address the problem. GRN first conducts pre-training based on ALBERT using next utterance prediction and utterance order prediction tasks specifically devised for response selection. These two customized pre-training tasks can endow our model with the ability of capturing semantical and chronological dependency between utterances. We then fine-tune the model on an integrated network with sequence reasoning and graph reasoning structures. The sequence reasoning module conducts inference based on the highly summarized context vector of utterance-response pairs from the global perspective. The graph reasoning module conducts the reasoning on the utterance-level graph neural network from the local perspective. Experiments on two conversational reasoning datasets show that our model can dramatically outperform the strong baseline methods and can achieve performance which is close to human-level.

Tagging based methods are one of the mainstream methods in relational triple extraction. However, most of them suffer from the class imbalance issue greatly. Here we propose a novel tagging based model that addresses this issue from following two aspects. First, at the model level, we propose a three-step extraction framework that can reduce the total number of samples greatly, which implicitly decreases the severity of the mentioned issue. Second, at the intra-model level, we propose a confidence threshold based cross entropy loss that can directly neglect some samples in the major classes. We evaluate the proposed model on NYT and WebNLG. Extensive experiments show that it can address the mentioned issue effectively and achieves state-of-the-art results on both datasets. The source code of our model is available at: https://github.com/neukg/ConCasRTE.

Neural conversation models have shown great potentials towards generating fluent and informative responses by introducing external background knowledge.Nevertheless, it is laborious to construct such knowledge-grounded dialogues, and existing models usually perform poorly when transfer to new domains with limited training samples.Therefore, building a knowledge-grounded dialogue system under the low-resource setting is a still crucial issue.In this paper, we propose a novel threestage learning framework based on weakly supervised learning which benefits from large scale ungrounded dialogues and unstructured knowledge base.To better cooperate with this framework, we devise a variant of Transformer with decoupled decoder which facilitates the disentangled learning of response generation and knowledge incorporation.Evaluation results on two benchmarks indicate that our approach can outperform other state-of-the-art methods with less training data, and even in zero-resource scenario, our approach still performs well.

Table filling based relational triple extraction methods are attracting growing research interests due to their promising performance and their abilities on extracting triples from complex sentences.However, this kind of methods are far from their full potential because most of them only focus on using local features but ignore the global associations of relations and of token pairs, which increases the possibility of overlooking some important information during triple extraction.To overcome this deficiency, we propose a global feature-oriented triple extraction model that makes full use of the mentioned two kinds of global associations.Specifically, we first generate a table feature for each relation.Then two kinds of global associations are mined from the generated table features.Next, the mined global associations are integrated into the table feature of each relation.This "generate-mine-integrate" process is performed multiple times so that the table feature of each relation is refined step by step.Finally, each relation's table is filled based on its refined table feature, and all triples linked to this relation are extracted based on its filled table.We evaluate the proposed model on three benchmark datasets.Experimental results show our model is effective and it achieves state-of-the-art results on all of these datasets.The source code of our work is available at: https://github.com/neukg/GRTE.

Knowledge graph embedding is an important task and it will benefit lots of downstream applications.Currently, deep neural networks based methods achieve state-of-the-art performance.However, most of these existing methods are very complex and need much time for training and inference.To address this issue, we propose a simple but effective atrous convolution based knowledge graph embedding method.Compared with existing state-of-the-art methods, our method has following main characteristics.First, it effectively increases feature interactions by using atrous convolutions.Second, to address the original information forgotten issue and vanishing/exploding gradient issue, it uses the residual learning method.Third, it has simpler structure but much higher parameter efficiency.We evaluate our method on six benchmark datasets with different evaluation metrics.Extensive experiments show that our model is very effective.On these diverse datasets, it achieves better results than the compared state-of-theart methods on most of evaluation metrics.The source codes of our model could be found at https://github.

Relation classification is an important task in natural language processing fields. State-of-the-art methods usually concentrate on building deep neural networks based classification models on the training data in which the relations of the labeled entity pairs are given. However, these methods usually suffer from the data sparsity issue greatly. On the other hand, we notice that it is very easily to obtain some concise text descriptions for almost all of the entities in a relation classification task. The text descriptions can provide helpful supplementary information for relation classification. But they are ignored by most of existing methods. In this paper, we propose DesRC, a new neural relation classification method which integrates entities’ text descriptions into deep neural networks models. We design a two-level attention mechanism to select the most useful information from the “intra-sentence” aspect and the “cross-sentence” aspect. Besides, the adversarial training method is also used to further improve the classification per-formance. Finally, we evaluate the proposed method on the SemEval 2010 dataset. Extensive experiments show that our method achieves much better experimental results than other state-of-the-art relation classification methods.