Compositional reasoning is a critical capability for multimodal models, enabling systematic understanding of complex scenes through structured combinations of objects, attributes, and relations. However, existing research on this ability primarily focuses on vision-language models (VLMs, e.g., CLIP and SigLIP), with limited exploration of multimodal large language models (MLLMs). To address this gap, we introduce CR³, a novel framework that enhances compositional reasoning abilities of MLLMs via rule-based reinforcement learning. CR³ leverages rule-based rewards to optimize the MLLM's policy on systematically curated multimodal instruction-following tasks, guided by a model-adaptive dynamic task mixing strategy. Our approach boosts performance by over 19% on three compositional reasoning benchmarks, significantly outperforming supervised fine-tuning (SFT) by at least 12%. Crucially, CR³ demonstrates superior generalization by improving performance on out-of-domain benchmarks where SFT methods degrade, highlighting its effectiveness and data efficiency.

Distinguishing spatial relations is a basic part of human cognition which requires fine-grained perception on cross-instance. Although benchmarks like MME, MMBench and SEED comprehensively have evaluated various capabilities which already include visual spatial reasoning(VSR). There is still a lack of sufficient quantity and quality evaluation and optimization datasets for Vision Large Language Models(VLLMs) specifically targeting visual positional reasoning. To handle this, we first diagnosed current VLLMs with the VSR dataset and proposed a unified test set. We found current VLLMs to exhibit a contradiction of over-sensitivity to language instructions and under-sensitivity to visual positional information. By expanding the original benchmark from two aspects of tunning data and model structure, we mitigated this phenomenon. To our knowledge, we expanded spatially positioned image data controllably using diffusion models for the first time and integrated original visual encoding(CLIP) with other 3 powerful visual encoders(SigLIP, SAM and DINO). After conducting combination experiments on scaling data and models, we obtained a VLLM VSR Expert(VSRE) that not only generalizes better to different instructions but also accurately distinguishes differences in visual positional information. VSRE achieved over a 27% increase in accuracy on the VSR test set. It becomes a performant VLLM on the position reasoning of both the VSR dataset and relevant subsets of other evaluation benchmarks. We hope it will accelerate advancements in VLLM on VSR learning.

Multimodal Emotion Recognition in Conversations (MERC) identifies utterance emotions by integrating both contextual and multimodal information from dialogue videos. Existing methods struggle to capture emotion shifts due to label replication and fail to preserve positive independent modality contributions during fusion. To address these issues, we propose a Dual Contrastive Learning Framework (DCLF) that enhances current MERC models without additional data. Specifically, to mitigate label replication effects, we construct context-aware contrastive pairs. Additionally, we assign pseudo-labels to distinguish modality-specific contributions. DCLF works alongside basic models to introduce semantic constraints at the utterance, context, and modality levels. Our experiments on two MERC benchmark datasets demonstrate performance gains of 4.67%-4.98% on IEMOCAP and 5.52%-5.89% on MELD, outperforming state-of-the-art approaches. Perturbation tests further validate DCLF’s ability to reduce label dependence. Additionally, DCLF incorporates emotion-sensitive independent modality features and multimodal fusion representations into final decisions, unlocking the potential contributions of individual modalities.

With the widespread use of Internet services, the risk of cyber attacks has increased significantly. Existing anomaly-based network intrusion detection systems suffer from slow processing speeds and low detection accuracy due to limitations in datasets and algorithms. Consequently, there is a pressing need to develop high-performance, flexible network intrusion detection systems. In response to these limitations, this paper designs and implements a network intrusion detection system based on P4-INT. This system uses the programmable packet processing language P4 to execute in-band network telemetry, enabling the real-time collection of network data. Additionally, a hybrid network model combining CNN, BiLSTM, and Multi-Head Attention mechanisms is developed to enhance detection capabilities. The proposed solution integrates telemetry technology with deep learning algorithms, effectively addressing the shortcomings of existing systems by improving detection accuracy. Compared to conventional network intrusion detection systems, this system can not only identify new attack patterns but also provide more precise detection results. It holds significant practical value and offers broad application prospects.

Fengyun-4 (FY-4) is a series of Chinese operational geostationary meteorological satellites, providing crucial data for weather forecasting, climate prediction, and environmental monitoring. Advanced Geostationary Radiation Imagers (AGRI) onboard the FY-4A and FY-4B satellites play a key role in observing the Earth’s surface, oceans, and atmosphere. However, their calibration stability is still uncertain, which clearly limits corresponding downstream applications. This study evaluates the long-term radiometric stability of AGRI reflective solar bands (RSBs) using a general cloud target (CT) calibration method, covering the periods from March 2018 to December 2024 for FY-4A/AGRI and from June 2022 to December 2024 for FY-4B/AGRI. By utilizing MODIS cloud products as references for cloud properties, we simulate the top-of-atmosphere (TOA) reflectances of CTs through the Discrete Ordinates Radiative Transfer (DISORT) model and compare results with observed reflectances to infer the instrumental calibration stability. Our results indicate that the radiometric responses of AGRI exhibit significant degradation in visible bands, while showing relatively smaller degradation rates in near- and shortwave-infrared bands. Specifically, the annual degradation rates for band 1 (0.47 μm) of FY-4A/AGRI and FY-4B/AGRI are 4.3% and 8.8% respectively. Both AGRIs demonstrate comparable degradation rates of approximately 3.5% for band 2 (0.65 μm). In contrast, bands 3 (0.83 μm), 5 (1.61 μm), and 6 (2.25 μm) show annual degradation rates around −1.0%, despite they exhibit notable fluctuations. The operational calibration of FY-4B/AGRI is more accurate than that of FY-4A/AGRI and with smaller fluctuations. By fitting the time series of relative errors between simulated and current calibrated reflectances, we calculate daily recalibration coefficients and effectively recalibrated the long-term data, with a calibration accuracy within ±3%. This study demonstrates that the CT-based calibration method can successfully track the radiometric stability of AGRI and provide a robust calibration solution to ensure data stability and accuracy.

Existing cold-start recommendation methods typically use item-level alignment strategies to align the content feature and collaborative feature of warm items during model training. However, these methods are less effective for cold items with low semantic similarity to the warm items when they first appear in the test stage, as they have no historical interactions to obtain the collaborative feature. In this paper, we propose a preference aware recommendation (PARec) model with hierarchical item alignment to solve the item cold-start issue. Our approach exploits user preference from historical records to achieve group-level alignment with item content feature, enhancing recommendation performance. Specifically, our hierarchical item alignment strategy improves recommendations for both high and low similarity cold items by using item-level alignment for high similarity cold items and introducing group-level alignment for low similarity cold items. Low similarity cold items can be successfully recommended through relationships among items, captured by our group-level alignment, based on their co-occurrence possibilities and semantic similarities. For model training, a hierarchical contrastive objective function is presented to balance the performance of warm and cold items, achieving better overall performance. Extensive experiments demonstrate the effectiveness of our method, with results showing its superiority compared to state-of-the-art approaches.

Existing cold-start recommendation methods often adopt item-level alignment strategies to align the content feature and the collaborative feature of warm items for model training, however, cold items in the test stage have no historical interactions with users to obtain the collaborative feature. These existing models ignore the aforementioned condition of cold items in the training stage, resulting in the performance limitation. In this paper, we propose a preference aware dual contrastive learning based recommendation model (PAD-CLRec), where the user preference is explored to take into account the condition of cold items for feature alignment. Here, the user preference is obtained by aggregating a group of collaborative feature of the warm items in the user's purchase records. Then, a group-level alignment between the user preference and the item's content feature can be realized via a proposed preference aware contrastive function for enhancing cold-item recommendation. In addition, a joint objective function is introduced to achieve a better trade-off between the recommendation performance of warm items and cold items from both item-level and group-level perspectives, yielding better overall recommendation performance. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method, and the results show the superiority of our method, as compared with the state-of-the-arts.

The Multi-Party Conversation (MPC) system has gained attention for its relevance in modern communication. Recent work has focused on developing specialized models for different MPC subtasks, improving state-of-the-art (SOTA) performance. However, since MPC demands often arise collaboratively, managing multiple specialized models is impractical. Additionally, dialogue evolves through diverse meta-information, where knowledge from specific subtasks can influence others. To address this, we propose UniMPC, a unified framework that consolidates common MPC subtasks. UniMPC uses a graph network with utterance nodes, a global node for combined local and global information, and two adaptable free nodes. It also incorporates discourse parsing to enhance model updates. We introduce MPCEval, a new benchmark for evaluating MPC systems. Experiments show UniMPC achieves over 95% of SOTA performance across all subtasks, with some surpassing existing SOTA, highlighting the effectiveness of the global node, free nodes, and dynamic discourse-aware graphs.

The fusion of language models (LMs) and knowledge graphs (KGs) is widely used in commonsense question answering, but generating faithful explanations remains challenging.Current methods often overlook path decoding faithfulness, leading to divergence between graph encoder outputs and model predictions.We identify confounding effects and LM-KG misalignment as key factors causing spurious explanations.To address this, we introduce the LM-KG Fidelity metric to assess KG representation reliability and propose the LM-KG Distribution-aware Alignment (LKDA) algorithm to improve explanation faithfulness.Without ground truth, we evaluate KG explanations using the proposed Fidelity-Sparsity Trade-off Curve.Experiments on Common-senseQA and OpenBookQA show that LKDA significantly enhances explanation fidelity and model performance, highlighting the need to address distributional misalignment for reliable commonsense reasoning.

This paper proposes a new multi-modal question-answering task, named as Cross-Modal Information Complementation based Question Answering (CroMIC-QA), to promote the exploration on bridging the semantic gap between visual and linguistic signals. The proposed task is inspired by the common phenomenon that, in most user-generated QA scenarios, the information of the given textual question is incomplete, and thus it is required to merge the semantics of both the text and the accompanying image to infer the complete real question. In this work, the CroMIC-QA task is first formally defined and compared with the classic Visual Question Answering (VQA) task. On this basis, a specified dataset, CroMIC-QA-Agri, is collected from an online QA community in the agriculture domain for the proposed task. A group of experiments is conducted on this dataset, with the typical multi-modal deep architectures implemented and compared. The experimental results show that the appropriate text/image presentations and text-image semantic interaction methods are effective to improve the performance of the framework.

In recent years, neural networks have achieved impressive performance on dialogue response generation. However, most of these models still suffer from some shortcomings, such as yielding uninformative responses and lacking explainable ability. This article proposes a Two-stage Dialogue Response Generation model (TSRG), which specifies a method to generate diverse and informative responses based on an interpretable procedure between stages. TSRG involves a two-stage framework that generates a candidate response first and then instantiates it as the final response. The positional information and a resident token are injected into the candidate response to stabilize the multi-stage framework, alleviating the shortcomings in the multi-stage framework. Additionally, TSRG allows adjusting and interpreting the interaction pattern between the two generation stages, making the generation response somewhat explainable and controllable. We evaluate the proposed model on three dialogue datasets that contain millions of single-turn message-response pairs between web users. The results show that, compared with the previous multi-stage dialogue generation models, TSRG can produce more diverse and informative responses and maintain fluency and relevance.

Previous works show that Pre-trained Language Models (PLMs) can capture factual knowledge.However, some analyses reveal that PLMs fail to perform it robustly, e.g., being sensitive to the changes of prompts when extracting factual knowledge.To mitigate this issue, we propose to let PLMs learn the deterministic relationship between the remaining context and the masked content.The deterministic relationship ensures that the masked factual content can be deterministically inferable based on the existing clues in the context.That would provide more stable patterns for PLMs to capture factual knowledge than randomly masking.Two pre-training tasks are further introduced to motivate PLMs to rely on the deterministic relationship when filling masks.Specifically, we use an external Knowledge Base (KB) to identify deterministic relationships and continuously pre-train PLMs with the proposed methods.The factual knowledge probing experiments indicate that the continuously pre-trained PLMs achieve better robustness in factual knowledge capturing.Further experiments on question-answering datasets show that trying to learn a deterministic relationship with the proposed methods can also help other knowledge-intensive tasks.

The recent surges in the open conversational data caused Emotion Recognition in Spoken Dialog (ERSD) to gain much attention. However, the existing ERSD datasets’ scale limits the model’s complete reasoning. Moreover, the artificial dialogue agent is ideally able to reference past dialogue experiences. This paper proposes a Commonsense Knowledge Enhanced Network with a retrospective loss, namely CKE-Net, to hierarchically perform dialog modeling, external knowledge integration, and historical state retrospect. Specifically, we first adopt a transformer-based encoder to model context in multi-view by elaborating different mask matrices. Then, the graph attention network is used to introduce commonsense knowledge, which benefits the complex emotional reasoning. Finally, a retrospective loss is added to utilize the model’s prior experience during training. Experiments on IEMOCAP and MELD datasets demonstrate that every designed module is consistently beneficial to the performance. Extensive experimental results show that our model outperforms the state-of-the-art models across the two benchmark datasets.

Collecting supporting evidence from large corpora of text (e.g., Wikipedia) is of great challenge for open-domain Question Answering (QA). Especially, for multi-hop open-domain QA, scattered evidence pieces are required to be gathered together to support the answer extraction. In this paper, we propose a new retrieval target, hop, to collect the hidden reasoning evidence from Wikipedia for complex question answering. Specifically, the hop in this paper is defined as the combination of a hyperlink and the corresponding outbound link document. The hyperlink is encoded as the mention embedding which models the structured knowledge of how the outbound link entity is mentioned in the textual context, and the corresponding outbound link document is encoded as the document embedding representing the unstructured knowledge within it. Accordingly, we build HopRetriever which retrieves hops over Wikipedia to answer complex questions. Experiments on the HotpotQA dataset demonstrate that HopRetriever outperforms previously published evidence retrieval methods by large margins. Moreover, our approach also yields quantifiable interpretations of the evidence collection process.

This paper presents a new methodology for modeling the local semantic distribution of responses to a given query in the human-conversation corpus, and on this basis, explores a specified adversarial learning mechanism for training Neural Response Generation (NRG) models to build conversational agents. Our investigation begins with the thorough discus- sions upon the objective function of general Generative Adversarial Nets (GAN) architectures, and the training instability problem is proved to be highly relative with the special local distributions of conversational corpora. Consequently, an energy function is employed to estimate the status of a local area restricted by the query and its responses in the semantic space, and the mathematical approximation of this energy-based distribution is finally found. Building on this foundation, a local distribution oriented objective is proposed and combined with the original objective, working as a hybrid loss for the adversarial training of response generation models, named as LocalGAN. Our experimental results demonstrate that the reasonable local distribution modeling of the query-response corpus is of great importance to adversarial NRG, and our proposed LocalGAN is promising for improving both the training stability and the quality of generated results.

Knowledge graph embedding, which aims to learn low-dimensional embeddings of entities and relations, plays a vital role in a wide range of applications. It is crucial for knowledge graph embedding models to model and infer various relation patterns, such as symmetry/antisymmetry, inversion, and composition. However, most existing methods fail to model the non-commutative composition pattern, which is essential, especially for multi-hop reasoning. To address this issue, we propose a new model called Rotate3D, which maps entities to the three-dimensional space and defines relations as rotations from head entities to tail entities. By using the non-commutative composition property of rotations in the three-dimensional space, Rotate3D can naturally preserve the order of the composition of relations. Experiments show that Rotate3D outperforms existing state-of-the-art models for link prediction and path query answering. Further case studies demonstrate that Rotate3D can effectively capture various relation patterns with a marked improvement in modeling the composition pattern.

In human-to-human conversations, the context generally provides several backgrounds and strategic points for the following response. Therefore, many response generation approaches have explored the methodologies to incorporate the context into the encoder-decoder architecture, to generate context-aware responses that are remarkably relevant and cohesive to the given context. However, most approaches pay less attention to semantic interactions implicitly existing within contextual utterances, which are of great importance to capture semantic clues of the given dialog context, indeed. This paper proposes a dynamic working memory mechanism to model long-term semantic hints in the conversation context, by performing semantic interactions between utterances and updating context representation dynamically. Then, the outputs of the dynamic working memory are employed to provide helpful clues for the encoder-decoder architecture to generate responses to the given dialog. We have evaluated the proposed approach on Twitter Customer Service Corpus and OpenSubtitles Corpus, with several automatic evaluation metrics and the human evaluation, and the empirical results show the effectiveness of the proposed method.

In real-time bidding advertising (RTB), the buyers bid for individual advertisement impressions provided by publishers in real time. The final goal of the buyers is to maximize the return on their investment. To gain higher returns, buyers prefer to first purchase more conversion impressions than click-only ones and then purchase more click-only impressions prior to non-click ones. Simultaneously, to reduce the expense, they need to accurately estimate a reasonable bid price, the predicted precision of which depends on the precision of the predicted conversion rate (CVR) or predicted click-through rate (CTR). Therefore, the predicted CVR or predicted CTR must provide not only good ranking values but also correct regression estimations. This paper is focused on the CVR estimation problem for buy-sides in RTB and a combined regression and tripletwise ranking method (CRT) is proposed that jointly considers regression loss and tripletwise ranking loss to estimate the CVR. This method attempts to rank conversion impressions above click-only ones and simultaneously rank click-only impressions above non-click ones. Meanwhile, through simultaneously utilizing the historical conversion and click information to alleviate sparsity, the CRT method is also aimed to achieve a good two category-ranking performance, as well as a good regression performance for predicting the CVR.

Automatic chatbots (also known as chat-agents) have attracted much attention from both researching and industrial fields. Generally, the semantic relevance between users' queries and the corresponding responses is considered as the essential element for conversation modeling in both generation and ranking based chat systems. By contrast, it is a nontrivial task to adopt the users' information, such as preference, social role, etc., into conversational models reasonably, while users' profiles play a significant role in the procedure of conversations by providing the implicit contexts. This paper aims to address the personalized response ranking task by incorporating user profiles into the conversation model. In our approach, users' personalized representations are latently learned from the contents posted by them via a two-branch neural network. After that, a deep neural network architecture is further presented to learn the fusion representation of posts, responses, and personal information. In this way, the proposed model could understand conversations from the users' perspective; hence, the more appropriate responses are selected for a specified person. The experimental results on two datasets from social network services demonstrate that our approach is hopeful to represent users' personal information implicitly based on user generated contents, and it is promising to perform as an important component in chatbots to select the personalized responses for each user.

This paper presents a Generative Adversarial Network (GAN) to model singleturn short-text conversations, which trains a sequence-to-sequence (Seq2Seq) network for response generation simultaneously with a discriminative classifier that measures the differences between human-produced responses and machinegenerated ones.In addition, the proposed method introduces an approximate embedding layer to solve the non-differentiable problem caused by the sampling-based output decoding procedure in the Seq2Seq generative model.The GAN setup provides an effective way to avoid noninformative responses (a.k.a "safe responses"), which are frequently observed in traditional neural response generators.The experimental results show that the proposed approach significantly outperforms existing neural response generation models in diversity metrics, with slight increases in relevance scores as well, when evaluated on both a Mandarin corpus and an English corpus.

It is critical for automatic chat-bots to gain the ability of conversation comprehension, which is the essence to provide context-aware responses to conduct smooth dialogues with human beings. As the basis of this task, conversation modeling will notably benefit from the background knowledge, since such knowledge indeed implicates semantic hints that help to further clarify the relationships between sentences within a conversation. In this paper, a deep neural network is proposed to incorporate background knowledge for conversation modeling. Through a recall mechanism with a specially designed recall-gate, background knowledge as global memory can be motivated to cooperate with local cell memory of Long Short-Term Memory (LSTM), so as to enrich the ability of LSTM to capture the implicit semantic clues in conversations. In addition, this paper introduces the loose-structured domain knowledge as background knowledge, which can be built with slight amount of manual work and easily adopted by the recall-gate. Our model is evaluated on the context-oriented response selecting task, and experimental results on two datasets have shown that our approach is promising for modeling conversations and building key components of automatic chat systems.

Drug Entity Recognition (DER) is a crucial task for information extraction in biomedical text. Much of previous work for DER using known drugs to build features, however, the known drug resources are limited. In this paper, we proposed a semi-supervised learning to extend an existing drug dictionary. With the extended dictionary, the features for DER can be enriched. Using Conditional Random Fields (CRF) model with the enriched features, an F-measure of 89.26% is achieved on DDIExtraction2013 challenge data set, which outperforms the best system of the DDIExtraction 2013 challenge.

Xin Wang, Yuanchao Liu, Chengjie Sun, Baoxun Wang, Xiaolong Wang. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015.

The link prediction problem in social networks is to estimate the value of the link that can represent relationship between social members. Researchers have proposed several methods for solving link prediction and a number of features have been used. Most of these models are learned with only considering the features from one kind of data. In this paper, by considering the data from link network structure and user comment, both of which could imply the concept of link value, we propose multimodal learning based approaches to predict the link values. The experiment results done on dataset from typical social networks show that our model could learn the joint representation of these datas properly, and the method MDBN outperforms other state-of-art link prediction methods.

In this paper, convolutional neural networks are employed for English article error correction. Instead of employing features relying on human ingenuity and prior natural language processing knowledge, the words surrounding the context of the article are taken as features. Our approach could be trained both on an error annotated corpus and an error non-annotated corpus. Experiments are conducted on CoNLL-2013 data set. Our approach achieves 38.10 % in F1, and outperforms the best system (33.40 %) that participates in the task. Experimental results demonstrate the effectiveness of our proposed approach.

The human-generated question-answer pairs in the Web social communities are of great value for the research of automatic question-answering technique. Due to the large amount of noise information involved in such corpora, it is still a problem to detect the answers even though the questions are exactly located. Quantifying the semantic relevance between questions and their candidate answers is essential to answer detection in social media corpora. Since both the questions and their answers usually contain a small number of sentences, the relevance modeling methods have to overcome the problem of word feature sparsity. In this article, the deep learning principle is introduced to address the semantic relevance modeling task. Two deep belief networks with different architectures are proposed by us to model the semantic relevance for the question-answer pairs. According to the investigation of the textual similarity between the community-driven question-answering (cQA) dataset and the forum dataset, a learning strategy is adopted to promote our models’ performance on the social community corpora without hand-annotating work. The experimental results show that our method outperforms the traditional approaches on both the cQA and the forum corpora.

Extracting question-answer pairs from online forums is a meaningful work due to the huge amount of valuable user generated resource contained in forums. In this paper we consider the problem of extracting Chinese question-answer pairs for the first time. We present a strategy to detect Chinese questions and their answers. We propose a sequential rule based method to find questions in a forum thread, then we adopt non-textual features based on forum structure to improve the performance of answer detecting in the same thread. Experimental results show that our techniques are very effective.

Conditional Random Fields (CRFs) have been used for many sequence labeling tasks and got excellent results. Further, the supervised model strongly depends on the huge training data. Active learning is a different way rather than relying on a large amount random sampling. However, random sampling constructively participates in the optimal choosing training examples. Based on different query strategies, active learning can combine with other machine learning methods to reduce the annotation cost while maintaining the accuracy. This paper proposes a new active learning strategy based on Information Density (ID) integrated with CRFs for Chinese Named Entity Recognition (NER). On Sighan bakeoff 2006 MSRA NER corpus, an F1 score of 77.2% is achieved by using only 10,000 labeled training sentences chosen by the proposed active learning strategy.