The confluence of Federated Learning (FL) and Large Language Models (LLMs) is ushering in a new era in privacy-preserving natural language processing. However, the intensive memory requirements for fine-tuning LLMs pose significant challenges, especially when deploying on clients with limited computational resources. To circumvent this, we explore the novel integration of Memory-efficient Zeroth-Order Optimization within a federated setting, a synergy we term as FedMeZO. Our study is the first to examine the theoretical underpinnings of FedMeZO in the context of LLMs, tackling key questions regarding the influence of large parameter spaces on optimization behavior, the establishment of convergence properties, and the identification of critical parameters for convergence to inform personalized federated strategies. Our extensive empirical evidence supports the theory, showing that FedMeZO not only converges faster than traditional first-order methods such as FedAvg but also significantly reduces GPU memory usage during training to levels comparable to those during inference. Moreover, the proposed personalized FL strategy that is built upon the theoretical insights to customize the client-wise learning rate can effectively accelerate loss reduction. We hope our work can help to bridge theoretical and practical aspects of federated fine-tuning for LLMs, thereby stimulating further advancements and research in this area.

Emotional Support Conversation (ESC) systems are pivotal in providing empathetic interactions, aiding users through negative emotional states by understanding and addressing their unique experiences. In this paper, we tackle two key challenges in ESC: enhancing contextually relevant and empathetic response generation through dynamic demonstration retrieval, and advancing cognitive understanding to grasp implicit mental states comprehensively. We introduce Dynamic Demonstration Retrieval and Cognitive-Aspect Situation Understanding (D2RCU), a novel approach that synergizes these elements to improve the quality of support provided in ESCs. By leveraging in-context learning and persona information, we introduce an innovative retrieval mechanism that selects informative and personalized demonstration pairs. We also propose a cognitive understanding module that utilizes four cognitive relationships from the ATOMIC knowledge source to deepen situational awareness of help-seekers' mental states. Our supportive decoder integrates information from diverse knowledge sources, underpinning response generation that is both empathetic and cognitively aware. The effectiveness of D2RCU is demonstrated through extensive automatic and human evaluations, revealing substantial improvements over numerous state-of-the-art models, with up to 13.79% enhancement in overall performance of ten metrics. Our codes are available for public access to facilitate further research and development.

Question generation (QG) aims to automatically generate fluent and relevant questions, where the two most mainstream directions are generating questions from unstructured contextual texts (CQG), such as news articles, and generating questions from structured factoid texts (FQG), such as knowledge graphs or tables. Existing methods for these two tasks mainly face challenges of limited internal structural information as well as scarce background information, while these two tasks can benefit each other for alleviating these issues. For example, when meeting the entity mention “United Kingdom” in CQG, it can be inferred that it is a country in European continent based on the structural knowledge “(Europe, countries_within, United Kingdom)” in FQG. And when meeting the entity “Houston Rockets” in FQG, more background information, such as “an American professional basketball team based in Houston since 1971”, can be found in the related passages of CQG. To this end, we propose a unified framework for the tasks of CQG and FQG, where: (i) two types of task-sharing modules are developed to learn shared contextual and structural knowledge, where the task format is unified with a pseudo passage reformulation strategy; (ii) for the CQG task, a task-specific knowledge module with a knowledge selection and aggregation mechanism is introduced, so as to incorporate more factoid knowledge from external knowledge graphs and alleviate the word ambiguity problem; and (iii) for the FQG task, a task-specific passage module with a multi-level passage fusion mechanism is designed to extract fine-grained word-level knowledge. Experimental results in both automatic and human evaluation show the effectiveness of our proposed method.

The goal of semi-supervised text classification (SSTC) is to train a model by exploring both a small number of labeled data and a large number of unlabeled data, such that the learned semi-supervised classifier performs better than the supervised classifier trained on solely the labeled samples. Pseudo-labeling is one of the most widely used SSTC techniques, which trains a teacher classifier with a small number of labeled examples to predict pseudo labels for the unlabeled data. The generated pseudo-labeled examples are then utilized to train a student classifier, such that the learned student classifier can outperform the teacher classifier. Nevertheless, the predicted pseudo labels may be inaccurate, making the performance of the student classifier degraded. The student classifier may perform even worse than the teacher classifier. To alleviate this issue, in this paper, we introduce a dual meta-learning ( DML ) technique for semi-supervised text classification, which improves the teacher and student classifiers simultaneously in an iterative manner. Specifically, we propose a meta-noise correction method to improve the student classifier by proposing a Noise Transition Matrix (NTM) with meta-learning to rectify the noisy pseudo labels. In addition, we devise a meta pseudo supervision method to improve the teacher classifier. Concretely, we exploit the feedback performance from the student classifier to further guide the teacher classifier to produce more accurate pseudo labels for the unlabeled data. In this way, both teacher and student classifiers can co-evolve in the iterative training process. Extensive experiments on four benchmark datasets highlight the effectiveness of our DML method against existing state-of-the-art methods for semi-supervised text classification. We release our code and data of this paper publicly at https://github.com/GRIT621/DML.

Cross-modal retrieval has attracted much attention lately for its various applications in Internet data mining. Existing approaches mainly adopt the projection function learning paradigm to construct dual-stream models, which suffer from two limitations: 1) They only utilize the correlations provided by cross-modal data pairs but the multiple correlations among data are unexplored. 2) They typically face the challenge of abstractness of semantics, which means that an instance may have distinct semantic information in different scenarios. In this paper, we propose a novel graph learning based framework termed Multimodal Graph Learning for cross-modal retrieval (MGL), which aims to fully exploit multiple correlations embedded in multimodal data and leverage a graph neural network to capture complementary information to alleviate the information sparsity and abstractness of semantics. First, we propose a graph construction algorithm to explore diverse multimedia information. Second, a modal feature projector is designed to learn modality-shared information, and a co-attention mechanism module is proposed to capture complementary information and perform dynamic feature integration. Third, a fusion and gate module is proposed to fully aggregate captured information and perform denoising. Furthermore, we employ a graph sampling algorithm to make our approach flexible to large-scale scenarios. Experimental results on three benchmark datasets prove the effectiveness of MGL.

Knowledge graph embedding aims at modeling knowledge by projecting entities and relations into a low-dimensional semantic space. Most of the works on knowledge graph embedding construct negative samples by negative sampling as knowledge graphs typically only contain positive facts. Although substantial progress has been made by dynamic distribution based sampling methods, selecting plausible and prior information-engaged negative samples still poses many challenges. Inspired by type constraint methods, we propose Hierarchical Type Enhanced Negative Sampling (HTENS) which leverages hierarchical entity type information and entity-relation cooccurrence information to optimize the sampling probability distribution of negative samples. The experiments performed on the link prediction task demonstrate the effectiveness of HTENS. Additionally, HTENS shows its superiority in versatility and can be integrated into scalable systems with enhanced negative sampling.

Entity Set Expansion (ESE) is a valuable task that aims to find entities of the target semantic class described by given seed entities. Various Natural Language Processing (NLP) and Information Retrieval (IR) downstream applications have benefited from ESE due to its ability to discover knowledge. Although existing corpus-based ESE methods have achieved great progress, they still rely on corpora with high-quality entity information annotated, because most of them need to obtain the context patterns through the position of the entity in a sentence. Therefore, the quality of the given corpora and their entity annotation has become the bottleneck that limits the performance of such methods. To overcome this dilemma and make the ESE models free from the dependence on entity annotation, our work aims to explore a new ESE paradigm, namely corpus-independent ESE. Specifically, we devise a context pattern generation module that utilizes autoregressive language models (e.g., GPT-2) to automatically generate high-quality context patterns for entities. In addition, we propose the GAPA, a novel ESE framework that leverages the aforementionedGenerAtedPAtterns to expand target entities. Extensive experiments and detailed analyses on three widely used datasets demonstrate the effectiveness of our method. All the codes of our experiments are available athttps://github.com/geekjuruo/GAPA.

The task of ACE Event Detection (ED) often encounters ambiguous and unseen trigger words. Most conventional ED systems exclusively consider the semantic or syntactic patterns as the additional evidence to resolve the problem of the ambiguous and unseen triggers, but rarely consider taking advantages of structured knowledge of the event itself. In this study, we propose Dynamic Word-Trigger-Argument Graph Neural Networks (DWTA-GNN), a novel framework that leverages event structure knowledge to facilitate the two issues simultaneously. In our approach, we utilize words, entities, and event annotations from training to construct an event background graph, which can provide sufficient information of event structure to better disambiguate polysemous triggers and identify unseen triggers. To make full use of the constructed background graph, we further design a knowledge matching module to dynamically match appropriate event structure knowledge and construct a subgraph for each incoming sentence. Besides, an event-selective graph convolution is applied to filter out the noise in the matched knowledge so as to enhance event representation. Experiments on the ACE2005 dataset show that our model achieves competitive performance and advances previous approaches on ambiguous and unseen trigger words, verifying the effectiveness of incorporating event structure knowledge for event detection.

Entity Set Expansion (ESE) is a promising task which aims to expand entities of the target semantic class described by a small seed entity set. Various NLP and IR applications will benefit from ESE due to its ability to discover knowledge. Although previous ESE methods have achieved great progress, most of them still lack the ability to handle hard negative entities (i.e., entities that are difficult to distinguish from the target entities), since two entities may or may not belong to the same semantic class based on different granularity levels we analyze on. To address this challenge, we devise an entity-level masked language model with contrastive learning to refine the representation of entities. In addition, we propose the ProbExpan, a novel probabilistic ESE framework utilizing the entity representation obtained by the aforementioned language model to expand entities. Extensive experiments and detailed analyses on three datasets show that our method outperforms previous state-of-the-art methods.

Answer selection, which is involved in many natural language processing applications, such as dialog systems and question answering (QA), is an important yet challenging task in practice, since conventional methods typically suffer from the issues of ignoring diverse real-world background knowledge. In this article, we extensively investigate approaches to enhancing the answer selection model with external knowledge from knowledge graph (KG). First, we present a context-knowledge interaction learning framework, Knowledge-aware Neural Network, which learns the QA sentence representations by considering a tight interaction with the external knowledge from KG and the textual information. Then, we develop two kinds of knowledge-aware attention mechanism to summarize both the context-based and knowledge-based interactions between questions and answers. To handle the diversity and complexity of KG information, we further propose a Contextualized Knowledge-aware Attentive Neural Network, which improves the knowledge representation learning with structure information via a customized Graph Convolutional Network and comprehensively learns context-based and knowledge-based sentence representation via the multi-view knowledge-aware attention mechanism. We evaluate our method on four widely used benchmark QA datasets, including WikiQA, TREC QA, InsuranceQA, and Yahoo QA. Results verify the benefits of incorporating external knowledge from KG and show the robust superiority and extensive applicability of our method.

In this paper, we propose a novel abstractive text summarization method with hierarchical multi-scale abstraction modeling and dynamic memory (called MADY). First, we propose a hierarchical multi-scale abstraction modeling method to capture the temporal dependencies of the document from multiple hierarchical levels of abstraction, which mimics the process of how human beings comprehend an article by learning fine timescales for low-level abstraction layers and coarse timescales for high-level abstraction layers. By applying this adaptive updating mechanism, the high-level abstraction layers are updated less frequently and expected to remember the long-term dependency better than the low-level abstraction layer. Second, we propose a dynamic key-value memory-augmented attention network to keep track of the attention history and comprehensive context information for the salient facets within the input document. In this way, our model can avoid generating repetitive words and faultiness summaries. Extensive experiments on two widely-used datasets demonstrate the effectiveness of the proposed MADY model in terms of both automatic evaluation and human evaluation. For reproducibility, we submit the code and data at: https://github.com/siat-nlp/MADY.git.

Knowledge graphs (KG) often encounter knowledge incompleteness. The path reasoning that predicts the unknown path relation between pairwise entities based on existing facts is one of the most promising approaches to the knowledge graph completion. However, most conventional path reasoning methods exclusively consider the entity description included in fact triples, ignoring both the type information of entities and the interaction between different semantic representations. In this study, we propose a novel method, Type-aware Attentive Path Reasoning (TAPR), to complete the knowledge graph by simultaneously considering KG structural information, textual information, and type information. More specifically, we first leverage types to enrich the representational learning of entities and relationships. Next, we describe a type-level attention to select the most relevant type of given entity in a specific triple without any predefined rules or patterns to reduce the impact of noisy types. After learning the distributed representation of all paths, path-level attention assigns different weights to paths, from which relations among entity pairs are calculated. We conduct a series of experiments on a real-world dataset to demonstrate the effectiveness of TAPR. Experimental results show that our method significantly outperforms all baselines on link prediction and entity prediction tasks.

Heterogeneous information networks (HINs) have been ubiquitous in the real-world. HIN embeddings, which encode various information of the networks into low-dimensional vectors, can facilitate a wide range of applications on graph-structured data. Existing HIN embedding methods include random walk based methods that may not fully utilize the edge semantics and knowledge graph embedding methods that restrict the expression ability of topological information. In this paper, we propose a novel adaptive embedding framework, which integrates these two kinds of methods to preserve both topological information and relational information. By incorporating an assistant knowledge graph embedding model, the proposed framework performs efficient biased random walk under the guidance of edge semantics.

In this paper, we propose a multi-source domain adaptation method with a Granger-causal objective (MDA-GC) for cross-domain sentiment classification. Specifically, for each source domain, we build an expert model by using a novel sentiment-guided capsule network, which captures the domain invariant knowledge that bridges the knowledge gap between the source and target domains. Then, an attention mechanism is devised to assign importance weights to a mixture of experts, each of which specializes in a different source domain. In addition, we propose a Granger causal objective to make the weights assigned to individual experts correlate strongly with their contributions to the decision at hand. Experimental results on a benchmark dataset demonstrate that the proposed MDA-GC model significantly outperforms the compared methods.

Non-factoid question answering (QA) is one of the most extensive yet challenging application and research areas of retrieval-based question answering. In particular, answers to non-factoid questions can often be too lengthy and redundant to comprehend, which leads to the great demand on answer sumamrization in non-factoid QA. However, the multi-level interactions between QA pairs and the interrelation among different answer sentences are usually modeled separately on current answer summarization studies. In this paper, we propose a unified model to bridge hierarchical and sequential context modeling for question-driven extractive answer summarization. Specifically, we design a hierarchical compare-aggregate method to integrate the interaction between QA pairs in both word-level and sentence-level into the final question and answer representations. After that, we conduct the question-aware sequential extractor to produce a summary for the lengthy answer. Experimental results show that answer summarization benefits from both hierarchical and sequential context modeling and our method achieves superior performance on WikiHowQA and PubMedQA.

Textual entailment is a central problem of language variability, which has been attracting a lot of interest and it poses significant issues in front of systems aimed at natural language understanding. Recently, various frameworks have been proposed for textual entailment recognition, ranging from traditional computational linguistics techniques to deep learning model based methods. However, recent deep neural networks that achieve the state of the art on textual entailment task only consider the context information of the given sentences rather than the real-world background information and knowledge beyond the context. In the paper, we propose a Knowledge-Context Interactive Textual Entailment Network (KCI-TEN) that learns graph level sentence representations by harnessing external knowledge graph with graph attention network. We further propose a text-graph interaction mechanism for neural based entailment matching learning, which endows the redundancy and noise with less importance and put emphasis on the informative representations. Experiments on the SciTail dataset demonstrate that KCI-TEN outperforms the state-of-the-art methods.

Knowledge Graph (KG) contains entities and the relations between entities. Due to its representation ability, KG has been successfully applied to support many medical/healthcare tasks. However, in the medical domain, knowledge holds under certain conditions. Such conditions for medical knowledge are crucial for decision-making in various medical applications, which is missing in existing medical KGs. In this paper, we aim to discovery medical knowledge conditions from texts to enrich KGs. Electronic Medical Records (EMRs) are systematized collection of clinical data and contain detailed information about patients, thus EMRs can be a good resource to discover medical knowledge conditions. Unfortunately, the amount of available EMRs is limited due to reasons such as regularization. Meanwhile, a large amount of medical question answering (QA) data is available, which can greatly help the studied task. However, the quality of medical QA data is quite diverse, which may degrade the quality of the discovered medical knowledge conditions. In the light of these challenges, we propose a new truth discovery method, MedTruth, for medical knowledge condition discovery, which incorporates prior source quality information into the source reliability estimation procedure, and also utilizes the knowledge triple information for trustworthy information computation. We conduct series of experiments on real-world medical datasets to demonstrate that the proposed method can discover meaningful and accurate conditions for medical knowledge by leveraging both EMR and QA data. Further, the proposed method is tested on synthetic datasets to validate its effectiveness under various scenarios.

In this paper, we propose a multi-task learning approach for cross-modal image-text retrieval. First, a correlation network is proposed for relation recognition task, which helps learn the complicated relations and common information of different modalities. Then, we propose a correspondence cross-modal autoencoder for cross-modal input reconstruction task, which helps correlate the hidden representations of two uni-modal autoencoders. In addition, to further improve the performance of cross-modal retrieval, two regularization terms (variance and consistency constraints) are introduced to the cross-modal embeddings such that the learned common information has large variance and is modality invariant. Finally, to enable large-scale cross-modal similarity search, a flexible binary transform network is designed to convert the text and image embeddings into binary codes. Extensive experiments on two benchmark datasets demonstrate that our model has robust superiority over the compared strong baseline methods. Source code is available at \urlhttps://github.com/daerv/DAEVR.

Knowledge graphs (KGs) have been applied to many semantic-driven applications, including knowledge interchange and semantic inference. However, most KGs are far from complete and are growing rapidly. Although significant progress has been made in the symbolic representation learning of KGs with structural information, the textual knowledge that plays a crucial role in relation prediction is underutilized, and the issues of redundancy and noise path remain to be settled. In this paper, a Path-based Attribute-aware Representation Learning model (PARL) has been proposed to perform path denoising and path representation learning for the relation prediction task. We develop a novel text-enhanced relation prediction architecture, which interactively learns KG structural and textual representations to vary the sparsity and reliability of KG. Moreover, a path denoising algorithm is presented to emphasize paths with rich information and reduce the impact of redundancy and noise path. Experiments on a public dataset demonstrate that PARL consistently outperforms state-of-the-art methods on relation prediction and KG completion tasks.

Knowledge Based Question Answering (KBQA) is one of the most promising approaches to provide suitable answers for the queries posted by users. Relation detection that aims to take full advantage of the substantial knowledge contained in knowledge base (KB) becomes increasingly important. Significant progress has been made in performing relation detection over KB. However, recent deep neural networks that achieve the state of the art on KB-based relation detection task only consider the context information of question sentences rather than the relatedness between question and answer candidates, and exclusively extract the relation from KB triple rather than learn informative relational path. In this paper, we propose a Knowledge-driven Relation Detection network (KRD) to interactively learn answer-enhanced question representations and path-aware relation representations for relation detection. A Siamese LSTM is employed into a similarity matching process between the question representation and relation representation. Experimental results on the SimpleQuestions and WebQSP datasets demonstrate that KRD outperforms the state-of-the-art methods. In addition, a series of ablation test show the robust superiority of the proposed method.

This study takes the lead to study the aspect/sentiment-aware abstractive review summarization in domain adaptation scenario. The proposed model CASAS (neural attentive model for Cross-domain Aspect/Sentiment-aware Abstractive review Summarization) leverages domain classification task, working on datasets of both source and target domains, to recognize the domain information of texts and transfer knowledge from source domains to target domains. The extensive experiments on Amazon reviews demonstrate that CASAS outperforms the compared methods in both out-of-domain and in-domain setups.

With the rising importance of knowledge exchange, ontologies have become a key technology in the development of shared knowledge models for semantic-driven applications, such as knowledge interchange and semantic integration. Significant progress has been made in the use of entropy to measure the predictability and redundancy of knowledge bases, particularly ontologies. However, the current entropy applications used to evaluate ontologies consider only single-point connectivity rather than path connectivity, assign equal weights to each entity and path, and assume that vertices are static. To address these deficiencies, the present study proposes a Path-based Text-aware Entropy Computation method, PTEC, by considering the path information between different vertices and the textual information within the path to calculate the connectivity path of the whole network and the different weights between various nodes. Information obtained from structure-based embedding and text-based embedding is multiplied by the connectivity matrix of the entropy computation. An experimental evaluation of three real-world ontologies is performed based on ontology statistical information (data quantity), entropy evaluation (data quality), and a case study (ontology structure and text visualization). These aspects mutually demonstrate the reliability of our method. Experimental results demonstrate that PTEC can effectively evaluate ontologies, particularly those in the medical field.

Ranking question answer pairs has attracted increasing attention recently due to its broad applications such as information retrieval and question answering (QA). Significant progresses have been made by deep neural networks. However, background information and hidden relations beyond the context, which play crucial roles in human text comprehension, have received little attention in recent deep neural networks that achieve the state of the art in ranking QA pairs. In the paper, we propose KABLSTM, a Knowledge-aware Attentive Bidirectional Long Short-Term Memory, which leverages external knowledge from knowledge graphs (KG) to enrich the representational learning of QA sentences. Specifically, we develop a context-knowledge interactive learning architecture, in which a context-guided attentive convolutional neural network (CNN) is designed to integrate knowledge embeddings into sentence representations. Besides, a knowledge-aware attention mechanism is presented to attend interrelations between each segments of QA pairs. KABLSTM is evaluated on two widely-used benchmark QA datasets: WikiQA and TREC QA. Experiment results demonstrate that KABLSTM has robust superiority over competitors and sets state-of-the-art.