Shang Qin, Jingheng Ye, Yinghui Li, Hai-Tao Zheng, Qi Li, Jinxiao Shan, Zhixing Li, Hong-Gee Kim. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Jiwei Tang, Zhicheng Zhang, Shunlong Wu, Jingheng Ye, Lichen Bai, Zitai Wang, Tingwei Lu, Lin Hai, Yiming Zhao, Hai-Tao Zheng, Hong-Gee Kim. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Neural sequence models, despite their proficiency in generating highly fluent sentences, have also exhibited a tendency to hallucinate, introducing additional content that lacks grounding in the input data, as evidenced by recent investigations. This variety of fluent yet erroneous outputs poses a significant challenge, as it is difficult for users to discern the veracity of the presented content and identify inaccuracies. Several methods have been proposed to address this challenge. However, they sometimes detect the synonym of the true token in the generation as hallucination. To alleviate the above problem, we propose a novel architecture, i.e., Synonym Knowledge Graph Enhanced Language Model (SKGELM). We construct and prune a synonym knowledge graph according to the source sentence, which can help the subsequently graph attention network and classifier detect hallucination. Empirical results on SUMMAC benchmark and multi-domain Chinese–English translation benchmark show that our method achieves state-of-the-art performance and improves the best baseline significantly.

Large language models (LLMs) have demonstrated remarkable performance in multimodal tasks even with frozen LLM Block and only a few trainable parameters. However, the underlying mechanisms of how LLMs enhance multimodal performance remains unclear. In this work, we focus on the phenomenon that ``Merely concatenating a frozen LLM block to the Vision Transformer (ViT) encoder can yield significant performance enhancements. Moreover, the choice of LLM block and insertion position can have a substantial impact, leading to varying degrees of improvement''. We analyze the optimization of the training process from the perspective of gradient dynamics and find that frozen LLM blocks act as gradient coherence rectifiers, aligning the gradients of different samples more closely during training. Furthermore, we demonstrate that the representation similarity between the inserted LLM block and the adjacent ViT block influences performance, with greater similarity tending to yield larger positive gains. Through these findings, we can justify the selection of suitable LLM blocks to be inserted at appropriate positions, and introduce additional gradient backpropagation paths by incorporating LLM blocks, could improve the performance of vanilla ViT through the rectification effect of gradient consistency during the training process, without the need to add LLM blocks during inference. Our experiments demonstrate the effectiveness of this strategy, making the practical application of the gradient rectification effect feasible.

Existing studies explore the explainability of Grammatical Error Correction (GEC) in a limited scenario, where they ignore the interaction between corrections and explanations and have not established a corresponding comprehensive benchmark. To bridge the gap, this paper first introduces the task of EXplainable GEC (EXGEC), which focuses on the integral role of correction and explanation tasks. To facilitate the task, we propose EXCGEC, a tailored benchmark for Chinese EXGEC consisting of 8,216 explanation-augmented samples featuring the design of hybrid edit-wise explanations. We then benchmark several series of LLMs in multi-task learning settings, including post-explaining and pre-explaining. To promote the development of the task, we also build a comprehensive evaluation suite by leveraging existing automatic metrics and conducting human evaluation experiments to demonstrate the human consistency of the automatic metrics for free-text explanations. Our experiments reveal the effectiveness of evaluating free-text explanations using traditional metrics like METEOR and ROUGE, and the inferior performance of multi-task models compared to the pipeline solution, indicating its challenges to establish positive effects in learning both tasks.

Jingheng Ye, Zishan Xu, Yinghui Li, Linlin Song, Qingyu Zhou, Hai-Tao Zheng, Ying Shen, Wenhao Jiang, Hong-Gee Kim, Ruitong Liu, Xin Su, Zifei Shan. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Session-based Social Recommendation (SSR) enhances item recommendations by incorporating both session interactions and social network data. Despite recent progress, existing SSR methods-primarily based on Graph Neural Networks-are highly susceptible to session noise (irrelevant or unintentional interactions) and social noise (misleading signals from connected users). Prior denoising strategies often rely on heuristic resampling or reweighting techniques, which lack generalizability and robustness across diverse datasets. In this work, we explore a novel direction by introducing diffusion models for denoising in SSR. However, applying diffusion to SSR presents unique challenges due to heterogeneous data modalities, incompatible noise patterns, and the absence of semantic guidance during the reverse process. To overcome these challenges, we propose D3MRec, a Dual Denoising Diffusion Model specifically designed for SSR. D3MRec employs a dual-branch architecture that independently models session sequences and social graphs, applying denoising diffusion in their respective hidden representation spaces. This decoupled design preserves the structural integrity of each modality while enabling modality-specific denoising. Moreover, we introduce cross-modal guidance by leveraging collaborative signals from the other branch during the reverse diffusion process, enhancing alignment between session intents and social preferences. The dual denoising processes not only mitigate noise within each modality but also serve as mutual priors, facilitating robust and consistent representation learning across modalities. Extensive experiments on multiple benchmarks show that D3MRec significantly outperforms state-of-the-art models, particularly under noisy conditions, demonstrating its effectiveness and robustness.

How to better evaluate the capabilities of Large Language Models (LLMs) is the focal point and hot topic in current LLMs research. Previous work has noted that due to the extremely high cost of iterative updates of LLMs, they are often unable to answer the latest dynamic questions well. To promote the improvement of Chinese LLMs’ ability to answer dynamic questions, in this paper, we introduce CDQA, a Chinese Dynamic QA benchmark containing question-answer pairs related to the latest news on the Chinese Internet. We obtain high-quality data through a pipeline that combines humans and models, and carefully classify the samples according to the frequency of answer changes to facilitate a more fine-grained observation of LLMs’ capabilities. We have also evaluated and analyzed mainstream and advanced Chinese LLMs on CDQA. Extensive experiments and valuable insights suggest that our proposed CDQA is challenging and worthy of more further study. We believe that the benchmark we provide will become one of the key data resources for improving LLMs’ Chinese question-answering ability in the future.

As queries in retrieval-augmented generation (RAG) pipelines powered by large language models (LLMs) become increasingly complex and diverse, dense retrieval models have demonstrated strong performance in semantic matching. Nevertheless, they often struggle with fine-grained retrieval tasks, where precise keyword alignment and span-level localization are required, even in cases with high lexical overlap that would intuitively suggest easier retrieval. To systematically evaluate this limitation, we introduce two targeted tasks, keyword retrieval and part-of-passage retrieval, designed to simulate practical fine-grained scenarios. Motivated by these observations, we propose LexSemBridge, a unified framework that enhances dense query representations through fine-grained, input-aware vector modulation. LexSemBridge constructs latent enhancement vectors from input tokens using three paradigms: Statistical (SLR), Learned (LLR), and Contextual (CLR), and integrates them with dense embeddings via element-wise interaction. Theoretically, we show that this modulation preserves the semantic direction while selectively amplifying discriminative dimensions. LexSemBridge operates as a plug-in without modifying the backbone encoder and naturally extends to both text and vision modalities. Extensive experiments across semantic and fine-grained retrieval tasks validate the effectiveness and generality of our approach. All code and models are publicly available at https://github.com/Jasaxion/LexSemBridge/

Jingheng Ye, Shen Wang, Deqing Zou, Yibo Yan, Kun Wang, Hai-Tao Zheng, Ruitong Liu, Zenglin Xu, Irwin King, Philip S. Yu, Qingsong Wen. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

The rapid development of multimodal large language models (MLLMs) has brought significant improvements to a wide range of tasks in realworld applications. However, LLMs still exhibit certain limitations in extracting implicit semantic information. In this paper, we applies MLLMs to the Multi-modal Entity Set Expansion (MESE) task, which aims to expand a handful of seed entities with new entities belonging to the same semantic class, and multi-modal information is provided with each entity. We explore the capabilities of MLLMs to understand implicit semantic information at the entity-level granularity through the MESE task, introducing a listwise ranking method LUSAR that maps local scores to global rankings. Our LUSAR demonstrates significant improvements in MLLM’s performance on the MESE task, marking the first use of generative MLLM for ESE tasks and extending the applicability of listwise ranking.

Visual storytelling, a complex task in natural language generation, aims to create coherent and engaging narratives from a sequence of images, requiring more intricate and lengthy descriptions than typical image captioning. Current methods generally employ sophisticated modal interaction modules and require substantial additional data for training. This paper introduces two innovative techniques to advance visual storytelling capabilities. First, we propose a method to distill descriptive word embeddings from model parameters as an extension to the model’s pre-trained embeddings. This approach focuses the training process on these descriptive words, allowing for efficient fine-tuning and eliminates the need for additional visual modules and reduces computational overhead. Second, we develop a dynamic decoding strategy that adapts the text generation process based on the characteristic of the image features. This ensures a improved alignment with image features and the generated stories. Our experiments on the VIST dataset demonstrate that these methods achieve an impressive performance. Ablation studies show significant improvements in training efficiency and narrative quality compared to existing approaches.

Chinese grammatical error correction (CGEC) aims to detect and correct errors in the input Chinese sentences. Recently, Pre-trained Language Models (PLMS) have been employed to improve the performance. However, current approaches ignore that correction difficulty varies across different instances and treat these samples equally, enhancing the challenge of model learning. To address this problem, we propose a multi-granularity Curriculum Learning (CL) framework. Specifically, we first calculate the correction difficulty of these samples and feed them into the model from easy to hard batch by batch. Then Instance-Level CL is employed to help the model optimize in the appropriate direction automatically by regulating the loss function. Extensive experimental results and comprehensive analyses of various datasets prove the effectiveness of our method.

Entity Set Expansion (ESE) aims to identify new entities belonging to the same semantic class as the given set of seed entities. Traditional methods solely relied on positive seed entities to represent the target fine-grained semantic class, rendering them tough to represent ultra-fine-grained semantic classes. Specifically, merely relying on positive seed entities leads to two inherent shortcomings: (i) Ambiguity among ultra-fine-grained semantic classes. (ii) Inability to define “unwanted” semantics. Hence, previous ESE methods struggle to address the ultra-fine-grained ESE (Ultra-ESE) task. To solve this issue, we first introduce negative seed entities in the inputs, which jointly describe the ultra-fine-grained semantic class with positive seed entities. Negative seed entities eliminate the semantic ambiguity by providing a contrast between positive and negative attributes. Meanwhile, it provides a straightforward way to express “unwanted”. To assess model performance in Ultra-ESE and facilitate further research, we also constructed UltraWiki, the first large-scale dataset tailored for Ultra-ESE. UltraWiki encompasses 50,973 entities and 394,097 sentences, alongside 236 ultra-fine-grained semantic classes, where each class is represented with 3–5 positive and negative seed entities. Moreover, a retrieval-based framework RetExpan and a generation-based framework GenExpan are proposed to provide powerful baselines for Ultra-ESE. Additionally, we devised two strategies to enhance models' comprehension of ultra-fine-grained entities' semantics: contrastive learning and chain-of-thought reasoning. Extensive experiments confirm the effectiveness of our proposed strategies and also reveal that there remains a large space for improvement in Ultra-ESE. All the codes, dataset, and supplementary notes are available at https://github.com/THUKElab/UltraWiki.

How to alleviate the hallucinations of Large Language Models (LLMs) has always been the fundamental goal pursued by the LLMs research community. Looking through numerous hallucination-related studies, a mainstream category of methods is to reduce hallucinations by optimizing the knowledge representation of LLMs to change their output. Considering that the core focus of these works is the knowledge acquired by models, and knowledge has long been a central theme in human societal progress, we believe that the process of models refining knowledge can greatly benefit from the way humans learn. In our work, by imitating the human learning process, we design an Adaptive Contrastive Learning strategy. Our method flexibly constructs different positive and negative samples for contrastive learning based on LLMs' actual mastery of knowledge. This strategy helps LLMs consolidate the correct knowledge they already possess, deepen their understanding of the correct knowledge they have encountered but not fully grasped, forget the incorrect knowledge they previously learned, and honestly acknowledge the knowledge they lack. Extensive experiments and detailed analyses on widely used datasets demonstrate the effectiveness and competitiveness of our method.

Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the “hallucination” issue inherent in multimodal large language models (MLLMs). Although promising, existing heuristic mRAGs typically predefined fixed retrieval processes, which causes two issues: (1) Non-adaptive Retrieval Queries. (2) Overloaded Retrieval Queries. However, these flaws cannot be adequately reflected by current knowledge-seeking visual question answering (VQA) datasets, since the most required knowledge can be readily obtained with a standard two-step retrieval. To bridge the dataset gap, we first construct Dyn-VQA dataset, consisting of three types of ``dynamic'' questions, which require complex knowledge retrieval strategies variable in query, tool, and time: (1) Questions with rapidly changing answers. (2) Questions requiring multi-modal knowledge. (3) Multi-hop questions. Experiments on Dyn-VQA reveal that existing heuristic mRAGs struggle to provide sufficient and precisely relevant knowledge for dynamic questions due to their rigid retrieval processes. Hence, we further propose the first self-adaptive planning agent for multimodal retrieval, **OmniSearch**. The underlying idea is to emulate the human behavior in question solution which dynamically decomposes complex multimodal questions into sub-question chains with retrieval action. Extensive experiments prove the effectiveness of our OmniSearch, also provide direction for advancing mRAG. Code and dataset will be open-sourced.

Leveraging mathematical Large Language Models (LLMs) for proof generation is a fundamental topic in LLMs research. We argue that the ability of current LLMs to prove statements largely depends on whether they have encountered the relevant proof process during training. This reliance limits their deeper understanding of mathematical theorems and related concepts. Inspired by the pedagogical method of "proof by counterexamples" commonly used in human mathematics education, our work aims to enhance LLMs’ ability to conduct mathematical reasoning and proof through counterexamples. Specifically, we manually create a high-quality, university-level mathematical benchmark, COUNTERMATH, which requires LLMs to prove mathematical statements by providing counterexamples, thereby assessing their grasp of mathematical concepts. Additionally, we develop a data engineering framework to automatically obtain training data for further model improvement. Extensive experiments and detailed analyses demonstrate that COUNTERMATH is challenging, indicating that LLMs, such as OpenAI o1, have insufficient counterexample-driven proof capabilities. Moreover, our exploration into model training reveals that strengthening LLMs’ counterexample-driven conceptual reasoning abilities is crucial for improving their overall mathematical capabilities. We believe that our work offers new perspectives on the community of mathematical LLMs.

Large Language Models (LLMs) have demonstrated impressive capabilities in many tasks such as code generation and automated program repair. However, code LLMs have ignored another important task in programmers’ daily development work, which is to improve the maintainability, readability, and scalability of the program. All of these characteristics are related to code smells and we study how to improve them by detecting and removing code smells. Most works on code smells still rely on using measures formulated by experts as features, but lack of use of the rich prior knowledge contained in code LLMs. In this paper, we propose SmellDetector, a comprehensive model for both code smell detection and refactoring opportunities detection in Java. We train the model with the designed prompt which contains both code smells of class-level and method-level in the same code snippet, including more than 20 types. We achieve state-of-the-art performance on the code smell detection task and change the basic paradigm of code smell detection from binary classification problem to multi-label classification. Finally, it has been verified through experiments that good code smell detection helps to detect refactoring opportunities.

In software engineering, code smell detection has always been an important research task because it affects the readability and maintainability of programs. Traditional code smell detection work mostly focuses on a single file, and the study of repo-level code smell caused by the interaction of multiple files is still relatively lacking, although it is more practical in reality. Although large language models (LLMs) have recently achieved remarkable success in the field of code generation, simply copying the fine-tuning method based on LLM is not good enough in this task because it is difficult to model the logical relationship between multiple files. In this paper, we propose a repo-level code smell detection model(RSD), which divides the levels according to the distance between the rest class and the problem class, uses cross attention and CNN to model global and local information, and uses LLM to generate teacher vectors in the same scenario for assistance. Finally, the detection performance surpasses the fine-tuning baseline method based on multiple code LLMs and has achieved the state-of-the-art. Finally, we also contribute the BenchMark used in this article to help researchers in the repo-level smell detection task.

In retrieval-augmented generation (RAG) systems, Large Language Models (LLMs) have been shown to be effective for re-ranking. However, existing research often prioritizes passage relevance over reliability, which can result in the incorporation of conflicting information and the generation of ambiguous responses. This issue becomes particularly pronounced when addressing inter-context knowledge conflicts, where candidate documents present contradictory information that may mislead the model. To mitigate this problem, we propose a novel cross-validation re-ranking technique designed specifically to resolve inter-context knowledge conflicts during the retrieval process. We also develop a new dataset, ContraPRT, to evaluate the ability of models to rank passages containing conflicting knowledge. Experimental results using GPT-4 and LlaMA3-70B demonstrate that our approach not only effectively filters out conflicting information but also ensures accurate passage rankings, thereby providing reliable supplementary knowledge for the generation module.

Sequential recommendation (SR) aims to capture dynamic user preferences from users' historical behaviors. Recently, benefiting from astonishing understanding ability of pre-trained language models (PLMs), text-enhanced sequential recommender becomes a promising direction, which employs PLMs to extract semantic information for user/item representation. Although promising in improving performance and transferability, few existing text-enhanced SR studies have analyzed the differences between PLMs and recommenders, restricting the ability of PLMs for recommendation. In this paper, we make an in-depth comparison and conclude their discrepancies in representation and knowledge level, respectively, caused by different multi-modal content and task-oriented capabilities. Based on this, we propose a Recommendation-Oriented Language Model Adaptation framework (named ROMA) using multi-modal multi-domain item sequences. To empower PLMs with a rational understanding of user/item modeling and the recommendation task, ROMA partitions a PLM into bottom and top layers, respectively, allowing representation-level and task-level adaptation with elaborately designed architectures, transferring strategy and learning framework. Our experimental results on public benchmarks demonstrate the effectiveness and transferability of our framework. Additionally, we showcase the application value of ROMA on the recommender system of Huawei's AppGallery through online A/B testing, which shows significant improvements in online metrics.

The quadratic complexity of self-attention limits Large Language Models (LLMs) in processing long contexts, a capability vital for many advanced applications. Context compression aims to mitigate this computational barrier while preserving essential semantic information. However, existing methods often falter: explicit methods can sacrifice local detail, while implicit ones may exhibit positional biases, struggle with information degradation, or fail to capture long-range semantic dependencies. We introduce AdmTree, a novel framework for adaptive, hierarchical context compression designed with a core focus on maintaining high semantic fidelity while keep efficiency. AdmTree dynamically segments input based on information density, employing gist tokens to summarize variable-length segments as leaves in a semantic binary tree. This structure, combined with a lightweight aggregation mechanism and a frozen backbone LLM (minimizing new trainable parameters), enables efficient hierarchical abstraction of the context. By effectively preserving fine-grained details alongside global semantic coherence, mitigating position bias, and adapting dynamically to content, AdmTree comprehensively preserves the semantic information of lengthy context.

In dense retrieval, embedding long texts into dense vectors can result in information loss, leading to inaccurate query-text matching. Additionally, low-quality texts with excessive noise or sparse key information are unlikely to align well with relevant queries. Recent studies mainly focus on improving the sentence embedding model or retrieval process. In this work, we introduce a novel text augmentation framework for dense retrieval. This framework transforms raw documents into information-dense text formats, which supplement the original texts to effectively address the aforementioned issues without modifying embedding or retrieval methodologies. Two text representations are generated via large language models (LLMs) zero-shot prompting: question-answer pairs and element-driven events. We term this approach QAEA-DR: unifying question-answer generation and event extraction in a text augmentation framework for dense retrieval. To further enhance the quality of generated texts, a scoring-based evaluation and regeneration mechanism is introduced in LLM prompting. Our QAEA-DR model has a positive impact on dense retrieval, supported by both theoretical analysis and empirical experiments.

The Entity Set Expansion (ESE) task aims to expand a handful of seed entities with new entities belonging to the same semantic class. Conventional ESE methods are based on mono-modality (i.e., literal modality), which struggle to deal with complex entities in the real world such as (1) Negative entities with fine-grained semantic differences. (2) Synonymous entities. (3) Polysemous entities. (4) Long-tailed entities. These challenges prompt us to propose novel Multi-modal Entity Set Expansion (MESE), where models integrate information from multiple modalities to represent entities. Intuitively, the benefits of multi-modal information for ESE are threefold: (1) Different modalities can provide complementary information. (2) Multi-modal information provides a unified signal via common visual properties for the same semantic class or entity. (3) Multi-modal information offers robust alignment signals for synonymous entities. To assess model performance in MESE, we constructed the MESED dataset which is the first multi-modal dataset for ESE with large-scale and elaborate manual calibration. A powerful multi-modal model MultiExpan is proposed which is pre-trained on four multimodal pre-training tasks. The extensive experiments and analyses on MESED demonstrate the high quality of the dataset and the effectiveness of our MultiExpan, as well as pointing the direction for future research. The benchmark and code are public at https://github.com/THUKElab/MESED.

Recently, instruction-following Large Language Models (LLMs) , represented by ChatGPT, have exhibited exceptional performance in general Natural Language Processing (NLP) tasks. However, the unique characteristics of E-commerce data pose significant challenges to general LLMs. An LLM tailored specifically for E-commerce scenarios, possessing robust cross-dataset/task generalization capabilities, is a pressing necessity. To solve this issue, in this work, we proposed the first E-commerce instruction dataset EcomInstruct, with a total of 2.5 million instruction data. EcomInstruct scales up the data size and task diversity by constructing atomic tasks with E-commerce basic data types, such as product information, user reviews. Atomic tasks are defined as intermediate tasks implicitly involved in solving a final task, which we also call Chain-of-Task tasks. We developed EcomGPT with different parameter scales by training the backbone model BLOOMZ with the EcomInstruct. Benefiting from the fundamental semantic understanding capabilities acquired from the Chain-of-Task tasks, EcomGPT exhibits excellent zero-shot generalization capabilities. Extensive experiments and human evaluations demonstrate that EcomGPT outperforms ChatGPT in term of cross-dataset/task generalization on E-commerce tasks. The EcomGPT will be public at https://github.com/Alibaba-NLP/EcomGPT.

Large language models (LLMs) have shown impressive abilities for open-domain NLP tasks. However, LLMs are sometimes too footloose for natural language understanding (NLU) tasks which always have restricted output and input format. Their performances on NLU tasks are highly related to prompts or demonstrations and are shown to be poor at performing several representative NLU tasks, such as event extraction and entity typing. To this end, we present SeqGPT, a bilingual (i.e., English and Chinese) open-source autoregressive model specially enhanced for open-domain natural language understanding. We express all NLU tasks with two atomic tasks, which define fixed instructions to restrict the input and output format but still ``open'' for arbitrarily varied label sets. The model is first instruction-tuned with extremely fine-grained labeled data synthesized by ChatGPT and then further fine-tuned by 233 different atomic tasks from 152 datasets across various domains. The experimental results show that SeqGPT has decent classification and extraction ability, and is capable of performing language understanding tasks on unseen domains. We also conduct empirical studies on the scaling of data and model size as well as on the transfer across tasks. Our models are accessible at https://github.com/Alibaba-NLP/SeqGPT.

Yinghui Li, Zishan Xu, Shaoshen Chen, Haojing Huang, Yangning Li, Shirong Ma, Yong Jiang, Zhongli Li, Qingyu Zhou, Hai-Tao Zheng, Ying Shen. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Recent works show that text-to-image generative models are surprisingly vulnerable to a variety of poisoning attacks. Empirical results find that these models can be corrupted by altering associations between individual text prompts and associated visual features. Furthermore, a number of concurrent poisoning attacks can induce "model implosion," where the model becomes unable to produce meaningful images for unpoisoned prompts. These intriguing findings highlight the absence of an intuitive framework to understand poisoning attacks on these models. In this work, we establish the first analytical framework on robustness of image generative models to poisoning attacks, by modeling and analyzing the behavior of the cross-attention mechanism in latent diffusion models. We model cross-attention training as an abstract problem of "supervised graph alignment" and formally quantify the impact of training data by the hardness of alignment, measured by an Alignment Difficulty (AD) metric. The higher the AD, the harder the alignment. We prove that AD increases with the number of individual prompts (or concepts) poisoned. As AD grows, the alignment task becomes increasingly difficult, yielding highly distorted outcomes that frequently map meaningful text prompts to undefined or meaningless visual representations. As a result, the generative model implodes and outputs random, incoherent images at large. We validate our analytical framework through extensive experiments, and we confirm and explain the unexpected (and unexplained) effect of model implosion while producing new, unforeseen insights. Our work provides a useful tool for studying poisoning attacks against diffusion models and their defenses.

Artificial intelligence has been applied in various aspects of online education to facilitate teaching and learning. However, few approaches have been made towards a complete AI-powered tutoring system. In this work, we explore the development of a full-fledged intelligent tutoring system based on large language models (LLMs). The proposed system ChatTutor, powered by state-of-the-art LLMs, is equipped with automatic course planning and adjusting, informative instruction, and adaptive quiz offering and evaluation. ChatTutor is decomposed into three inter-connected core processes: interaction, reflection, and reaction. Each process is implemented by chaining LLM-powered tools along with dynamically updated memory modules. To demonstrate the mechanism of each working module and the benefits of structured memory control and adaptive reflection, we conduct a wide range of analysis based on statistical results and user study. The analysis shows the designed processes boost system consistency and stability under long-term interaction and intentional disruptions, with up to 5% and 20% increase in performance respectively. Meanwhile, we also compare the system with scripts from real-world online learning platform and discuss the potential issues unique to LLM-based systems.

Learning high-quality item embeddings is crucial for recommendation tasks such as matching and ranking. However, existing methods often rely on ID-based item embeddings learned end-to-end with downstream recommendation models, which may suffer from overfitting and limited generalizability. In this paper, we aim to learn universal item embeddings (dubbed UniEmbedding) that capture multi-modal semantics, generalize across multiple domains, and serve different downstream tasks. To achieve this goal, we introduce the UniEmbedding pretraining framework, which includes three modules: a domain-aware multi-modal adapter, a user-view projection module, and contrastive learning objectives across domains. Compared to naive ID embeddings, UniEmbedding provides rich semantic information that generalizes more effectively across domains. Unlike multi-modal embeddings directly extracted from off-the-shelf pretrained models, UniEmbedding achieves better alignment between content semantics and behaviors. We evaluated UniEmbedding on both public and industrial datasets, demonstrating its effectiveness in matching and ranking tasks. Furthermore, UniEmbedding has been deployed in multiple recommendation applications at Huawei, resulting in significant gains in user engagement metrics.

Entity Set Expansion (ESE) is a critical task aiming at expanding entities of the target semantic class described by seed entities. Most existing ESE methods are retrieval-based frameworks that need to extract contextual features of entities and calculate the similarity between seed entities and candidate entities. To achieve the two purposes, they iteratively traverse the corpus and the entity vocabulary, resulting in poor efficiency and scalability. Experimental results indicate that the time consumed by the retrieval-based ESE methods increases linearly with entity vocabulary and corpus size. In this paper, we firstly propose Generative Entity Set Expansion (GenExpan) framework, which utilizes a generative pre-trained auto-regressive language model to accomplish ESE task. Specifically, a prefix tree is employed to guarantee the validity of entity generation, and automatically generated class names are adopted to guide the model to generate target entities. Moreover, we propose Knowledge Calibration and Generative Ranking to further bridge the gap between generic knowledge of the language model and the goal of ESE task. For efficiency, expansion time consumed by GenExpan is independent of entity vocabulary and corpus size, and GenExpan achieves an average 600% speedup compared to strong baselines. For expansion effectiveness, our framework outperforms previous state-of-the-art ESE methods.

Continual learning is an emerging area of machine learning that deals with the issue where models adapt well to the latest data but lose the ability to remember past data due to changes in the data source. A widely adopted solution is by keeping a small memory of previous learned data that use replay. Most of the previous studies on continual learning focused on classification tasks, such as image classification and text classification, where the model needs only to categorize the input data. Inspired by the human ability to incrementally learn knowledge and solve different problems using learned knowledge, we considered a more pratical scenario, knowledge based quesiton answering about continual learning. In this scenario, each single question is different from others(means different fact trippes to answer them) while classification tasks only need to find feature boundaries of different categories, which are the curves or surfaces that separate different categories in the feature space. To address this issue, we proposed a depth aware hierarchical replay framework which include a tree structure classfier to have a sense of knowledge distribution and fill the gap between text classfication tasks and question-answering tasks for continual learning, a local sampler to grasp these critical samples and a depth aware learning network to reconstructe the feature space of a single learning round. In our experiments, we have demonstrated that our proposed model outperforms previous continual learning methods in mitigating the issue of catastrophic forgetting.

With the evolution of LLMs, they are endowed with impressive logical reasoning, or vertical thinking capabilities. But can they think out of the box? Do they possess proficient lateral thinking abilities? Following the setup of Lateral Thinking Puzzles, we propose a novel evaluation benchmark, LatEval, which assesses the model’s lateral thinking within an interactive framework. In our benchmark, we challenge LLMs with 2 aspects: (1) posing high-quality questions that break out of conventional norms but are beneficial for puzzle-solving. (2) integrating existing information to gradually deduce the truth through reasoning. We observe that it is hard for most LLMs to accomplish lateral thinking during interactions. Even the most powerful LLM, GPT-4, faces challenges in achieving satisfactory performance, and for most open-source models, simply completing this task is quite difficult. This evaluation benchmark provides LLMs with a highly challenging and differentiating task that is crucial to an effective AI assistant. Our dataset and source codes are available at https://github.com/THUKElab/LatEval.

Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in multimodal understanding, reasoning, and interaction. However, existing MLLMs prevalently suffer from serious hallucination problems, generating text that is not factually grounded in associated images. The problem makes existing MLLMs untrustworthy and thus impractical in real-world (especially high-stakes) applications. To address the challenge, we present RLHF-V, which enhances MLLM trustworthiness via behavior alignment from fine-grained correctional human feedback. Specifically, RLHF-V collects human preference in the form of segment-level corrections on hallucinations, and performs dense direct preference optimization over the human feedback. Comprehensive experiments on five benchmarks in both automatic and human evaluation show that, RLHF-V can enable substantially more trustworthy MLLM behaviors with promising data and computation efficiency. Remarkably, using 1.4k annotated data samples, RLHF-V significantly reduces the hallucination rate of the base MLLM by 34.8%, outperforming the concurrent LLaVA-RLHF trained on 10k annotated data. The final model achieves state-of-the-art performance in trustwor-thiness among open-source MLLMs, and shows better ro-bustness than GPT-4V in preventing hallucinations aroused from over-generalization.

Noise is a pervasive issue in recommendation systems, which can stem from user behaviors that do not align with their intentions. As a result, noise reduction has become a prominent area of research in the field of recommendation systems. However, existing noise reduction techniques in recommendation tend to compromise the performance of certain task objectives. Moreover, they require modifying the structure of the model, which introduces inference latency and additional space cost. In this paper, we propose a straightforward yet powerful approach, Multi-layer Tensor trace Norm (MTN), to address noise-related challenges. Our method achieves this by promoting information sharing across different tasks using tensor trace norms. By leveraging norms, MTN effectively reduces noise without modifying the model’s structure or incurring substantial time and space complexities. Extensive experiments on public datasets and generated noisy datasets demonstrate the effectiveness of MTN on several of the most popular multi-task models.

Citation field learning is to segment a citation string into fields of interest such as author, title, and venue. Extracting such fields from citations is crucial for citation indexing, researcher profile analysis, etc. User-generated resources like academic homepages and Curriculum Vitae, provide rich citation field information. However, extracting fields from these resources is challenging due to inconsistent citation styles, incomplete sentence syntax, and insufficient training data. To address these challenges, we propose a novel algorithm, CIFAL (citation field learning by anchor learning), to boost the citation field learning performance. CIFAL leverages the anchor learning, which is model-agnostic for any Pre-trained Language Model, to help capture citation patterns from the data of different citation styles. The experiments demonstrate that CIFAL outperforms state-of-the-art methods in citation field learning, achieving a 2.68% improvement in field-level F1-scores. Extensive analysis of the results further confirms the effectiveness of CIFAL quantitatively and qualitatively.

Meta-learning has achieved promising results in low-resource text classification, which aims to identify target classes by transferring knowledge from source classes through a series of small tasks called episodes. However, the current meta-learning algorithms that solely rely on learning from meta-training tasks may struggle to generalize well to meta-testing tasks. To address this problem, we propose a method called Retrieval-Augmented Meta Learning (RAML) that utilizes external knowledge to compensate for the performance degradation when meta-training tasks do not adequately support meta-testing tasks. RAML first utilizes a retriever to retrieve knowledge relevant to the query from an external corpus, and then employs the Multi-View Passages Fusion Network to integrate the retrieved knowledge for performing few-shot classification. This network can effectively combine the probability distributions of classifications obtained from multiple messages by considering the importance of different messages. Furthermore, inspired by knowledge distillation, we iteratively train the retriever model using the synthetic labels generated by the aforementioned network. Extensive experiments demonstrate that RAML significantly outperforms current state-of-the-art baselines(e.g., ChatGPT).

In this paper, we focus on the challenging and realistic Generalized Few-Shot Intent Detection (GFSID), which requires to categorize both seen and novel intents simultaneously. Moreover, there are only few training samples for novel intents. Generalized few-shot intent detection has to deal with two major challenges: learning novel intents from only few samples and preventing forgetting knowledge of seen intents. To address the dilemma, we propose to convert the GFSID task into the class incremental learning paradigm. Specifically, we propose a two-phase learning framework based on prompt learning, which sequentially training the model on the data of seen intents and novel intents. Furthermore, to alleviate the forgetting of knowledge related to seen intents, we introduce prompt-based intra-class relation knowledge distillation. To the best of our knowledge, this is the first study to simultaneously address both aspects in the context of GFSID. Extensive experiments and detailed analyses conducted on two widely used datasets demonstrate that our proposed framework achieves promising performance.

Recently, Large Language Models (LLMs) make remarkable evolutions in language understanding and generation. Following this, various benchmarks for measuring all kinds of capabilities of LLMs have sprung up. In this paper, we challenge the reasoning and understanding abilities of LLMs by proposing a FaLlacy Understanding Benchmark (FLUB) containing cunning texts that are easy for humans to understand but difficult for models to grasp. Specifically, the cunning texts that FLUB focuses on mainly consist of the tricky, humorous, and misleading texts collected from the real internet environment. And we design three tasks with increasing difficulty in the FLUB benchmark to evaluate the fallacy understanding ability of LLMs. Based on FLUB, we investigate the performance of multiple representative and advanced LLMs, reflecting our FLUB is challenging and worthy of more future study. Interesting discoveries and valuable insights are achieved in our extensive experiments and detailed analyses. We hope that our benchmark can encourage the community to improve LLMs' ability to understand fallacies. Our data and codes are available at https://github.com/THUKElab/FLUB.

Multi-label unknown intent detection is a challenging task where each utterance may contain not only multiple known but also unknown intents. To tackle this challenge, pioneers proposed to predict the intent number of the utterance first, then compare it with the results of known intent matching to decide whether the utterence contains unknown intent(s). Though they have made remarkable progress on this task, their methods still suffer from two important issues: (1) It is inadequate to extract multiple intents using only utterance encoding; (2) Optimizing two sub-tasks (intent number prediction and known intent matching) independently leads to inconsistent predictions. In this article, we propose to incorporate segment augmentation rather than only use utterance encoding to better detect multiple intents. We also design a prediction consistency module to bridge the gap between the two sub-tasks. Empirical results on MultiWOZ2.3 and MixSNIPS datasets show that our method achieves state-of-the-art performance and significantly improves the best baseline.

Video-language pre-training for text-based video retrieval tasks is vitally important. Previous pre-training methods suffer from the semantic misalignments. The reason is that these methods ignore sequence alignments but focusing on critical token alignment. To alleviate the problem, we propose a video-language pre-training framework, termed videolanguage pre-training For lEarning sEmantic aLignments (FEEL), to learn semantic alignments at the sequence level. Specifically, the global modality reconstruction and the cross- modal self-contrasting method is utilized to learn the alignments at the sequence level better. Extensive experimental results demonstrate the effectiveness of FEEL on text-based video retrieval and text-based video corpus moment retrieval.

Data augmentation with Mixup has been proven an effective method to regularize the current deep neural networks. Mixup generates virtual samples and corresponding labels simultaneously by linear interpolation. However, the one-stage generation paradigm and the use of linear interpolation have two defects: (1) The label of the generated sample is simply combined from the labels of the original sample pairs without reasonable judgment, resulting in ambiguous labels. (2) Linear combination significantly restricts the sampling space for generating samples. To address these issues, we propose a novel and effective augmentation method, Global Mixup, based on global clustering relationships. Specifically, we transform the previous one-stage augmentation process into two-stage by decoupling the process of generating virtual samples from the labeling. And for the labels of the generated samples, relabeling is performed based on clustering by calculating the global relationships of the generated samples. Furthermore, we are no longer restricted to linear relationships, which allows us to generate more reliable virtual samples in a larger sampling space. Extensive experiments for CNN, LSTM, and BERT on five tasks show that Global Mixup outperforms previous baselines. Further experiments also demonstrate the advantage of Global Mixup in low-resource scenarios.

Large-scale commonsense knowledge bases empower a broad range of AI applications, where the automatic extraction of commonsense knowledge (CKE) is a fundamental and challenging problem. CKE from text is known for suffering from the inherent sparsity and reporting bias of commonsense in text. Visual perception, on the other hand, contains rich commonsense knowledge about real-world entities, e.g., (person, can_hold, bottle), which can serve as promising sources for acquiring grounded commonsense knowledge. In this work, we present CLEVER, which formulates CKE as a distantly supervised multi-instance learning problem, where models learn to summarize commonsense relations from a bag of images about an entity pair without any human annotation on image instances. To address the problem, CLEVER leverages vision-language pre-training models for deep understanding of each image in the bag, and selects informative instances from the bag to summarize commonsense entity relations via a novel contrastive attention mechanism. Comprehensive experimental results in held-out and human evaluation show that CLEVER can extract commonsense knowledge in promising quality, outperforming pre-trained language model-based methods by 3.9 AUC and 6.4 mAUC points. The predicted commonsense scores show strong correlation with human judgment with a 0.78 Spearman coefficient. Moreover, the extracted commonsense can also be grounded into images with reasonable interpretability. The data and codes can be obtained at https://github.com/thunlp/CLEVER.

Yulin Chen, Ning Ding, Xiaobin Wang, Shengding Hu, Haitao Zheng, Zhiyuan Liu, Pengjun Xie. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Multi-label unknown intent detection is a challenging task where each utterance may contain not only multiple known but also unknown intents. To tackle this challenge, pioneers proposed to predict the intent number of the utterance first, then compare it with the results of known intent matching to decide whether the utterance contains unknown intent(s). Though they have made remarkable progress on this task, their method still suffers from two important issues: 1) It is inadequate to extract multiple intents using only utterance encoding; 2) Optimizing two sub-tasks (intent number prediction and known intent matching) independently leads to inconsistent predictions. In this paper, we propose to incorporate segment augmentation rather than only use utterance encoding to better detect multiple intents. We also design a prediction consistency module to bridge the gap between the two sub-tasks. Empirical results on MultiWOZ2.3 show that our method achieves state-of-the-art performance and improves the best baseline significantly.

Session-based Recommendation (SBR) aims to predict the next item for a session, which consists of several clicked items in a transaction. Most SBR approaches follow an underlying assumption that all sequential information should be strictly utilized. Thus, they model temporal information for items using implicit, explicit, or ensemble methods. In fact, users may recall previously clicked items but might not remember the exact order in which they were clicked. Therefore, focusing on representing item temporal information in various ways could make learning session intents challenging. In this paper, we rethink the necessity of temporal information for items in SBR. We propose Aggregating the Contextual intents of the session with Attentive networks, namely ACARec. Specifically, we avoid explicitly modeling positional embeddings and learn contextual intents through aggregation methods (convolutions or poolings). We also demonstrate that even an entirely position-agnostic aggregation approach can yield promising results. Extensive experiments on real-world datasets validate our arguments. We hope our study can provide insights into SBR and inspire future research in the community.

Recently, there has been a surge in the use of generated data to enhance the performance of downstream models, largely due to the advancements in pre-trained language models. However, most prevailing methods trained generative and discriminative models in isolation, which left them unable to adapt to changes in each other. These approaches lead to generative models that are prone to deviating from the true data distribution and providing limited benefits to discriminative models. While some works have proposed jointly training generative and discriminative language models, their methods remain challenging due to the non-differentiable nature of discrete data. To overcome these issues, we introduce a self-consistent learning framework in the text field that involves training a discriminator and generator cooperatively in a closed-loop manner until a scoring consensus is reached. By learning directly from selected samples, our framework are able to mitigate training instabilities such as mode collapse and non-convergence. Extensive experiments on four downstream benchmarks, including AFQMC, CHIP-STS, QQP, and MRPC, demonstrate the efficacy of the proposed framework.

Evaluating the performance of Grammatical Error Correction (GEC) systems is a challenging task due to its subjectivity.Designing an evaluation metric that is as objective as possible is crucial to the development of GEC task.However, mainstream evaluation metrics, i.e., referencebased metrics, introduce bias into the multireference evaluation by extracting edits without considering the presence of multiple references.To overcome this issue, we propose Chunk-LEvel Multi-reference Evaluation (CLEME), designed to evaluate GEC systems in the multireference evaluation setting.CLEME builds chunk sequences with consistent boundaries for the source, the hypothesis and references, thus eliminating the bias caused by inconsistent edit boundaries.Furthermore, we observe the consistent boundary could also act as the boundary of grammatical errors, based on which the F 0.5 score is then computed following the correction independence assumption.We conduct experiments on six English reference sets based on the CoNLL-2014 shared task.Extensive experiments and detailed analyses demonstrate the correctness of our discovery and the effectiveness of CLEME.Further analysis reveals that CLEME is robust to evaluate GEC systems across reference sets with varying numbers of references and annotation styles 1 .

Entity alignment (EA) for knowledge graphs (KGs) plays a critical role in knowledge engineering. Existing EA methods mostly focus on utilizing the graph structures and entity attributes (including literals), but ignore images that are common in modern multi-modal KGs. In this study we first constructed Multi-OpenEA — eight large-scale, image-equipped EA benchmarks, and then evaluated some existing embedding-based methods for utilizing images. In view of the complementary nature of visual modal information and logical deduction, we further developed a new multi-modal EA method named LODEME using logical deduction and multi-modal KG embedding, with state-of-the-art performance achieved on Multi-OpenEA and other existing multi-modal EA benchmarks.

Structured prediction models aim at solving tasks where the output is a complex structure, rather than a single variable. Performing knowledge distillation for such problems is non- trivial due to their exponentially large output space. Previous works address this problem by developing particular distillation strategies (e.g., dynamic programming) that are both complicated and of low run-time efficiency. In this work, we propose an approach that is much simpler in its formulation, far more efficient for training than existing methods, and even performs better than our baselines. Specifically, we transfer the knowledge from a teacher model to its student by locally matching their computations on all internal structures rather than the final outputs. In this manner, we avoid time-consuming techniques like Monte Carlo Sampling for decoding output structures, permitting parallel computation and efficient training. Besides, we show that it encourages the student model to better mimic the internal behavior of the teacher model. Experiments on two structured prediction tasks demonstrate that our approach not only halves the time cost, but also outperforms previous methods on two widely adopted benchmark datasets.1 2

The task of Points of Interest (POI) description generation aims to generate an objective and informative description for a given POI based on POI-related information. High-quality descriptions can better guide users and improve the performance of POI-related recommendation systems. A practical POI description generation model should have effective multimodal fusion and information encoding methods suitable for various data forms. However, due to model structure and data utilization limitations, the previous method is challenging to meet the above requirements. We propose a novel Guide-Select multimodal fusion method that combines the guiding and selecting process to fuse various POI-related information efficiently. In addition, we propose a reasonable review encoding method and a category encoding method that has strong generalization ability. We integrate these methods into our Guide-Select Generation Model (GSGM). Experimental results demonstrate that our model significantly outperforms the state-of-the-art model while having a strong generalization ability on category information.

The session-based social recommendation task aims to leverage knowledge from social networks to predict user actions based on their sessions. Most previous studies pay more attention to complex transitions to get item embedding in various ways and neglect the importance of users’ role in social network. Therefore, we design a High-level Feature Fusion Network to address these issues. Firstly, to better leverage the knowledge from social networks, we use a heterogeneous graph neural network to enhance the user/item representation. Secondly, a user-based graph attention network is adapted to learn the user’s deep interest evolution process. Next, user and session features are transferred to our feature fusion module to generate fusion features, and the original and fusion features are combined to make recommendations. Extensive experiments on three public datasets show that the proposed model outperforms existing state-of-the-art models.

Existing approaches to session-based recommendation are focused on using advanced deep neural networks (DNNs). Recent studies have shown that some traditional methods can outperform some DNN-based models. However, in recent years, few studies have tried to build traditional models. In this paper, we investigate this issue and propose a concise rule-based method for session-based recommendation. Specifically, we make item adjacent and N-gram-based rules to extract frequenting, sequential and other patterns of the limited historical information to construct item correlation dictionaries. Then, we exploit these dictionaries at the inference stage by constructing candidate item generation and item set fusion rules to acquire candidate items. By this means, we can leverage short-term user-item interaction records to generate candidate items for anonymous sessions. Finally, we sort the candidate items to make a recommendation. Extensive experimental results on three real public datasets show that our method can significantly outperform existing traditional methods and even outperforms several representative DNN-based models. The model just costs a few seconds to train and inference, and occupies less memory space than that. Compared with the previous models, it has a huge lead in time and space.

Chinese Spell Checking (CSC) task aims to detect and correct Chinese spelling errors. Recently, related researches focus on introducing character similarity from confusion set to enhance the CSC models, ignoring the context of characters that contain richer information. To make better use of contextual information, we propose a simple yet effective Curriculum Learning (CL) framework for the CSC task. With the help of our model-agnostic CL framework, existing CSC models will be trained from easy to difficult as humans learn Chinese characters and achieve further performance improvements. Extensive experiments and detailed analyses on widely used SIGHAN datasets show that our method outperforms previous state-of-the-art methods. More instructively, our study empirically suggests that contextual similarity is more valuable than character similarity for the CSC task.

Recent research has reported a performance degradation in self-supervised contrastive learning for specially designed efficient networks, such as MobileNet and EfficientNet. A common practice to address this problem is to introduce a pretrained contrastive teacher model and train the lightweight networks with distillation signals generated by the teacher. However, it is time and resource consuming to pretrain a teacher model when it is not available. In this work, we aim to establish a stronger baseline for lightweight contrastive models without using a pretrained teacher model. Specifically, we show that the optimal recipe for efficient models is different from that of larger models, and using the same training settings as ResNet50, as previous research does, is inappropriate. Additionally, we observe a common issue in contrastive learning where either the positive or negative views can be noisy, and propose a smoothed version of InfoNCE loss to alleviate this problem. As a result, we successfully improve the linear evaluation results from 36.3% to 62.3% for MobileNet-V3-Large and from 42.2% to 65.8% for EfficientNet-B0 on ImageNet, closing the accuracy gap to ResNet50 with 5× fewer parameters. We hope our research will facilitate the usage of lightweight contrastive models.1

In recent years, Transformer-based abstractive summarization models have been proven to be effective and widely used in multiple domains. However, most of these Transformer-based methods are based on maximum likelihood estimation and they still focus on token-level optimization. In addition, the generated summary should have the same semantics as the gold summary. Therefore, we propose a new abstractive summarization model in this paper. Specifically, we optimize the traditional model from the semantic perspective, so that the semantics of the generated summary is more similar to that of the gold summary. We also add topic information so that more key information in the text can be retained in the model. We prove in the CNN/DailyMail dataset that the method proposed in this paper greatly improve the classic abstractive model based on BART, and achieve SOTA results in the SAMSUM dataset.

Search result diversification aims to balance the relevance and diversity of retrieved documents to satisfy the different information needs of users. Three types of approaches have proliferated: explicit models that are based on explicit features (e.g., subtopic coverage), implicit models that are based on implicit features (e.g., the novelty of documents), and ensemble models that utilize both implicit and explicit features. However, the subtopics used by most explicit and ensemble models are usu-ally mined from queries (e.g., using Google Search Suggestions), which may not match the subtopics covered by the candidate documents. Besides, the implicit features used by most implicit models are formulated as either the similarity of documents or the intent of documents. The former cannot directly reflect the relationships of documents at the subtopic level, while the latter cannot capture non-pairwise relationships among documents. To tackle these issues, we propose a novel model that dynamically mines subtopics from the candidate documents and leverages thehypergraph structure to model the diversity of candidate documents, named HGDIV. Specifically, we dynamically mine subtopics from the candidate documents, rather than mining subtopics from queries or using static subtopics as existing methods do. More importantly, we introduce the hypergraph structure to model the diversity of candidate documents for search result diversification, which can capture the non-pairwise relationships among documents. Furthermore, we innovatively model the global and local subtopic distributions to extract the diversity of candidate documents. Experimental results on the public diversity benchmark TREC datasets demonstrate the superiority of our model over state-of-the-art models.

With the promotion of the pre-training paradigm, researchers are increasingly focusing on injecting external knowledge, such as entities and triplets from knowledge graphs, into pre-trained language models (PTMs) to improve their understanding and logical reasoning abilities. This results in significant improvements in natural language understanding and generation tasks and some level of interpretability. In this paper, we propose a novel two-stage entity knowledge enhancement pipeline for Chinese pre-trained models based on “bidirectional” prompt tuning. The pipeline consists of a “forward” stage, in which we construct fine-grained entity type prompt templates to boost PTMs injected with entity knowledge, and a “backward” stage, where the trained templates are used to generate type-constrained context-dependent negative samples for contrastive learning. Experiments on six classification tasks in the Chinese Language Understanding Evaluation (CLUE) benchmark demonstrate that our approach significantly improves upon the baseline results in most datasets, particularly those that have a strong reliance on diverse and extensive knowledge.

The goal of sequential recommendation (SR) is to predict a user's potential interested items based on her/his historical interaction sequences. Most existing sequential recommenders are developed based on ID features, which, despite their widespread use, often underperform with sparse IDs and struggle with the cold-start problem. Besides, inconsistent ID mappings hinder the model's transferability, isolating similar recommendation domains that could have been co-optimized. This paper aims to address these issues by exploring the potential of multi-modal information in learning robust and generalizable sequence representations. We propose MISSRec, a multi-modal pre-training and transfer learning framework for SR. On the user side, we design a Transformer-based encoder-decoder model, where the contextual encoder learns to capture the sequence-level multi-modal synergy while a novel interest-aware decoder is developed to grasp item-modality-interest relations for better sequence representation. On the candidate item side, we adopt a dynamic fusion module to produce user-adaptive item representation, providing more precise matching between users and items. We pre-train the model with contrastive learning objectives and fine-tune it in an efficient manner. Extensive experiments demonstrate the effectiveness and flexibility of MISSRec, promising an practical solution for real-world recommendation scenarios.

Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance. Their success has been recently expanded to text generation via generating all tokens within a sequence concurrently. However, natural language exhibits a far more pronounced sequential dependency in comparison to images, and the majority of existing language models are trained with a left-to-right auto-regressive approach. To account for the inherent sequential characteristic of natural language, we introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that the generation of tokens on the right depends on the generated ones on the left, a mechanism achieved through employing a dynamic number of denoising steps that vary based on token position. This results in tokens on the left undergoing fewer denoising steps than those on the right, thereby enabling them to generate earlier and subsequently influence the generation of tokens on the right. In a series of experiments on various text generation tasks, including text summarization, machine translation, and common sense generation, AR-Diffusion clearly demonstrated its superiority over existing diffusion language models and that it can be $100\times\sim600\times$ faster when achieving comparable results. Our code is available at https://github.com/microsoft/ProphetNet/tree/master/AR-diffusion.

Pre-trained language models (PLMs) with external knowledge have demonstrated their remarkable performance on a variety of downstream natural language processing tasks.The typical methods of integrating knowledge into PLMs are designing different pre-training tasks and training from scratch, which requires high-end hardware, massive storage resources, and computing time.Prompt learning is an effective approach to tune PLMs for specific tasks, and it can also be used to infuse knowledge.However, most prompt learning methods accept one token as the answer instead of multiple tokens.To tackle this problem, we propose the long-answer prompt learning method (KLAPrompt) to incorporate semantic knowledge in Xinhua Dictionary into pre-trained language models.The proposed method splits the whole answer space into several answer subspaces according to the token's position in the long answer.Extensive experimental results on five datasets demonstrate the effectiveness of our approach.

Session-based Recommendation (SR) aims to predict users' next click based on their behavior within a short period, which is crucial for online platforms. However, most existing SR methods somewhat ignore the fact that user preference is not necessarily strongly related to the order of interactions. Moreover, they ignore the differences in importance between different samples, which limits the model-fitting performance. To tackle these issues, we put forward the method, Mining Interest Trends and Adaptively Assigning Sample Weight, abbreviated as MTAW. Specifically, we model users' instant interest based on their present behavior and all their previous behaviors. Meanwhile, we discriminatively integrate instant interests to capture the changing trend of user interest to make more personalized recommendations. Furthermore, we devise a novel loss function that dynamically weights the samples according to their prediction difficulty in the current epoch. Extensive experimental results on two benchmark datasets demonstrate the effectiveness and superiority of our method.

Student modeling, the task of inferring a student's learning characteristics through their interactions with coursework, is a fundamental issue in intelligent education. Although the recent attempts from knowledge tracing and cognitive diagnosis propose several promising directions for improving the usability and effectiveness of current models, the existing public datasets are still insufficient to meet the need for these potential solutions due to their ignorance of complete exercising contexts, fine-grained concepts, and cognitive labels. In this paper, we present MoocRadar, a fine-grained, multi-aspect knowledge repository consisting of 2,513 exercise questions, 5,600 knowledge concepts, and over 12 million behavioral records. Specifically, we propose a framework to guarantee a high-quality and comprehensive annotation of fine-grained concepts and cognitive labels. The statistical and experimental results indicate that our dataset provides the basis for the future improvements of existing methods. Moreover, to support the convenient usage for researchers, we release a set of tools for data querying, model adaption, and even the extension of our repository, which are now available at https://github.com/THU-KEG/MOOC-Radar.

Entity Set Expansion (ESE) is a valuable task that aims to find entities of the target semantic class described by given seed entities. Various Natural Language Processing (NLP) and Information Retrieval (IR) downstream applications have benefited from ESE due to its ability to discover knowledge. Although existing corpus-based ESE methods have achieved great progress, they still rely on corpora with high-quality entity information annotated, because most of them need to obtain the context patterns through the position of the entity in a sentence. Therefore, the quality of the given corpora and their entity annotation has become the bottleneck that limits the performance of such methods. To overcome this dilemma and make the ESE models free from the dependence on entity annotation, our work aims to explore a new ESE paradigm, namely corpus-independent ESE. Specifically, we devise a context pattern generation module that utilizes autoregressive language models (e.g., GPT-2) to automatically generate high-quality context patterns for entities. In addition, we propose the GAPA, a novel ESE framework that leverages the aforementionedGenerAtedPAtterns to expand target entities. Extensive experiments and detailed analyses on three widely used datasets demonstrate the effectiveness of our method. All the codes of our experiments are available athttps://github.com/geekjuruo/GAPA.

We study neural data-to-text generation to generate a sentence to describe a target entity based on its attributes. Specifically, we address two problems of the encoder-decoder framework for data-to-text generation: i) how to encode a non-linear input (e.g., a set of attributes); and ii) how to order the attributes in the generated description. Existing studies focus on the encoding problem but do not address the ordering problem, i.e., they learn the content-planning implicitly. The other approaches focus on two-stage models but overlook the encoding problem. To address the two problems at once, we propose a model namedTransCPto explicitly learn content-planning and integrate them into a description generation model in an end-to-end fashion. We propose a novel Transformer-based Pointer Network withgated residual attentionandimportance maskingto learn a content-plan. To integrate the content-plan with a description generator, we propose a tracking mechanism to trace the extent to which the content-plan is exposed in the previous decoding time-step. This helps the description generator select the attributes to be mentioned in proper order. Experimental results show that our model consistently outperforms state-of-the-art baselines by up to 2% and 3% in terms of BLEU score on two real-world datasets.

The task of ACE Event Detection (ED) often encounters ambiguous and unseen trigger words. Most conventional ED systems exclusively consider the semantic or syntactic patterns as the additional evidence to resolve the problem of the ambiguous and unseen triggers, but rarely consider taking advantages of structured knowledge of the event itself. In this study, we propose Dynamic Word-Trigger-Argument Graph Neural Networks (DWTA-GNN), a novel framework that leverages event structure knowledge to facilitate the two issues simultaneously. In our approach, we utilize words, entities, and event annotations from training to construct an event background graph, which can provide sufficient information of event structure to better disambiguate polysemous triggers and identify unseen triggers. To make full use of the constructed background graph, we further design a knowledge matching module to dynamically match appropriate event structure knowledge and construct a subgraph for each incoming sentence. Besides, an event-selective graph convolution is applied to filter out the noise in the matched knowledge so as to enhance event representation. Experiments on the ACE2005 dataset show that our model achieves competitive performance and advances previous approaches on ambiguous and unseen trigger words, verifying the effectiveness of incorporating event structure knowledge for event detection.

With the rapid growth of large-scale knowledge bases (KBs), knowledge base question answering (KBQA) has attracted increasing attention recently. Relation detection plays an important role in the KBQA system, which finds a compatible answer by analyzing the semantics of questions and querying and reasoning with multiple KB triples. Significant progress has been made by deep neural networks. However, existing methods often concern on detecting single-hop relation without path reasoning, and a few of these methods exploit the multihop relation reasoning, which involves the answer reasoning from the noisy and abundant relational paths in the KB. Meanwhile, the relatedness between question and answer candidates has received little attention and remains unsolved. This article proposes a novel knowledge-based reasoning network (KRN) for relation detection, including both single-hop relation and multihop relation. To address the semantic gap problem in question-answer interaction, we first learn attentive question representations according to the influence of answer aspects. Then, we learn the single-hop relation sequence through different levels of abstraction and adopt the KB entity and structure information to denoise the multihop relation detection task. Finally, we adopt a Siamese network to measure the similarity between question representation and relation representation for both single-hop and multihop relation KBQA tasks. We conduct experiments on two well-known benchmarks, SimpleQuestions and WebQSP, and the results show the superiority of our approach over the state-of-the-art models for both single-hop and multihop relation detection. Our model also achieves a significant improvement over existing methods on KBQA end task. Further analysis demonstrates the robustness and the applicability of the proposed approach.

Session-based Recommendation (SBR) aims to predict the next item for the current session, which consists of several clicked items in a short period by an anonymous user. Most of the sequential modeling approaches to SBR are focusing on adopting advanced Deep Neural Networks (DNNs), and these methods require increasingly longer training times. Existing studies have shown that some traditional SBR methods can outperform some DNN-based sequential models, however, few studies have attempted to investigate the effectiveness of traditional methods in recent years. In this paper, we propose a novel and concise SBR model inspired by the basic concept of autocorrelation in the Stochastic Process. Autocorrelation measures the correlation of a process at different moments. Therefore, it is natural to use it to model the correlation of clicked item sequences at different time shifts. Specifically, we use Fast Fourier Transforms (FFT) to compute the autocorrelation and combine it with several linear transformations to enhance the session representation. By this means, our proposed method can learn better session preferences and is more efficient than most DNN-based models. Extensive experiments on two public datasets show that the proposed method outperforms state-of-the-art models in both effectiveness and efficiency.

Pre-trained Language Models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding. Various Chinese PLMs have been successively proposed for learning better Chinese language representation. However, most current models use Chinese characters as inputs and are not able to encode semantic information contained in Chinese words. While recent pre-trained models incorporate both words and characters simultaneously, they usually suffer from deficient semantic interactions and fail to capture the semantic relation between words and characters. To address the above issues, we propose a simple yet effective PLM CLOWER, which adopts the Contrastive Learning Over Word and charactER representations. In particular, CLOWER implicitly encodes the coarse-grained information (i.e., words) into the fine-grained representations (i.e., characters) through contrastive learning on multi-grained information. CLOWER is of great value in realistic scenarios since it can be easily incorporated into any existing fine-grained based PLMs without modifying the production pipelines. Extensive experiments conducted on a range of downstream tasks demonstrate the superior performance of CLOWER over several state-of-the-art baselines.

For task-oriented dialogue systems, Natural Language Generation (NLG) is the last and vital step which aims at generating an appropriate response according to the dialogue act (DA). While end-to-end neural networks have achieved promising performances on this task, the existing models still struggle to avoid slot mistakes. To address this challenge, we propose a novel segmented generation approach in this paper. The proposed method operates by progressively generating text for the span between two adjacent keywords (act type and slots) in semantically ordered DA. This procedure is recursively applied from left to right until a response is completed. Besides, a retrieval mechanism is utilized to better match the diversity and fluency in human language. Experimental results on four datasets demonstrate that our model achieves state-of-the-art slot error rate and also gets competitive performance on BLEU score with all strong baselines.

Existing text- and image-based multimodal dialogue systems use the traditional Hierarchical Recurrent Encoder-Decoder (HRED) framework, which has an utterance-level encoder to model utterance representation and a context-level encoder to model context representation. Although pioneer efforts have shown promising performances, they still suffer from the following challenges: (1) the interaction between textual features and visual features is not fine-grained enough. (2) the context representation can not provide a complete representation for the context. To address the issues mentioned above, we propose a non-hierarchical attention network with modality dropout, which abandons the HRED framework and utilizes attention modules to encode each utterance and model the context representation. To evaluate our proposed model, we conduct comprehensive experiments on a public multimodal dialogue dataset. Automatic and human evaluation demonstrate that our proposed model outperforms the existing methods and achieves state-of-the-art performance.

The task of Session-based Social Recommendation (SSR) aims to utilize the social networks to make recommendations in session-based scenarios. Existing SSR methods mainly focused on using graph networks to capture complex item transition patterns, ignoring the sequential information. Few studies combined two aspects of features to enhance session preferences, resulting in information loss. Besides, modeling the entire session that some items are invalid or repeatedly clicked will interfere with the results. In this paper, to address the information loss issue in SSR, we propose a novel Dual-Channel Attentive Network (DCAN) to leverage both sequential infor-mation and complex item transitions. Specifically, we construct one channel by a light graph attention layer to capture item transitions, and we elaborate a concise attention-based layer to build the other channel to learn sequential information. To solve the invalid or repeatedly clicked problem in the session, we introduce new self-supervised learning (SSL) learning method, which allows model learning to distinguish and discard these items. However, the effect of SSL in SSR has not been investigated yet. Besides, these studies require negative sampling, which makes its performance depend on negative sampling strategies. Then, we investigate the effect of adding existing SSL frameworks in DCAN, but it has not achieved good results. Besides, we propose a novel SSL framework that does not require negative sampling for SSR, denoted as Positive sampling SSL (PSSL). Furthermore, we combined DCAN and PSSL to make more accurate recommendations, denoted as DCAN - PSSL. Extensive experiments on three public benchmark datasets demonstrate that both DCAN and DCAN - PSSL consistently outperform the state-of-the-art models.

Graph neural networks have been widely used in recent recommender systems, where negative sampling plays an important role. Existing negative sampling methods restrict the relationship between nodes as either hard positive pairs or hard negative pairs. This leads to the loss of structural information, and lacks the mechanism to generate positive pairs for nodes with few neighbors. To overcome limitations, we propose a novel soft link-based sampling method, namely MixDec Sampling, which consists of Mixup Sampling module and Decay Sampling module. The Mixup Sampling augments node features by synthesizing new nodes and soft links, which provides sufficient number of samples for nodes with few neighbors. The Decay Sampling strengthens the digestion of graph structure information by generating soft links for node embedding learning. To the best of our knowledge, we are the first to model sampling relationships between nodes by soft links in GNN-based recommender systems. Extensive experiments demonstrate that the proposed MixDec Sampling can significantly and consistently improve the recommendation performance of several representative GNN-based models on various recommendation benchmarks.

Although BERT has achieved significant improvements on many downstream NLP tasks, it has difficulty handling long text because of its quadratic computation complexity. A typical approach to this issue is splitting the input into shorter segments and utilizing order-independent attention mechanism to conduct intersegment interaction, but the approach ignores the segment order information, which is greatly beneficial for capturing implicit relations across different segments. To address this problem, we propose a novel multi-task learning framework, named LordBERT, which fully exploits both intra- and inter-segment information in long text by segment ordering with BERT. LordBERT learns segment-level representations from segments through BERT and a reasoner, and utilizes an auxiliary segment ordering module to reorder disordered segments. With this module, the model implicitly encodes intersegment relations and global information of long text into segment representations. The downstream task and the ordering task are jointly optimized during training, while for inferencing we mainly conduct the downstream task. Experimental results show that LordBERT outperforms the state-of-the-art models by up to 0.58% in accuracy for text classification tasks on long text.

Text segmentation is a classical task in NLP that aims to split a document into several coherent sections. Existing work typically regards it as a binary classification task and predicts whether a paragraph or sentence is the end of a section through bottom-up paragraph content coherence modeling. However, the high-level gist, i.e., the essential meaning of a certain section, is not well utilized. In this work, we propose a gist-driven text segmentation model (GiTS), which locates section boundaries utilizing both the top-down section gist and the bottom-up paragraph contents. Specifically, GiTS consists of a gist generator and a boundary predictor. The RNN-based gist generator generates section gist in order to consider high-level content coherence and distinguishability. The boundary predictor predicts boundary through a pointer network. We also propose an auxiliary gist processor module, which can seamlessly extend GiTS to diverse NLP tasks, such as heading generation, question answering and text classification. Experimental results on text segmentation show our model outperforms the RNN-based state-of-the-art model by 6.83 % on average in terms of$P_{k}$. We also show that combining GiTS with the auxiliary gist processor outperforms the state-of-the-art heading generation models by 22.68% in accuracy and 44.20% in Rouge-1.

Learning an empirically effective model with generalization using limited data is a challenging task for deep neural networks. In this paper, we propose a novel learning framework called Purified Learning to exploit task-irrelevant features extracted from task-irrelevant labels when training models on small-scale datasets. Particularly, we purify feature representations by using the expression of task-irrelevant information, thus facilitating the learning process of classification. Our work is built on solid theoretical analysis and extensive experiments, which demonstrate the effectiveness of Purified Learning. According to the theory we proved, Purified Learning is model-agnostic and doesn't have any restrictions on the model needed, so it can be combined with any existing deep neural networks with ease to achieve better performance.

Generating text from abstract meaning repre-sentation (AMR) is a challenging task. Graph-to-sequence (Graph2Seq-based) methods and pre-trained-based methods are proposed for this task. However, both methods have advantages and disadvantages. Graph2Seq-based methods can make use of the structural information of the graph but not the extra knowledge, while pre-trained-based methods have the advantage of the utilization of extra knowledge but may lose the structural information. In addition, both types of methods often suffer from the under- and over-translation problem. To address these prob-lems, we propose a graph structure reconstruction and coverage enhanced model for this task. The graph structure reconstruction uses two auxiliary objectives, relationship prediction and distance prediction of nodes in AMR graphs to enhance the information of graph structure. In addition, we design a coverage mechanism to solve the problem of information under-translation or over-translation in AMR-to-text generation. Experimental results on three datasets show that our proposed method outperforms the existing methods significantly.

Traditional knowledge distillation adopts a two-stage training process in which a teacher model is pre-trained and then transfers the knowledge to a compact student model. To overcome the limitation, online knowledge distillation is proposed to perform one-stage distillation when the teacher is unavailable. Recent researches on online knowledge distillation mainly focus on the design of the distillation objective, including attention or gate mechanism. Instead, in this work, we focus on the design of the global architecture and propose Tree-Structured Auxiliary online knowledge distillation (TSA), which adds more parallel peers for layers close to the output hierarchically to strengthen the effect of knowledge distillation. Different branches construct different views of the inputs, which can be the source of the knowledge. The hierarchical structure implies that the knowledge transfers from general to task-specific with the growth of the layers. Extensive experiments on 3 computer vision and 4 natural language processing datasets show that our method achieves state-of-the-art performance without bells and whistles. To the best of our knowledge, we are the first to demonstrate the effectiveness of online knowledge distillation for machine translation tasks. 1 1 Code is available at https://github.com/Linwenye/Tree-Supervised.

In a multimedia recommender system, rich multimodal dynamics of user-item interactions are worth availing ourselves of and have been facilitated by Graph Convolutional Networks (GCNs). Yet, the typical way of conducting multimodal fusion with GCN-based models is either through graph mergence fusion that delivers insufficient inter-modal dynamics, or through node alignment fusion that brings in noises which potentially harm multimodal modelling. Unlike existing works, we propose EgoGCN, a structure that seeks to enhance multimodal learning of user-item interactions. At its core is a simple yet effective fusion operation dubbed EdGe-wise mOdulation (EGO) fusion. EGO fusion adaptively distils edge-wise multimodal information and learns to modulate each unimodal node under the supervision of other modalities. It breaks isolated unimodal propagations, allows the most informative inter-modal messages to spread, whilst preserving intra-modal processing. We present a hard modulation and a soft modulation to fully investigate the multimodal dynamics behind. Experiments on two real-world datasets show that EgoGCN comfortably beats prior methods.

Entity Set Expansion (ESE) is a promising task which aims to expand entities of the target semantic class described by a small seed entity set. Various NLP and IR applications will benefit from ESE due to its ability to discover knowledge. Although previous ESE methods have achieved great progress, most of them still lack the ability to handle hard negative entities (i.e., entities that are difficult to distinguish from the target entities), since two entities may or may not belong to the same semantic class based on different granularity levels we analyze on. To address this challenge, we devise an entity-level masked language model with contrastive learning to refine the representation of entities. In addition, we propose the ProbExpan, a novel probabilistic ESE framework utilizing the entity representation obtained by the aforementioned language model to expand entities. Extensive experiments and detailed analyses on three datasets show that our method outperforms previous state-of-the-art methods.

Motivation:The prediction of drug-target interaction (DTI) from heterogeneous biological data is critical to predict drugs and therapeutic targets for known diseases such as tumor and bowel disease. The study of DTI based on drug representation learning can strengthen or integrate our knowledge of pharmacological and chemical phenomena. Therefore, there is a strong motivation to develop effective methods that can detect these potential drug-target interactions.Results:We have developed a novel Knowledge-Enhanced Multi-View framework (KEMV) to predict unknown DTIs from pharmacological data and chemical data on a large scale. The proposed method consists of two steps: (i) learning more comprehensive drug representations via the proposed multi-view attention mechanism, which bridges pharmacological and chemical information, and interactively summarizes the attention values depending on varying interactions between different pairs of drug features. (ii) predicting unknown drug-target interactions based on the drug and target representations. The method is tested on real-world dataset KEGG with three classes of important drug–target interactions involving enzymes, ion channels, and G-protein-coupled receptors. Our framework is proven to uncover potential DTIs with scientific evidences explaining the mechanism of the interactions through the processing of high-dimensional, heterogeneous, and sparse drug data.Availability:The originality of the proposed method lies in the attentive integration of pharmacological and chemical information for representation of drug candidate compounds and the prediction of drug-target interaction toward drug discovery in a unified framework. Our results are reproducible and and code is available at:https://github.com/YuanKQ/DTI-Prediction.

Network traffic classification is important to intrusion detection and network management. Most of existing methods are based on machine learning techniques and rely on the features extracted manually from flows or packets. However, with the rapid growth of network applications, it is difficult for these approaches to handle new complex applications. In this article, we design a novel neural network, the Extended Byte Segment Neural Network (EBSNN), to classify netwrk traffic. EBSNN first divides a packet into header segments and payload segments, which are then fed into encoders composed of the recurrent neural networks with the attention mechanism. Based on the outputs, another encoder learns the high-level representation of the whole packet. In particular, side-channel features are learned from header segments to improve the performance. Finally, the label of the packet is obtained by the softmax function. Furthermore, EBSNN can classify network flows by examining the first few packets. Thorough experiments on the real-world datasets show that EBSNN achieves better performance than the state-of-the-art methods in both the application identification task and the website identification task.

Automatic comment generation is a special and challenging task to verify the model ability on news content comprehension and language generation. Comments not only convey salient and interesting information in news articles, but also imply various and different reader characteristics which we treat as the essential clues for diversity. However, most of the comment generation approaches only focus on saliency information extraction, while the reader-aware factors implied by comments are neglected. To address this issue, we propose a unified reader-aware topic modeling and saliency information detection framework to enhance the quality of generated comments. For reader-aware topic modeling, we design a variational generative clustering algorithm for latent semantic learning and topic mining from reader comments. For saliency information detection, we introduce Bernoulli distribution estimating on news content to select saliency information. The obtained topic representations as well as the selected saliency information are incorporated into the decoder to generate diversified and informative comments. Experimental results on three datasets show that our framework outperforms existing baseline methods in terms of both automatic metrics and human evaluation. The potential ethical issues are also discussed in detail.

Despite pre-trained language models such as BERT have achieved appealing performance in a wide range of Natural Language Processing (NLP) tasks, they are computationally expensive to be deployed in real-time applications. A typical method is to adopt knowledge distillation to compress these large pre-trained models (teacher models) to small student models. However, for a target domain with scarce training data, the teacher can hardly pass useful knowledge to the student, which yields performance degradation for the student models. To tackle this problem, we propose a method to learn to augment data for BERT Knowledge Distillation in target domains with scarce labeled data, by learning a cross-domain manipulation scheme that automatically augments the target domain with the help of resource-rich source domains. Specifically, the proposed method generates samples acquired from a stationary distribution near the target data and adopts a reinforced controller to automatically refine the augmentation strategy according to the performance of the student. Extensive experiments demonstrate that the proposed method significantly outperforms state-of-the-art baselines on different NLP tasks, and for the data-scarce domains, the compressed student models even perform better than the original large teacher model, with much fewer parameters (only ~13.3%) when only a few labeled examples available.

Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, Zhiyuan Liu. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Dong Wang, Ning Ding, Piji Li, Haitao Zheng. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Transfer learning (TL) seeks to improve the learning of a data-scarce target domain by using information from source domains.However, the source and target domains usually have different data distributions, which may lead to negative transfer.To alleviate this issue, we propose a Wasserstein Selective Transfer Learning (WSTL) method.Specifically, the proposed method considers a reinforced selector to select helpful data for transfer learning.We further use a Wasserstein-based discriminator to maximize the empirical distance between the selected source data and target data.The TL module is then trained to minimize the estimated Wasserstein distance in an adversarial manner and provides domain invariant features for the reinforced selector.We adopt an evaluation metric based on the performance of the TL module as delayed reward and a Wasserstein-based metric as immediate rewards to guide the reinforced selector learning.Compared with the competing TL approaches, the proposed method selects data samples that are closer to the target domain.It also provides better state features and reward signals that lead to better performance with faster convergence.Extensive experiments on three real-world text mining tasks demonstrate the effectiveness of the proposed method.

Multi-entity collaborative relationship extraction is an important but challenging task, which has been attracting a lot of interest and poses significant issues in front of systems aimed at natural language understanding. Instead of designing specific models for single relationship extraction tasks, this paper aims to propose a general framework to extract multiple relations among multiple entities in unstructured text by taking advantage of existing models. Based on performing named entity recognition and relation extraction collaboratively, the framework exploits correlations and information propagation among words and relations in a graph network to grasp fundamental features for final classification. The experimental results on two real-world datasets demonstrate that our framework has remarkable applicability and generalizability, and consistently outperforms the strong competitors by a noticeable margin for multi-entity relation extraction.

Beginners of foreign languages tend to pose their linguistic questions online. To describe the confusion, they usually use several example sentences in a foreign language, which makes the linguistic problem tedious long, and composed of two different languages. Therefore, generating a brief headline that covers the main points of the linguist question is important. We propose a new task: Generating Headlines for online Linguistic Questions (GHLQ). The task differs from the previous headline generation task. The main point is that the linguistic question and the headline contain two languages with specific corresponding morphology and syntax. For this reason, we propose a bilingual self-attention network, which can jointly model the information in two different languages effectively. Besides, we construct two datasets, JapLQ and GerLQ, from the real world, and we conduct experiments on these datasets. By the ways of automatic evaluation and human evolution, results show the effectiveness of our model.

Existing works for Dialogue Act Recognition (DAR) pay little attention to the imbalanced distribution of the Dialogue Acts (DAs) and exclusively train their models over very fine-grained DAs in one pass, which leads to a limited performance in recognizing low-frequent DAs. To address this issue, we propose a hierarchical label structured network that explicitly introduces coarse-grained DAs to the original fine-grained DAs. A two-pass multi-head attention mechanism is devised to integrate different levels of DA information into the utterance encoding process, thereby utilizing the information learned from the coarse-grained DAs to guide the recognition of the target fine-grained DAs. Besides, a transfer learning over large-scale dialogue datasets is also employed to further boost label representation of the coarse-grained DAs. Extensive experiments show that our model significantly outperforms the state-of-the-art methods, verifying the effectiveness of integrating the hierarchical structure among DAs.

Automatic question generation from a sentence or paragraph is a challenging task. Recently, impressive progress has been made, mainly owing to the advance of deep neural networks. However, existing neural approaches tend to simply copy words from the given answer, making the generated question equivocal and informatively redundant. In this work, we propose an edit-based network, named Multi-stage Answers Editing Network (MultiEdit), to generate the question in the three-stage operations. In the initial stage, we convert the answer into an edit vector and then combine the edit vector and sentence representation to initialize the decoder, which helps choose an interrogative word matching the answer type. In the second stage, we edit the contextual representation by computing the weighted average between the context representation and the edit vector. In the final stage, we adopt the additional reinforcement learning to improve the effectiveness of generated semantics and evaluation indicators. Experimental results on both Chinese and English datasets show that our approach outperforms the state-of-the-art methods in terms of both automatic and human evaluation.

Knowledge graphs (KG) often encounter knowledge incompleteness. The path reasoning that predicts the unknown path relation between pairwise entities based on existing facts is one of the most promising approaches to the knowledge graph completion. However, most conventional path reasoning methods exclusively consider the entity description included in fact triples, ignoring both the type information of entities and the interaction between different semantic representations. In this study, we propose a novel method, Type-aware Attentive Path Reasoning (TAPR), to complete the knowledge graph by simultaneously considering KG structural information, textual information, and type information. More specifically, we first leverage types to enrich the representational learning of entities and relationships. Next, we describe a type-level attention to select the most relevant type of given entity in a specific triple without any predefined rules or patterns to reduce the impact of noisy types. After learning the distributed representation of all paths, path-level attention assigns different weights to paths, from which relations among entity pairs are calculated. We conduct a series of experiments on a real-world dataset to demonstrate the effectiveness of TAPR. Experimental results show that our method significantly outperforms all baselines on link prediction and entity prediction tasks.

Paraphrase generation aims to rewrite a text with different words while keeping the same meaning. Previous work performs the task based solely on the given dataset while ignoring the availability of external linguistic knowledge. However, it is intuitive that a model can generate more expressive and diverse paraphrase with the help of such knowledge. To fill this gap, we propose Knowledge-Enhanced Paraphrase Network (KEPN), a transformer-based framework that can leverage external linguistic knowledge to facilitate paraphrase generation. (1) The model integrates synonym information from the external linguistic knowledge into the paraphrase generator, which is used to guide the decision on whether to generate a new word or replace it with a synonym. (2) To locate the synonym pairs more accurately, we adopt an incremental encoding scheme to incorporate position information of each synonym. Besides, a multi-task architecture is designed to help the framework jointly learn the selection of synonym pairs and the generation of expressive paraphrase. Experimental results on both English and Chinese datasets show that our method significantly outperforms the state-of-the-art approaches in terms of both automatic and human evaluation.

Fully supervised neural approaches have achieved significant progress in the task of Chinese word segmentation (CWS).Nevertheless, the performance of supervised models tends to drop dramatically when they are applied to outof-domain data.Performance degradation is caused by the distribution gap across domains and the out of vocabulary (OOV) problem.In order to simultaneously alleviate these two issues, this paper proposes to couple distant annotation and adversarial training for crossdomain CWS.For distant annotation, we rethink the essence of "Chinese words" and design an automatic distant annotation mechanism that does not need any supervision or pre-defined dictionaries from the target domain.The approach could effectively explore domain-specific words and distantly annotate the raw texts for the target domain.For adversarial training, we develop a sentence-level training procedure to perform noise reduction and maximum utilization of the source domain information.Experiments on multiple realworld datasets across various domains show the superiority and robustness of our model, significantly outperforming previous state-ofthe-art cross-domain CWS methods.

Conversational Emotion Recognition (CER) is a crucial task in Natural Language Processing (NLP) with wide applications.Prior works in CER generally focus on modeling emotion influences solely with utterance-level features, with little attention paid on phrase-level semantic connection between utterances.Phrases carry sentiments when they are referred to emotional events under certain topics, providing a global semantic connection between utterances throughout the entire conversation.In this work, we propose a two-stage Summarization and Aggregation Graph Inference Network (SumAggGIN), which seamlessly integrates inference for topic-related emotional phrases and local dependency reasoning over neighbouring utterances in a global-to-local fashion.Topic-related emotional phrases, which constitutes the global topic-related emotional connections, are recognized by our proposed heterogeneous Summarization Graph.Local dependencies, which captures short-term emotional effects between neighbouring utterances, are further injected via an Aggregation Graph to distinguish the subtle differences between utterances containing emotional phrases.The two steps of graph inference are tightly-coupled for a comprehensively understanding of emotional fluctuation.Experimental results on three CER benchmark datasets verify the effectiveness of our proposed model, which outperforms the stateof-the-art approaches.

Dialogue Act Recognition (DAR) is a challenging problem in Natural Language Understanding, which aims to attach Dialogue Act (DA) labels to each utterance in a conversation.However, previous studies cannot fully recognize the specific expressions given by users due to the informality and diversity of natural language expressions.To solve this problem, we propose a Heterogeneous User History (HUH) graph convolution network, which utilizes the user's historical answers grouped by DA labels as additional clues to recognize the DA label of utterances.To handle the noise caused by introducing the user's historical answers, we design sets of denoising mechanisms, including a History Selection process, a Similarity Re-weighting process, and an Edge Re-weighting process.We evaluate the proposed method on two benchmark datasets MSDialog and MRDA.The experimental results verify the effectiveness of integrating user's historical answers, and show that our proposed model outperforms the state-of-the-art methods.

Deep question generation (DQG) aims to generate complex questions through reasoning over multiple documents. The task is challenging and underexplored. Existing methods mainly focus on enhancing document representations, with little attention paid to the answer information, which may result in the generated question not matching the answer type and being answerirrelevant. In this paper, we propose an Answer-driven Deep Question Generation (ADDQG) model based on the encoder-decoder framework. The model makes better use of the target answer as a guidance to facilitate question generation. First, we propose an answer-aware initialization module with a gated connection layer which introduces both document and answer information to the decoder, thus helping to guide the choice of answer-focused question words. Then a semantic-rich fusion attention mechanism is designed to support the decoding process, which integrates the answer with the document representations to promote the proper handling of answer information during generation. Moreover, reinforcement learning is applied to integrate both syntactic and semantic metrics as the reward to enhance the training of the ADDQG. Extensive experiments on the HotpotQA dataset show that ADDQG outperforms state-of-the-art models in both automatic and human evaluations.

Response selection plays a vital role in building retrieval-based conversation systems.Despite that response selection is naturally a learning-to-rank problem, most prior works take a point-wise view and train binary classifiers for this task: each response candidate is labeled either relevant (one) or irrelevant (zero).On the one hand, this formalization can be sub-optimal due to its ignorance of the diversity of response quality.On the other hand, annotating grayscale data for learning-to-rank can be prohibitively expensive and challenging.In this work, we show that grayscale data can be automatically constructed without human effort.Our method employs off-the-shelf response retrieval models and response generation models as automatic grayscale data generators.With the constructed grayscale data, we propose multi-level ranking objectives for training, which can (1) teach a matching model to capture more fine-grained context-response relevance difference and (2) reduce the traintest discrepancy in terms of distractor strength.Our method is simple, effective, and universal.Experiments on three benchmark datasets and four state-of-the-art matching models show that the proposed approach brings significant and consistent performance improvements.

In this paper, we focus on essay generation, which aims at generating an essay (a paragraph) according to a set of topic words. Automatic essay generation can be applied to many scenarios to reduce human workload. Recently the recurrent neural networks (RNN) based methods are proposed to solve this task. However, the RNN-based methods suffer from incoherence problem and duplication problem. To overcome these shortcomings, we propose a self-attention and retrieval enhanced neural network for essay generation. We retrieve sentences relevant to topic words from corpus as material to assist in generation to alleviate the duplication problem. To improve the coherence of essays, the self-attention based encoders are applied to encode topic and material, and the self-attention based decoder are used to generate essay respectively. The final essay is generated under the guidance of topic and material. Experimental results on a real essay dataset show that our model outperforms state-of-the-art baselines according to automatic evaluation and human evaluation.

Answer selection which aims to select the most appropriate answer from a pre-selected candidate pool has become increasingly important in a variety of practical applications. Previous work tends to use complex attention mechanisms to capture contextual relevance between question-answer (QA) pairs while ignoring large scale commonsense knowledge. However, this commonsense knowledge provides real-world background information beyond the context, which can help to discover the latent relevance between two sentences. In this paper, we propose to integrate commonsense from the external knowledge graphs (KGs) into the answer selector through a knowledge-aware context-based attention mechanism. To explore the interrelations among knowledge and context, we leverage a compare-aggregate framework to capture more interactive features between questions and answers. Our model is evaluated on two widely-used benchmark QA datasets: WikiQA and TrecQA. The experiments show that our proposed model outperforms the state-of-the-art method.

We study the problem of infobox-to-text generation that aims to generate a textual description from a key-value table. Representing the input infobox as a sequence, previous neural methods using end-to-end models without order-planning suffer from the problems of incoherence and inadaptability to disordered input. Recent planning-based models only implement static order-planning to guide the generation, which may cause error propagation between planning and generation. To address these issues, we propose a Tree-like PLanning based Attention Network (Tree-PLAN) which leverages both static order-planning and dynamic tuning to guide the generation. A novel tree-like tuning encoder is designed to dynamically tune the static order-plan for better planning by merging the most relevant attributes together layer by layer. Experiments conducted on two datasets show that our model outperforms previous methods on both automatic and human evaluation, and demonstrate that our model has better adaptability to disordered input.

Generating a textual description from a set of RDF triplets is a challenging task in natural language generation. Recent neural methods have become the mainstream for this task, which often generate sentences from scratch. However, due to the huge gap between the structured input and the unstructured output, the input triples alone are insufficient to decide an expressive and specific description. In this paper, we propose a novel anchor-to-prototype framework to bridge the gap between structured RDF triples and natural text. The model retrieves a set of prototype descriptions from the training data and extracts writing patterns from them to guide the generation process. Furthermore, to make a more precise use of the retrieved prototypes, we employ a triple anchor that aligns the input triples into groups so as to better match the prototypes. Experimental results on both English and Chinese datasets show that our method significantly outperforms the state-of-the-art baselines in terms of both automatic and manual evaluation, demonstrating the benefit of learning guidance from retrieved prototypes to facilitate triple-to-text generation.

Automatic fault localization plays a significant role in assisting developers to fix software bugs efficiently. Although existing approaches, e.g., static methods and dynamic ones, have greatly alleviated this problem by analyzing static features in source code and diagnosing dynamic behaviors in software running state respectively, the fault localization accuracy still does not meet user requirements. To improve the fault locating ability with statement granularity, this paper proposes ALBFL, a novel neural ranking model that involves the attention mechanism and the LambdaRank model, which can integrate the static and dynamic features and achieve very high accuracy for identifying software faults. ALBFL first introduces a transformer encoder to learn the semantic features from software source code. Also, it leverages other static statistical features and dynamic features, i.e., eleven Spectrum-Based Fault Localization (SBFL) features, three mutation features, to evaluate software together. Specially, the two types of features are integrated through a self-attention layer, and fed into the LambdaRank model so as to rank a list of possible fault statements. Finally, thorough experiments are conducted on 5 open-source projects with 357 faulty programs in Defects4J. The results show that ALBFL outperforms 11 traditional SBFL methods (by three times) and 2 state-of-the-art approaches (by 13%) on ranking faulty statements in the first position.

Chinese relation extraction is conducted using neural networks with either character-based or word-based inputs, and most existing methods typically suffer from segmentation errors and ambiguity of polysemy.To address the issues, we propose a multi-grained lattice framework (MG lattice) for Chinese relation extraction to take advantage of multi-grained language information and external linguistic knowledge.In this framework, (1) we incorporate word-level information into character sequence inputs so that segmentation errors can be avoided.(2) We also model multiple senses of polysemous words with the help of external linguistic knowledge, so as to alleviate polysemy ambiguity.Experiments on three realworld datasets in distinct domains show consistent and significant superiority and robustness of our model, as compared with other baselines.The source code of this paper can be obtained from https://github.com/ thunlp/Chinese_NRE.

Textual entailment is a central problem of language variability, which has been attracting a lot of interest and it poses significant issues in front of systems aimed at natural language understanding. Recently, various frameworks have been proposed for textual entailment recognition, ranging from traditional computational linguistics techniques to deep learning model based methods. However, recent deep neural networks that achieve the state of the art on textual entailment task only consider the context information of the given sentences rather than the real-world background information and knowledge beyond the context. In the paper, we propose a Knowledge-Context Interactive Textual Entailment Network (KCI-TEN) that learns graph level sentence representations by harnessing external knowledge graph with graph attention network. We further propose a text-graph interaction mechanism for neural based entailment matching learning, which endows the redundancy and noise with less importance and put emphasis on the informative representations. Experiments on the SciTail dataset demonstrate that KCI-TEN outperforms the state-of-the-art methods.

Ning Ding, Ziran Li, Zhiyuan Liu, Haitao Zheng, Zibo Lin. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Many existing rating-based recommendation algorithms have achieved relative success. However, the real-world datasets are extremely sparse and most rating-based algorithms are still suffering from the data sparsity problem. Along with integer-valued ratings, we consider that the user-generated review is also an important user feedback. Furthermore, compared with the traditional recommendation algorithms which have the limited ability to learn the distributions of ratings and reviews simultaneously, the generative adversarial networks can learn better representations for data. In this paper, we propose Rating and Review Generative Adversarial Networks (RRGAN), an innovative framework for recommendation, in which the generative model and discriminative model play a minimax game. Specifically, the generative model predicts the ratings of top-N list for users or items based on reviews, while the discriminative model aims to distinguish the predicted ratings from real ratings. With the competition between these two models, RRGAN improves the ability of understanding users and items based on ratings and reviews. We introduce the user profiles, item representations and ratings into a matrix factorization model to predict the top-N list for the users. In addition, we study three different architectures to learn reasonable user profiles and item representations based on ratings and reviews to achieve better recommendations. To evaluate the performance of our model, we conduct the extensive experiments on three real-world amazon datasets in three parts, which are top-N recommendation analysis, case study and long-tail users analysis. The experimental results show that our method significantly outperforms various state-of-the-art methods, including LFM, LambdaFM, HFT, DeepCoNN and IRGAN methods.

Aggregated search aims to construct search result pages (SERPs) from blue-links and heterogeneous modules (such as news, images, and videos). Existing studies have largely ignored the correlations between blue-links and heterogeneous modules when selecting the heterogeneous modules to be presented. We observe that the top ranked blue-links, which we refer to as the context, can provide important information about query intent and helps identify the relevant heterogeneous modules. For example, informative terms like "streamed" and "recorded" in the context imply that a video module may better satisfy the query. To model and utilize the context information for aggregated search, we propose a model with context attention and representation learning (CARL). Our model applies a recurrent neural network with attention mechanism to encode the context, and incorporates the encoded context information into module embeddings. The context-aware module embeddings together with the ranking policy are jointly optimized under the Markov decision process (MDP) formulation. To achieve a more effective joint learning, we further propose an optimization function with self-supervision loss to provide auxiliary supervision signals. Experimental results based on two public datasets demonstrate the superiority of CARL over multiple baseline approaches, and confirm the effectiveness of the proposed optimization function in boosting the joint learning process.

The Top-N recommendation task aims to recommend users the items they like most. Generative Adversarial Net (GAN) has achieved good results on recommendation, which learns user-item matrix by a generative adversarial training process. There are two main scenarios, pointwise scenarios and pairwise scenarios. However, for recommendation, GANs in pairwise scenarios perform not as well as these in pointwise scenarios. As pairwise rank is a position-independent algorithm, it does not consider Top-N ranking sufficiently. Recommendation task is position-dependent. Especially the Top-N item ranking accuracy is much more important than the ranking accuracy of the tail item. In this paper, we propose LambdaGAN for Top-N recommendation. LambdaGAN introduces lambda rank into generative adversarial training process in order to consider the ranking information of the item. The proposed model enables generative adversarial training in pairwise scenarios available for recommendation by optimizing the rank based metrics directly. Moreover, we adjust lambda function according to the characteristics of recommendation. Two new designed lambda functions are proposed. Experimental results show that LambdaGAN outperforms state-of-the-art algorithms including BPR, PRFM, LambdaFM and IRGAN in terms of four standard evaluation metrics on two widely used datasets, Movielens-100K and Netflix.

Knowledge graphs (KGs) have been applied to many semantic-driven applications, including knowledge interchange and semantic inference. However, most KGs are far from complete and are growing rapidly. Although significant progress has been made in the symbolic representation learning of KGs with structural information, the textual knowledge that plays a crucial role in relation prediction is underutilized, and the issues of redundancy and noise path remain to be settled. In this paper, a Path-based Attribute-aware Representation Learning model (PARL) has been proposed to perform path denoising and path representation learning for the relation prediction task. We develop a novel text-enhanced relation prediction architecture, which interactively learns KG structural and textual representations to vary the sparsity and reliability of KG. Moreover, a path denoising algorithm is presented to emphasize paths with rich information and reduce the impact of redundancy and noise path. Experiments on a public dataset demonstrate that PARL consistently outperforms state-of-the-art methods on relation prediction and KG completion tasks.

Knowledge Based Question Answering (KBQA) is one of the most promising approaches to provide suitable answers for the queries posted by users. Relation detection that aims to take full advantage of the substantial knowledge contained in knowledge base (KB) becomes increasingly important. Significant progress has been made in performing relation detection over KB. However, recent deep neural networks that achieve the state of the art on KB-based relation detection task only consider the context information of question sentences rather than the relatedness between question and answer candidates, and exclusively extract the relation from KB triple rather than learn informative relational path. In this paper, we propose a Knowledge-driven Relation Detection network (KRD) to interactively learn answer-enhanced question representations and path-aware relation representations for relation detection. A Siamese LSTM is employed into a similarity matching process between the question representation and relation representation. Experimental results on the SimpleQuestions and WebQSP datasets demonstrate that KRD outperforms the state-of-the-art methods. In addition, a series of ablation test show the robust superiority of the proposed method.

BACKGROUND: Ontology has attracted substantial attention from both academia and industry. Handling uncertainty reasoning is important in researching ontology. For example, when a patient is suffering from cirrhosis, the appearance of abdominal vein varices is four times more likely than the presence of bitter taste. Such medical knowledge is crucial for decision-making in various medical applications but is missing from existing medical ontologies. In this paper, we aim to discover medical knowledge probabilities from electronic medical record (EMR) texts to enrich ontologies. First, we build an ontology by identifying meaningful entity mentions from EMRs. Then, we propose a symptom-dependency-aware naïve Bayes classifier (SDNB) that is based on the assumption that there is a level of dependency among symptoms. To ensure the accuracy of the diagnostic classification, we incorporate the probability of a disease into the ontology via innovative approaches. RESULTS: We conduct a series of experiments to evaluate whether the proposed method can discover meaningful and accurate probabilities for medical knowledge. Based on over 30,000 deidentified medical records, we explore 336 abdominal diseases and 81 related symptoms. Among these 336 gastrointestinal diseases, the probabilities of 31 diseases are obtained via our method. These 31 probabilities of diseases and 189 conditional probabilities between diseases and the symptoms are added into the generated ontology. CONCLUSION: In this paper, we propose a medical knowledge probability discovery method that is based on the analysis and extraction of EMR text data for enriching a medical ontology with probability information. The experimental results demonstrate that the proposed method can effectively identify accurate medical knowledge probability information from EMR data. In addition, the proposed method can efficiently and accurately calculate the probability of a patient suffering from a specified disease, thereby demonstrating the advantage of combining an ontology and a symptom-dependency-aware naïve Bayes classifier.

Relation detection is a key step in Knowledge Base Question Answering (KBQA), but far from solved due to the significant differences between questions and relations. Previous studies usually treat relation detection as a text matching task, and mainly focus on reducing the detection error with better representations of KB relations. However, the understanding of questions is also important since they are generally more varied. And the text pair representation requires improvement because KB relations are not always counterparts of questions. In this paper, we propose a novel system with enhanced question understanding and representation processes for KB relation detection (QURRD). We design a KBQA-specific slot filling module based on Bi-LSTM-CRF for question understanding. Besides, with two CNNs for modeling and matching text pairs respectively, QURRD obtains richer question-relation representations for semantic analysis, and achieves better performance through learning from multiple tasks. We conduct experiments on both single-relation (Simple-Questions) and multi-relation (WebQSP) benchmarks. Results show that QURRD is robust against the diversity of questions and outperforms the state-of-the-art system on both tasks.

Network traffic classification, which can map network traffic to protocols in the application layer, is a fundamental technique for network management and security issues such as Quality of Service, network measurement, and network monitoring. Recent researchers focus on extracting features for traditional machine learning methods from flows or datagrams of the specific protocol. However, as the rapid growth of network applications, previous works cannot handle complex novel protocols well. In this paper, we introduce the recurrent neural network to network traffic classification and design a novel neural network, the Byte Segment Neural Network (BSNN). BSNN treats network datagrams as input and gives the classification results directly. In BSNN, a datagram is firstly broken into serval byte segments. Then, these segments are fed to encoders which are based on the recurrent neural network. The information extracted by encoders is combined to a representation vector of the whole datagram. Finally, we apply the softmax function to use this vector for predicting the application protocol of this datagram. There are several key advantages of BSNN: 1) no need for prior knowledge of target applications; 2) can handle both connection-oriented protocols and connection-less protocols; 3) supports multi-classification for protocols; 4) shows outstanding accuracy in both traditional protocols and complex novel protocols. Our thorough experiments on real-world data with different protocols indicate that BSNN gains average F1-measure about 95.82% in multi-classification for five protocols including QQ, PPLive, DNS, 360 and BitTorrent. And it also shows excellent performance for detection of novel protocols. Furthermore, compared with two recent state-of-the-art works, BSNN has superiority over the traditional machine learning-based method and the packet inspection method.

Summary Recently, microblog sites such as Twitter attract a great deal of attention as an information resource for topic detection task. Most of existing feature‐pivot topic detection algorithms in Twitter just take a single feature into account rather than multiple features. Thus, these methods always only detect the topics related to the single feature and miss some important topics, which causes a relatively low performance. In this paper, we build a flexible term representation framework for feature‐pivot topic detection based on four features. A Learning‐based Topic Detection using Multiple Features (LTDMF) method is proposed to improve the performance of topic detection. We define a correlation function based on a specific neural network to integrate various features. A Hierarchical Agglomerative Clustering (HAC) algorithm is applied to cluster terms as topics. Based on multiple features, LTDMF detects all types of topics and improves the accuracy of topic detection to solve the problem of missing topics. Experiments show that LTDMF gets a better performance compared with several baseline methods in terms of precision and recall.

High utility mining is a fundamental topic in association rule mining, which aims to discover all itemsets with high utility from transaction database. The previous studies are mainly based on fixed databases, which are not applicable for incremental databases. Although incremental high utility pattern (IHUP) mining has been proposed, its tree structure IHUP-Tree is redundant and thus IHUP algorithm has relative low efficiency. To address this issue, we propose an incremental compressed high utility mining algorithm called iCHUM. The iCHUM algorithm utilizes items of high transaction weighted utilization (TWU) to construct its tree structure, namely iCHUM-Tree. The iCHUM algorithm updates iCHUM-Tree when new database is appended to the original database. The information of high utility itemsets is maintained in the iCHUM-Tree such that candidate itemsets can be generated through mining procedure. Performance analysis shows that our algorithm is more efficient than baseline approaches in incremental databases.

Current P2P applications are based on random connected overlays, which lead to generating a significant amount of inter-ISP traffic. Asking for more QoS requirements, such as a short delay and a stable streaming rate, few studies are dedicated to optimizing P2P live streaming applications, despite that recent work have proposed some solutions for P2P file distribution applications. In this paper, current traffic localization strategy is analyzed at first, and its two inherent flaws are revealed. Then we propose a novel strategy for ISP-friendly live streaming: based on a hybrid overlay, which is a two-tier structure, all ISPs are organized into an ISP-tree, and then local peers in each ISP form a mesh overlay. In each tier, a data scheduling is designed for inter-ISP traffic reduction and performance guarantee respectively. Compared with a famous live streaming strategy, R2, simulation results demonstrate that our strategy generates much less inter-ISP traffic and achieves a higher system performance.

In this paper, we address the problem of content replication in segmented peer-to-peer on-demand systems, with the objective of minimizing the content server's workload. We consider the system performance under heterogeneous environment. In this P2P-VoD system, multimedia content is divided into segments and peers can seek and cache any segments. Because different segments may be of different popularity, badly designed segment replication may lead to great server's workload. We deduce the “optimal replication ratio” in segmented P2P-VoD system such that peers will receive upload bandwidth from each other and at the same time, minimize the server's workload. We formulate the segment replication as an optimization problem and propose a model to solve it. We show that the proportional replication strategy is not optimal for segmented P2P-VoD systems and the segmented system can lead to less server's workload than non-segmented. We simulate our model, evaluate the performance of segmented P2P-VoD systems and show that our algorithm can greatly reduce the server's workload.

In this paper, we study the maximum multicast set scheduling problem in a plane wireless network under the physical interference model. Specifically, from a given set of multicast transmissions, select a maximum set of multicast transmissions with largest sum of receivers, where each receiver of a multicast transmission is interference-free from other transmissions subject to the physical interference constraint. This problem is known to be NP-hard and in this paper we develop an approximation algorithm with constant approximation bound guarantee. To the best of our knowledge, this is the first multicast scheduling algorithm under the physical interference model. For ease of implementation in distributed settings, we focus on the problem in an oblivious power assignment setting.

In this paper, we propose a keyword proximity search approach that can be applied to large and complex RDF database. We model RDF database as undirected data graph, construct three indexes for each data graph, only one index need be loaded into memory. Keyword graph is defined as search result, keyword tree and minimal keyword tree are proposed as middle structures for Keyword graph extraction, and we present a link join operation based algorithm to retrieve Keyword trees in this paper. We employ a technique of keyword node pruning to accelerate keyword tree retrieval and define a scoring function to rank search results. In experiments, our approach achieves both high efficiency and high accuracy, outperforms the existing approaches.

With increasing digital information availability, semantic web technologies have been employed to construct semantic digital libraries in order to ease information comprehension. The use of semantic web enables users to search or visualize resources in a semantic fashion. Semantic web generation is a key process in semantic digital library construction, which converts metadata of digital resources into semantic web data. Many text mining technologies, such as keyword extraction and clustering, have been proposed to generate semantic web data. However, one important type of metadata in publications, called affiliation, is hard to convert into semantic web data precisely because different authors, who have the same affiliation, often express the affiliation in different ways. To address this issue, this paper proposes a clustering method based on normalized compression distance for the purpose of affiliation disambiguation. The experimental results show that our method is able to identify different affiliations that denote the same institutes. The clustering results outperform the well-known k-means clustering method in terms of average precision, F-measure, entropy, and purity.

Abstract As a greater volume of information becomes increasingly available across all disciplines, many approaches, such as document clustering and information visualization, have been proposed to help users manage information easily. However, most of these methods do not directly extract key concepts and their semantic relationships from document corpora, which could help better illuminate the conceptual structures within given information. To address this issue, we propose an approach called “Clonto” to process a document corpus, identify the key concepts, and automatically generate ontologies based on these concepts for the purpose of conceptualization. For a given document corpus, Clonto applies latent semantic analysis to identify key concepts, allocates documents based on these concepts, and utilizes WordNet to automatically generate a corpus‐related ontology. The documents are linked to the ontology through the key concepts. Based on two test collections, the experimental results show that Clonto is able to identify key concepts, and outperforms four other clustering algorithms. Moreover, the ontologies generated by Clonto show significant informative conceptual structures.

In the age of increasing information availability, many techniques, such as document clustering and information visualization, have been developed to ease understanding of information for users. However, most of these methods do not help users directly understand key concepts and their semantic relationships in document corpora, which are critical for capturing their conceptual structures. Therefore, we propose a novel approach called 'Clonto' to identify the key concepts and automatically generate ontologies based on these concepts for conceptualization of document corpora. Clonto applies latent semantic analysis to identify key concepts, allocates documents based on these concepts, and utilizes WordNet to automatically generate a corpus-related ontology. The documents are linked to the ontology through the key concepts. The experimental results show that Clonto can identify key concepts with a high precision and the clustering results of Clonto outperform the STC (Suffix Tree Clustering) algorithm, the Lingo clustering algorithm, the Fuzzy Ants clustering algorithm, and clustering based on TRS (Tolerance Rough Set). Moreover, based on the same document corpus, the ontology generated by Clonto shows a significant informative conceptual structure.