Multi-domain knowledge graph completion (MKGC) seeks to predict missing triples in a target KG by leveraging triples from multiple KGs in different domains (e.g., languages or sources). Existing studies typically learn and fuse multi-domain KG representations solely with alignments or fusion modules, which can be affected by redundant information within KGs. This issue can conceal task-relevant information in representations, impeding further improvements when scaling to numerous KGs. To this end, we propose IMKGC, an information-theoretic MKGC framework to learn minimal sufficient representations. In particular, IMKGC learns entity representations by explicitly preserving endogenous contextual information within each KG, exogenous complementary information from other KGs, and consistent information of equivalent entities, while suppressing redundant information through variational constraints. Furthermore, we achieve compressed relation representations with a devised relation reasoning decoder that captures relatedness among relations, also improving triple prediction. Extensive experiments on 14 KGs in three benchmark datasets demonstrate that IMKGC significantly outperforms previous state-of-the-art methods, especially in redundant scenarios.

Attribute-specific fashion retrieval aims to enhance fine-grained image retrieval by emphasizing the similarity of specific attributes. Current methods primarily rely on attention mechanisms to extract attribute-related visual features but face two key challenges: the limitations of coarse-grained localization in achieving fine-grained accuracy, and an imbalance between global and local perception, where excessive focus on local features can undermine overall performance. To address these issues, we propose the fashion microscope ProFashion, which achieves pixel-level attribute awareness through optimal transport and neural semantic aggregation. The framework begins by employing optimal transport to align semantic attributes with visual patterns from a global perspective, generating an attribute-visual value map that highlights distinctive regions while reducing interference. This is followed by simulating the human brain's perception of attribute feature patterns through superpixel generation and aggregation, capturing attribute-related features at the pixel semantic level and forming key semantic clusters that preserve microstructures. Building on this, an attribute graph is constructed to facilitate feature clustering, significantly enhancing the framework's capability to handle overlapping features and cross-scale relationships. Comprehensive experiments on the FashionAI, DeepFashion, and DARN datasets demonstrate the framework's effectiveness, achieving overall MAP improvements of 3.11%, 3.70%, and 3.49%, respectively. Additionally, the framework delivers relative average throughput gains of 26.94%, 22.22%, and 24.78% on the FashionAI, DeepFashion, and DARN datasets, respectively.

Yilong Chen, Zitian Gao, Yihao Xiao, Jason Klein Liu, Xinyu Yang, Yifan Luo, Haoming Luo, Zhengmao Ye, Tingwen Liu, Ran Tao, Bryan Dai. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Yilong Chen, Junyuan Shang, Yuchen Feng, Zhenyu Zhang, Naibin Gu, Ziqi Wang, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Shuaiyi Nie, Dingsiyu, Wenyuan Zhang, Linhao Yu, Tianmeng Yang, Yao Chen, Weichong Yin, Yu Sun, Hua Wu, Tingwen Liu. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Yuanbo Xie, Yingjie Zhang, Yulin Li, Shouyou Song, Xiaokun Chen, Zhihan Liu, Liya Su, Tingwen Liu. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Juwei Yue, Chuanrui Hu, Jiawei Sheng, Zuyi Zhou, Wenyuan Zhang, Tingwen Liu, Li Guo, Yafeng Deng. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Live-streaming services have attracted widespread popularity due to their real-time interactivity and entertainment value. Users can engage with live-streaming authors by participating in live chats, posting likes, or sending virtual gifts to convey their preferences and support. However, the live-streaming services faces serious data-sparsity problem, which can be attributed to the following two points: (1) User's valuable behaviors are usually sparse, e.g., like, comment and gift, which are easily overlooked by the model, making it difficult to describe user's personalized preference. (2) The main exposure content on our platform is short-video, which is 9 times higher than the exposed live-streaming, leading to the inability of live-streaming content to fully model user preference. To this end, we propose a Frequency-Aware Model for Cross-Domain Live-Streaming Recommendation, termed as FARM. Specifically, we first present the intra-domain frequency aware module to enable our model to perceive user's sparse yet valuable behaviors, i.e., high-frequency information, supported by the Discrete Fourier Transform (DFT). To transfer user preference across the short-video and live-streaming domains, we propose a novel preference align before fuse strategy, which consists of two parts: the cross-domain preference align module to align user preference in both domains with contrastive learning, and the cross-domain preference fuse module to further fuse user preference in both domains using a serious of tailor-designed attention mechanisms. Extensive offline experiments and online A/B testing on Kuaishou live-streaming services demonstrate the effectiveness and superiority of FARM. Our FARM has been deployed in online live-streaming services and currently serves hundreds of millions of users on Kuaishou.

Multi-behavior sequential recommendation (MBSR) aims to learn the dynamic and heterogeneous interactions of users' multi-behavior sequences, so as to capture user preferences under target behavior for the next interacted item prediction. Unlike previous methods that adopt unidirectional modeling by mapping auxiliary behaviors to target behavior, recent concerns are shifting from behavior-fixed to behavior-specific recommendation. However, these methods still ignore the user's latent preference that underlying decision-making, leading to suboptimal solutions. Meanwhile, due to the asymmetric deterministic between items and behaviors, discriminative paradigm based on preference scoring is unsuitable to capture the uncertainty from low-entropy behaviors to high-entropy items, failing to provide efficient and diverse recommendation. To address these challenges, we propose FatsMB, a framework based diffusion model that guides preference generation From Behavior-Agnostic To Behavior-Specific in latent spaces, enabling diverse and accurate Multi-Behavior Sequential Recommendation. Specifically, we design a Multi-Behavior AutoEncoder (MBAE) to construct a unified user latent preference space, facilitating interaction and collaboration across Behaviors, within Behavior-aware RoPE (BaRoPE) employed for multiple information fusion. Subsequently, we conduct target behavior-specific preference transfer in the latent space, enriching with informative priors. A Multi-Condition Guided Layer Normalization (MCGLN) is introduced for the denoising. Extensive experiments on real-world datasets demonstrate the effectiveness of our model. We release our code at https://github.com/OrchidViolet/FatsMB.

User cold-start problem is a long-standing challenge in recommendation systems. Fortunately, cross-domain recommendation (CDR) has emerged as a highly effective remedy for the user cold-start challenge, with recently developed diffusion models (DMs) demonstrating exceptional performance. However, these DMs-based CDR methods focus on dealing with user-item interactions, overlooking correlations between items across the source and target domains. Meanwhile, the Gaussian noise added in the forward process of diffusion models would hurt user's personalized preference, leading to the difficulty in transferring user preference across domains. To this end, we propose a novel paradigm of Smoothing-Sharpening Process Model for CDR to cold-start users, termed as S2CDR which features a corruption-recovery architecture and is solved with respect to ordinary differential equations (ODEs). Specifically, the smoothing process gradually corrupts the original user-item/item-item interaction matrices derived from both domains into smoothed preference signals in a noise-free manner, and the sharpening process iteratively sharpens the preference signals to recover the unknown interactions for cold-start users. Wherein, for the smoothing process, we introduce the heat equation on the item-item similarity graph to better capture the correlations between items across domains, and further build the tailor-designed low-pass filter to filter out the high-frequency noise information for capturing user's intrinsic preference, in accordance with the graph signal processing (GSP) theory. Extensive experiments on three real-world CDR scenarios confirm that our S2CDR significantly outperforms previous SOTA methods in a training-free manner.

Multi-domain knowledge graph completion (MKGC) aims to improve missing triple prediction in a target KG by transferring knowledge from other support KGs. Existing methods typically enforce consistency constraints on equivalent entities across KGs to transfer knowledge, which risks suppressing domain-specific contextual information of entities. This design can also compromise entity representation information from all KG domains, impeding performance improvements, especially in low-resource data scenarios. To address this, we pioneer a generation-based paradigm for MKGC and propose DMKGC, a conditional diffusion-guided knowledge transfer framework. Our key insight is to treat each KG as a partial view of the entity entire information, and generate informative domain-general entity embeddings through diffusion models conditioned on support KGs. Particularly, we first initialize domain-agnostic entity embeddings as prior entity embeddings, and then encode them within individual KGs. Afterward, we fuse equivalent entities from support KGs as the conditional diffusion generation guidance. We leverage the prior entity embeddings as the proxy generation objective, which ensures this conditional generation to be unbiased towards any conditioned KGs. Simultaneously, we also train the generated embeddings to be predictive across KGs, thus preserving domain-specific information. Extensive experiments on 14 KGs in 3 benchmarks demonstrate a 4.3% average MRR improvement in tail entity prediction over state-of-the-art methods, with sustained gains in low-resource data settings.

Attribute-Specific Fashion Retrieval (ASFR) aims to improve fine-grained image retrieval by focusing on specific attributes. However, existing patch-based attention and Transformer methods often misalign with irregular attribute regions and are prone to background noise, limiting their ability to capture subtle, pixel-level microstructures. To tackle these challenges, we propose Super Fashion., the first ASFR framework that adopts superpixel tokens within a Transformer architecture. Super Fashion initially employs an attribute-guided attention mechanism to extract attribute-related features, which in turn guide the cropping of semantically meaningful image regions. Superpixel segmentation is then leveraged on these regions to generate compact, semantically coherent superpixel tokens. By incorporating modality-specific embeddings for both attribute and superpixel tokens, the superpixel token-based Transformer facilitates adaptive interaction and fusion, thereby enhancing attribute localization and discrimination. Extensive experiments on FashionAI, DARN, and DeepFashion demonstrate relative overall MAP improvements of 1.84%, 9.27%, and 9.35% over prior SOTA. Super Fashion offers a new solution for web-based image retrieval.

The convergence of social networks and the Internet of Things (IoT) forms a complex digital ecosystem that links physical sensor data with human social interactions. However, the growing presence of social bots and fake sensor accounts introduces deceptive behaviors that distort relationships and undermine trust in IoT-enabled social platforms. Existing graph neural network (GNN) approaches have shown promise in social bot detection, yet most are built on the homophily assumption, which constrains their capacity to capture adversarial relationships between genuine and malicious entities. To address this challenge, we propose CAGNet, a Community-Aware Gated Network that jointly models temporal and structural properties. Specifically, user posts are represented as temporal sequences through a recurrent neural network to capture dynamic behavioral patterns, while local clustering coefficients and neighbor feature deviations guide an adaptive fusion of neighbor information and node residuals. Empirical results demonstrate that CAGNet effectively captures both temporal and structural dependencies, enhancing the model’s resilience against adversarial manipulation and improving the reliability of social bot detection in IoT-driven environments. Furthermore, we introduce the expert-annotated Chinese Weibo benchmark for social bot detection, offering a high-quality, reusable resource for evaluation. Experimental results show that the proposed method improves accuracy by 2.01% over current methods.

Personalized news recommendation aims to recommend candidate news to the target user. Since the data and knowledge involved in traditional recommender systems are restricted, recent studies utilize large language models (LLMs) to generate news articles and augment the original dataset. However, despite the superiority of LLM-based augmentation in news recommendation, previous studies still suffer from two serious problems, i.e., structure-level deficiency and semantic-level noise. Since the LLM-based augmentation is mainly implemented at the semantic level, collaborative signals, the critical structure information in recommender systems, is neglected during the generation process. Thus, it is inappropriate to perform recommendation based on the augmented user-news bipartite, which manifests as multiple isolated cliques. Moreover, utilizing the open-world knowledge of LLMs to extend the closed systems will inevitably introduce noise information, leading to difficulties in mining users' real preferences. In this paper, we propose a novel Structure-aware and Semantic-aware approach for LLM-Empowered personalized News Recommendation, named S^2LENR, to tackle the mentioned problems. Specifically, we propose a structure-aware refinement module to inject collaborative information in a parametric way, in order to construct a valid augmented bipartite. Besides, we devise a semantic-aware denoising module utilizing contrastive learning paradigm to overcome the negative effects of noise information. Finally, we calculate the relevance score between target user and candidate news representations. We conduct experiments on two real-world news recommendation datasets MIND-Large, MIND-Small and empirical results demonstrate the effectiveness of our approach from multiple perspectives.

Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Temporal Knowledge Graph Reasoning (TKGR) aims to predict future facts based on historical data. Current mainstream models primarily use embedding techniques, which predict missing facts by representing entities and relations as low-dimensional vectors. However, these models often consider only the structural information of individual entities and relations, overlooking the broader structure of the entire TKG. To address these limitations, we propose a novel model called Relation Logical Reasoning and Relation-aware Entity Encoding (RLEE), drawing inspiration from attention mechanisms and logical rule-based techniques. RLEE introduces a two-layer representation of the TKG: an entity layer and a relation layer. At the relation layer, we extract relation paths to mine potential logical correlations between different relations, learning relation embeddings through a process of relation logical reasoning. At the entity layer, we use the relation-aware attention mechanism to learn the entity embeddings specific to the predicted query relations. These learned relation and entity embeddings are then used to predict facts at future timestamps. When evaluated on five commonly used public datasets, RLEE consistently outperforms state-of-the-art baselines.

Multimodal Large Language Models (MLLMs) excel in various tasks, yet often struggle with modality bias, where the model tends to rely heavily on a single modality and overlook critical information in other modalities, which leads to incorrect focus and generating irrelevant responses. In this paper, we propose using the paradigm of preference optimization to solve the modality bias problem, including RLAIF-V-Bias, a debiased preference optimization dataset, and a Noise-Aware Preference Optimization (NaPO) algorithm. Specifically, we first construct the dataset by introducing perturbations to reduce the informational content of certain modalities, compelling the model to rely on a specific modality when generating negative responses. To address the inevitable noise in automatically constructed data, we combine the noise-robust Mean Absolute Error (MAE) with the Binary Cross-Entropy (BCE) in Direct Preference Optimization (DPO) by a negative Box-Cox transformation, and dynamically adjust the algorithm’s noise robustness based on the evaluated noise levels in the data. Extensive experiments validate our approach, demonstrating not only its effectiveness in mitigating modality bias but also its significant role in minimizing hallucinations. The code and data is available at https://github.com/zhangzef/NaPO.

The significant computational demands of large language models have increased interest in distilling reasoning abilities into smaller models via Chain-of-Thought (CoT) distillation.Current CoT distillation methods mainly focus on transferring teacher-generated rationales for complex reasoning to student models.However, they do not adequately explore teachers' dynamic attention toward critical information during reasoning.We find that language models exhibit progressive attention shifts towards key information during reasoning, which implies essential clues for drawing conclusions.Building on this observation and analysis, we introduce a novel CoT distillation framework that transfers the teacher's stepwise attention on key information to the student model.This establishes structured guidance for the student's progressive concentration on key information during reasoning.More importantly, we develop a Mixture of Layers module enabling dynamic alignment that adapts to different layers between the teacher and student.Our method achieves consistent performance improvements across multiple mathematical and commonsense reasoning datasets.To our knowledge, it is the first method to leverage stepwise attention within CoT distillation to improve small model reasoning. Input Question(a) A sample from the SVAMP dataset.The distilled student model fails to adequately utilize numerical information, leading to erroneous results, whereas the teacher model, during stepwise reasoning, effectively utilizes all numerical information to arrive at the correct final result.

Large language model (LLM) role-playing has gained widespread attention.Authentic character knowledge is crucial for constructing realistic LLM role-playing agents.However, existing works usually overlook the exploration of LLMs' ability to detect characters' known knowledge errors (KKE) and unknown knowledge errors (UKE) while playing roles, which would lead to low-quality automatic construction of character trainable corpus.In this paper, we propose RoleKE-Bench to evaluate LLMs' ability to detect errors in KKE and UKE.The results indicate that even the latest LLMs struggle to detect these two types of errors effectively, especially when it comes to familiar knowledge.We experimented with various reasoning strategies and propose an agent-based reasoning method, Self-Recollection and Self-Doubt (S 2 RD), to explore further the potential for improving error detection capabilities.Experiments show that our method effectively improves the LLMs' ability to detect error character knowledge, but it remains an issue that requires ongoing attention 1 .* indicates corresponding author. 1 The resource is accessible at https://github.com/ WYRipple/rp_kw_errors.2 In this paper, "character" also refers to "role".

Document retrieval aims to find documents related to the query from all candidate documents. Existing studies develop the Generative Retrieval approach, which assigns a unique DocID to each document, and then measures document-query relevance based on the probability of generating the expected DocID for the given query. However, the generated DocID can have expressive limitation of different semantic topics, leading to semantic gaps in document retrieval. Besides, existing GR models usually suffer from the catastrophic forgetting when memorizing new documents incrementally, which makes real-world application impractical. To overcome these issues, we propose Translational Generative Retrieval, which translates each document into potential queries reflecting different topics, specifically by a sequence of token probability distributions. To better model the semantics of potential queries and effectively decode the target query from the distributions, we propose DirEcted Acyclic Graph Retrieval (DEAR) model, which reforms the distributions from a non-autoregressive generative model into a Directed Acyclic Graph. Experimental results demonstrate that DEAR outperforms existing retrieval models, setting a new state-of-the-art in generative retrieval.

Zero-shot cross-domain slot filling is becoming increasingly important due to its ability to generalize to new domains without the need for annotating domain-specific data, which aligns well with the requirements of industrial deployments. Recent advanced works deal with this task through question answering framework and make remarkable progress. However, they always rely on human efforts to manually construct question templates or prompts for all slot types, which is not only labor consuming, but also experience context inconsistency issue between the manual example and the specific test instance. To alleviate this problem, we introduce a retriever designed to extract reference samples from the training sets, serving as demonstrations to guide the model in generating the target slot entity through in-context learning. Building upon this retriever, we propose a retrieval-augmented generative framework that automatically constructs and tailors prompts to each specific test instance, eliminating the need for manual efforts. Experiment results verify that our approach attains the state-of-the-art.

Cross-domain recommendation (CDR) has demon-strated to be an effective solution for alleviating the user cold-start issue. By leveraging rich user-item interactions available in a richly informative source domain, CDR could improve the recommendation performance for cold-start users in the target domain. Previous CDR approaches mostly adhere the Embedding and Mapping (EMCDR) paradigm, which learns a user-shared mapping function to transfer users' preference from the source domain to the target domain, neglecting users' personalized preference. Recent CDR approaches further leverage the meta-learning paradigm, considering the CDR task for each user independently and learning user-specific mapping functions for each user. However, they mostly learn representations for each user individually, which ignores the common preference between different users, neglecting valuable information for CDR. In addition, all these approaches usually summarize the user's preference into an overall representation, which can hardly capture the user's multi-interest preference. To this end, we propose a personalized multi-interest modeling framework for CDR to cold-start users, termed as NF-NPCDR. Specifically, we propose a personalized preference encoder that enhances the neural process (NP) with the normalizing flow (NF) to convert the Gaussian (unimodal) distribution to a multimodal distribution, providing a novel way to capture the user's personalized multi-interest preference. Then, we propose a common preference encoder with a preference pool to capture the common preference between different users. Furthermore, we introduce a stochastic adaptive decoder to incorporate both the personalized and common preference for cold-start users, adaptively modulating both preference for better recommendation. Experimental evalu-ations demonstrate that NF-NPCDR outperforms previous SOTA approaches in five benchmark CDR scenarios.

Transformer models encounter inefficiency when scaling hidden dimensions due to the uniform expansion of parameters. When delving into the sparsity of hidden dimensions, we observe that only a small subset of dimensions are highly activated, where some dimensions are commonly activated across tokens, and some others uniquely activated for individual tokens. To leverage this, we propose MoHD (Mixture of Hidden Dimensions), a sparse architecture that combines shared sub-dimensions for common features and dynamically routes specialized sub-dimensions per token. To address the potential information loss from sparsity, we introduce activation scaling and group fusion mechanisms. MoHD efficiently expands hidden dimensions with minimal computational increases, outperforming vanilla Transformers in both parameter efficiency and task performance across 10 NLP tasks. MoHD achieves 1.7% higher performance with 50% fewer activatied parameters and 3.7% higher performance with 3$\times$ total parameters expansion at constant activated parameters cost. MoHD offers a new perspective for scaling the model, showcasing the potential of hidden dimension sparsity.

Graph neural networks (GNNs) leverage message passing mechanisms to learn the topological features of graph data. Traditional GNNs learns node features in a spatial domain unrelated to the topology, which can hardly ensure topological features. In this paper, we formulates message passing as a system of hyperbolic partial differential equations (hyperbolic PDEs), constituting a dynamical system that explicitly maps node representations into a particular solution space. This solution space is spanned by a set of eigenvectors describing the topological structure of graphs. Within this system, for any moment in time, a node features can be decomposed into a superposition of the basis of eigenvectors. This not only enhances the interpretability of message passing but also enables the explicit extraction of fundamental characteristics about the topological structure. Furthermore, by solving this system of hyperbolic partial differential equations, we establish a connection with spectral graph neural networks (spectral GNNs), serving as a message passing enhancement paradigm for spectral GNNs.We further introduce polynomials to approximate arbitrary filter functions. Extensive experiments demonstrate that the paradigm of hyperbolic PDEs not only exhibits strong flexibility but also significantly enhances the performance of various spectral GNNs across diverse graph tasks.

The rapid development and widespread adoption of large language models (LLMs) in recent years have introduced significant risks, necessitating robust detection methods to distinguish between AI-generated content and human-written text. Traditional training-based approaches often lack flexibility and frequently make predictions without supporting evidence, especially when adapting to new domains, leading to a lack of interpretability. To address this issue, we propose a novel zero-shot detection framework named Reading and Writing detection method. Our approach utilizes an autoregressive model to assess the intrinsic complexity of text, while leveraging an autoencoder model to quantify the difficulty of reconstructing the text. By integrating these two metrics, we effectively highlight the substantial differences between machine-generated and human-written text. We conduct extensive experiments on four large public datasets from state-of-the-art LLMs, including GPT-3.5, GPT-4, and open-source models like LLaMa. The results demonstrate that our detection method shows tremendous potential across various language generation models and text domains.

Knowledge Graph Completion (KGC) is important in addressing the incompleteness of Knowledge Graphs (KGs) and supporting intelligent infrastructures. Recently, numerous methods have been developed for the utilization of Large Language Models (LLMs) to complete KGs in a textual generation manner. However, integrating KGs with LLMs presents multiple challenges. First, the output of LLMs is often unconstrained, and directly performing KGC with LLMs may generate entities beyond the scope of KGs and suffer from hallucination. Second, the inherent token length limitation of LLMs may hinder the integration of multi-aspect information, thereby restricting the effectiveness and efficiency in inference. Third, existing methods that integrate LLMs with KGs typically leverage partial aspects of KG contexts, overlooking the crucial role of multi-aspect information in prompting KGC. In fact, there exists crucial multi-aspect information of KG contexts that support the correctness of a factual triple, such as entity/relation descriptions, reasoning paths and entity neighbors. To this end, we propose a novel framework to explore the impact of multi-aspect information of KG contexts for KGC, termed as MAKGC. Particularly, given a target incomplete triple, MAKGC generates a list of candidate entities using an embedding model, incorporates the most relevant relation paths and entity descriptions as embeddings, and integrates them with structural embeddings into a set of instructions. In this way, the multi-aspect information facilitate LLMs in making accurate predictions. Extensive experiments demonstrate the effectiveness on benchmark datasets, and our model outperform previous competitive methods.

Cross-domain recommendation (CDR) has been proven as a promising way to alleviate the cold-start issue, in which the most critical problem is how to draw an informative user representation in the target domain via the transfer of user preference existing in the source domain. Prior efforts mostly follow the embedding-and-mapping paradigm, which first integrate the preference into user representation in the source domain, and then perform a mapping function on this representation to the target domain. However, they focus on mapping features across domains, neglecting to explicitly model the preference integration process, which may lead to learning coarse user representation. Diffusion models (DMs), which contribute to more accurate user/item representations due to their explicit information injection capability, have achieved promising performance in recommendation systems. Nevertheless, these DMs-based methods cannot directly account for valuable user preference in other domains, leading to challenges in adapting to the transfer of preference for cold-start users. Consequently, the feasibility of DMs for CDR remains underexplored. To this end, we explore to utilize the explicit information injection capability of DMs for user preference integration and propose a Preference-Guided Diffusion Model for CDR to cold-start users, termed as DMCDR. Specifically, we leverage a preference encoder to establish the preference guidance signal with the user's interaction history in the source domain. Then, we explicitly inject the preference guidance signal into the user representation step by step to guide the reverse process, and ultimately generate the personalized user representation in the target domain, thus achieving the transfer of user preference across domains. Furthermore, we comprehensively explore the impact of six DMs-based variants on CDR. Extensive experiments on three real-world CDR scenarios demonstrate the superiority of our DMCDR over SOTA methods and six DMs-based variants.

Low-resource complex named entity recognition focuses on identifying complex entities such as creative work, product name and so on, in scenarios where annotated training data is limited. Recent advanced works deal with this task through data augmentation and make substantial progress. However, existing methods ignore the influence of different types or levels of augmented data on model optimization in different learning stages. To address it, we propose a dual-perspective data augmentation and curriculum learning framework. Specifically, we first employ the large language model (LLM) to construct two kinds of augmented datasets from context-perspective and entity-perspective, respectively. Then, we present a multi-stage curriculum learning strategy including a novel adaptive curriculum arrangement algorithm to automatically select the most suitable kind of augmented set to optimize the target model at each training epoch, thus using the augmented data more effectively and controllably. Experimental results on the public benchmark across various low-resource settings show that our framework outperforms previous works.

Multi-Modal Entity Alignment (MMEA) aims to retrieve equivalent entities from different Multi-Modal Knowledge Graphs (MMKGs), a critical information retrieval task.Existing studies have explored various fusion paradigms and consistency constraints to improve the alignment of equivalent entities, while overlooking that the visual modality may not always contribute positively.Empirically, entities with low-similarity images usually generate unsatisfactory performance, highlighting the limitation of overly relying on visual features.We believe the model can be biased toward the visual modality, leading to a shortcut image-matching task.To address this, we propose a counterfactual debiasing framework for MMEA, termed CDMEA, which investigates visual modality bias from a causal perspective.Our approach aims to leverage both visual and graph modalities to enhance MMEA while suppressing the direct causal effect of the visual modality on model predictions.By estimating the Total Effect (TE) of both modalities and excluding the Natural Direct Effect (NDE) of the visual modality, we ensure that the model predicts based on the Total Indirect Effect (TIE), * Corresponding author.

Dynamics modeling has been introduced as a novel paradigm in message passing (MP) of graph neural networks (GNNs). Existing methods consider MP between nodes as a heat diffusion process, and leverage heat equation to model the temporal evolution of nodes in the embedding space. However, heat equation can hardly depict the wave nature of graph signals in graph signal processing. Besides, heat equation is essentially a partial differential equation (PDE) involving a first partial derivative of time, whose numerical solution usually has low stability, and leads to inefficient model training. In this paper, we would like to depict more wave details in MP, since graph signals are essentially wave signals that can be seen as a superposition of a series of waves in the form of eigenvector. This motivates us to consider MP as a wave propagation process to capture the temporal evolution of wave signals in the space. Based on wave equation in physics, we innovatively develop a graph wave equation to leverage the wave propagation on graphs. In details, we demonstrate that the graph wave equation can be connected to traditional spectral GNNs, facilitating the design of graph wave networks (GWNs) based on various Laplacians and enhancing the performance of the spectral GNNs. Besides, the graph wave equation is particularly a PDE involving a second partial derivative of time, which has stronger stability on graphs than the heat equation that involves a first partial derivative of time. Additionally, we theoretically prove that the numerical solution derived from the graph wave equation are constantly stable, enabling to significantly enhance model efficiency while ensuring its performance. Extensive experiments show that GWNs achieve state-of-the-art and efficient performance on benchmark datasets, and exhibit outstanding performance in addressing challenging graph problems, such as over-smoothing and heterophily. Our code is available at https://github.com/YueAWu/Graph-Wave-Networks.

Yilong Chen, Junyuan Shang, Zhenyu Zhang, Shiyao Cui, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, Hua Wu. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Document layout generation, a burgeoning field of document intelligence, entails positioning and sizing various elements within given constraints. While significant strides have been made in single-page layout generation, real-world documents predominantly span multiple pages, and exploring multi-page layout generation methods has also become the key to meeting the contemporary dramatically increased document processing demands. Despite the promise of leveraging large language models (LLMs) like GPT-4 for their powerful in-context learning abilities, the task transition to multi-page layouts, which contains considerably complex data, presents formidable challenges including excessively long prompts and strict consistency between pages. To this end, we propose a novel framework called Multi-Page Layout Generation via Consistency-Oriented modeling (MuLCO) that capitalizes on in-context learning of LLMs without the need for training or fine-tuning. MuLCO employs three key components: serialization based on code blocks maps intricate document layouts to code-style exemplars, self-correcting reasoning hint decomposes the complex generation task into numerous steps to improve reasoning interpretability, and consistency-oriented multi-round generation predicts coherent multi-page layouts in form of a continuous dialogue. To summarize, we contribute by proposing MuLCO and developing a task-specific dataset and evaluation mechanism. Extensive experiments validate the effectiveness of the MuLCO framework for multi-page layout generation.

Despite the notable progress achieved by large-scale vision-language pre-training models in a wide range of multi-modal tasks, their performance often falls short in image-text matching challenges that require an in-depth understanding of structured representations. For instance, when distinguishing between texts or images that are generally similar but have distinct structured knowledge (such as entities and relationships in text, or objects and object attributes in images), the model's capabilities are limited. In this paper, we propose a advancing Multi-modal Structured Knowledge Representation with synergistic hard negative samples (MSKR), thereby significantly improving the model's matching capability for such data. Specifically, our model comprises a structured knowledge-enhanced encoder designed to bolster the structured knowledge inherent in textual data, such as entities, their attributes, and the relationships among these entities as well as structured knowledge within images, focusing on elements like objects and their attributes. To further refine the model's learning process, we produce both image and text challenging negative samples. Extensive experimental evaluations on the Winoground, InpaintCOCO, and MSCOCO benchmark reveal that MSKR significantly outperforms the baseline model, showcasing marked improvements 2.66% on average in structured representation learning compared to the baseline. Moreover, general representation results illustrate that our model not only excels in structured representation learning but also maintains its proficiency in general representation learning.

Multi-modal entity alignment (MMEA) aims to identify equivalent entities between two multi-modal knowledge graphs (MMKGs), whose entities can be associated with relational triples and related images. Most previous studies treat the graph structure as a special modality, and fuse different modality information with separate uni-modal encoders, neglecting valuable relational associations in modalities. Other studies refine each uni-modal information with graph structures, but may introduce unnecessary relations in specific modalities. To this end, we propose a novel local-to-global interaction network for MMEA, termed as LoginMEA. Particularly, we first fuse local multi-modal interactions to generate holistic entity semantics and then refine them with global relational interactions of entity neighbors. In this design, the uni-modal information is fused adaptively, and can be refined with relations accordingly. To enrich local interactions of multi-modal entity information, we devise modality weights and low-rank interactive fusion, allowing diverse impacts and element-level feature interactions among modalities. To capture global interactions of graph structures, we adopt relation reflection graph attention networks, which fully capture relational associations between entities. Extensive experiments demonstrate superior results of our method over 5 cross-KG or bilingual benchmark datasets, indicating the effectiveness of capturing local and global interactions.

Chinese Spelling Correction (CSC) aims to detect and correct the misspelled characters in Chinese texts. Recent studies have achieved great success by incorporating the phonetic information for task predictions. Still, existing methods suffer from two limitations: 1) The differentiated information between textual characters and Pinyin pronunciation are underexplored. 2) The task predictions are performed with the over-emphasises on either the textual or phonetic sequence, ignoring the balanced modeling and adaptive fusion. In this work, we proposed a method to alleviate the issues above. For the first issue, a Coupled Attention Module (CAM) is proposed where a couple of attention functions capture the associated and differentiated text-phonetics information simultaneously. For the second issue, an adaptive fusion is designed to derive the phonetics-aware textual representations and text-aware phonetic representations for task predictions. Experiments on three public benchmarks demonstrate the effectiveness of our proposed method.

Text-video cross-modal retrieval is an increasingly prominent and challenging task that has garnered significant attention. Traditional models typically embed videos and texts into global vectors, aiming to capture the global features of these modalities. While the models often fall short in capturing fine-grained semantic details. Relying solely on global features proves insufficient to address this challenge. Hence, there is a pressing need to bridge the gap between different modalities by incorporating fine-grained features. In light of this, we propose a highly efficient model designed to capture the fine-grained features of videos and texts including question answer semantic alignment, object alignment and text-video feature fusion. For texts, our model includes the incorporation of entity information and part-of-speech information including adjectives, nouns and verbs information, while for videos, the identification of objects plays a crucial role in facilitating text-video retrieval. Our model undergoes extensive training on the WebVid and CC3M datasets, yielding unequivocal evidence of its superior performance over baseline models. It excels particularly in zero-shot text-video cross-modal retrieval tasks, offering substantial reductions in required computational resources.

Graph contrastive learning aims to learn a representative model by maximizing the agreement between different views of the same graph. Existing studies usually allow multifarious noise in data augmentation, and suffer from trivial and inconsistent generation of graph views. Moreover, they mostly impose contrastive constraints on pairwise representations, limiting the structural correlations among multiple nodes. Both problems may hinder graph contrastive learning, leading to suboptimal node representations. To this end, we propose a novel graph contrastive learning framework, namely GCL-LS, via low-rank and sparse subspace decomposition. In particular, it decomposes node representations into low-rank and sparse components, preserving structural correlations and compressed features in the low-rank and sparse subspace, respectively. By contrasting the representations in the subspaces, it naturally disentangles low-quality noise in data augmentation, and captures structural correlations and substantial features of nodes in contrastive learning. Experimental results show that our method significantly improves downstream node classification accuracy, and further analysis demonstrates the effectiveness of the subspace decomposition in graph contrastive learning. The code is released at https://github.com/zhanggehang/GCL_LS.

Aspect sentiment quad prediction (ASQP) aims to predict the quad sentiment elements for a given sentence, which is a critical task in the field of aspect-based sentiment analysis. However, the data imbalance issue has not received sufficient attention in ASQP task. In this paper, we divide the issue into two-folds, quad-pattern imbalance and aspect-category imbalance, and propose an Adaptive Data Augmentation (ADA) framework to tackle the imbalance issue. Specifically, a data augmentation process with a condition function adaptively enhances the tail quad patterns and aspect categories, alleviating the data imbalance in ASQP. Following previous studies, we also further explore the generative framework for extracting complete quads by introducing the category prior knowledge and syntax-guided decoding target. Experimental1results demonstrate that data augmentation for imbalance in ASQP task can improve the performance, and the proposed ADA method is superior to naive data oversampling.

Topic modeling aims to identify and group related topics within collections of text documents. As multi-modal data becomes increasingly effective and prevalent, researchers are delving into the integration of diverse data types (e.g., images) into topic modeling. However, existing methods for topic modeling struggle with unrelated modal features (e.g., visual features) and efficiency. In this paper, we propose a dynamic multi-modal representation learning method (DMMR) that adaptively integrates multi-modal features to enhance the effectiveness and efficiency of modeling multi-modal data. Concretely, a gating network controls the modality-level decision to choose text, image, or both. Based on the sample-wise choice predicted by the gating network, DMMR performs the fusion of multi-modal features encoded by modality-related expert network. With the dynamic modality selection and fusion, the representative modal features can be chosen to exclude irrelevant modality and speed the inference. Extensive experiments on public datasets demonstrate that the proposed method significantly improves the topic quality (e.g., coherence and diversity) and increases the efficiency by 20.37%.

With the rapid development of the Internet, social media platforms are filled with a large amount of article information, so the task of topic generation, which can greatly improve browsing and reading efficiency, has gradually developed. Currently, manually organizing and generating topic sentences requires a lot of human resources, and the previous researches on automatic topic generation suffer from poor effectiveness, poor readability and high redundancy due to the generation of topic words only. To address these problems, our work introduces the maturing text summarization technology into the automatic topic sentence generation task to improve the coherence and readability of the generated topic sentences. We present a two-stage summary-based news topic sentence generation model, TTSG, which acquires summaries from each article, clusters and sorts the summaries, and then performs a secondary summary to obtain the final topic sentence. Experimental results show that TTSG outperforms the baseline model and generates more coherent and readable topic sentences. Our code is available at https://github.com/sutaoyu/TTSG.

The table-based fact verification task is important yet challenging as it requires models to have both linguistic reasoning and symbolic reasoning ability. Existing studies mainly established two kinds of methods, program-enhanced methods and table-based pre-trained models, to excel at different reasoning aspects. In this paper, we propose a Program-enhanced Table Parser (Prog-TAPAS) to effectively combine these two methods. Leveraging a program enhanced module with program tables and synthesize program statements, we combine the strengths of symbolic reasoning into language-based inference models to support our fact verification model. Experiments show that our proposed framework outperforms all baseline systems with a considerable margin and achieves competitive results, on the TABFACT benchmark.

With the growing interest in Large Language Models (LLMs), integrating visual tasks has led to the development of Multi-Layer Language Models (MLLMs). Despite their advancements, MLLMs face challenges in accuracy and generalization, often due to resource and time constraints. Addressing these issues, our paper introduces a novel Multi-Agent Collaborative Network for MLLMs (MLLM network). This framework harnesses collective intelligence and cooperation among multiple agents to enhance the accuracy and generalizability of MLLMs. The collaborative nature of our MLLMs—featuring inter-layer neuron interaction and information exchange—facilitates superior processing and integration of multi-modal data. This leads to marked improvements in performance. The findings underscore the efficacy and potential of our proposed framework, presenting a robust solution for complex multi-modal challenges in machine learning and artificial intelligence. Our experimental evaluations demonstrate that this approach significantly surpasses traditional single MLLM architectures in task accuracy and generalization.

Multi-modal entity alignment (MMEA) aims to identify equivalent entities between multi-modal knowledge graphs (MMKGs), where entities can be associated with related images.Most existing studies rely heavily on the automatically learned multi-modal fusion modules, which may allow redundant information such as misleading clues in the generated entity representations, impeding the feature consistency of equivalent entities.To this end, we propose a variational framework for MMEA via information bottleneck, termed as IBMEA, by emphasizing alignment-relevant information while suppressing alignment-irrelevant information in entity representations.Specifically, we first develop multi-modal variational encoders that represent modal-specific features as probability distributions.Then, we propose four modal-specific information bottleneck regularizers to limit the misleading clues in the modal-specific entity representations.Finally, we propose a modal-hybrid information contrastive regularizer to integrate modal-specific representations and ensure the similarity of equivalent entities between MMKGs to achieve MMEA.We conduct extensive experiments on 2 cross-KG and 3 bilingual MMEA datasets.Experimental results demonstrate that our model consistently outperforms previous state-of-the-art methods, and also shows promising and robust performance especially in the low-resource and high-noise data scenarios.

Large language models (LLMs) with billions of parameters demonstrate impressive performance. However, the widely used Multi-Head Attention (MHA) in LLMs incurs substantial computational and memory costs during inference. While some efforts have optimized attention mechanisms by pruning heads or sharing parameters among heads, these methods often lead to performance degradation or necessitate substantial continued pre-training costs to restore performance. Based on the analysis of attention redundancy, we design a Decoupled-Head Attention (DHA) mechanism. DHA adaptively configures group sharing for key heads and value heads across various layers, achieving a better balance between performance and efficiency. Inspired by the observation of clustering similar heads, we propose to progressively transform the MHA checkpoint into the DHA model through linear fusion of similar head parameters step by step, retaining the parametric knowledge of the MHA checkpoint. We construct DHA models by transforming various scales of MHA checkpoints given target head budgets. Our experiments show that DHA remarkably requires a mere 0.25\% of the original model's pre-training budgets to achieve 96.1\% of performance while saving 75\% of KV cache. Compared to Group-Query Attention (GQA), DHA achieves a 5$\times$ training acceleration, a maximum of 13.93\% performance improvement under 0.01\% pre-training budget, and 5\% relative improvement under 0.05\% pre-training budget.

MultiModal Sentence Summarization (MMSS) aims to generate a brief summary based on the given source sentence and its associated image. Previous studies on MMSS have achieved success by either selecting the task-relevant visual information or filtering out the task-irrelevant visual information to help the textual modality to generate the summary. However, enhancing from a single perspective usually introduces over-preservation or over-compression problems. To tackle these issues, we resort to Information Bottleneck (IB), which seeks to find a maximally compressed mapping of the input information that preserves as much information about the target as possible. Specifically, we propose a novel method, T(3), which adopts IB to balance the Trade-off between Task-relevant and Task-irrelevant visual information through the variational inference framework. In this way, the task-irrelevant visual information is compressed to the utmost while the task-relevant visual information is maximally retained. With the holistic perspective, the generated summary could maintain as many key elements as possible while discarding the unnecessary ones as far as possible. Extensive experiments on the representative MMSS dataset demonstrate the superiority of our proposed method. Our code is available at https://github.com/YuanMinghuan/T3.

Low-resource Complex Named Entity Recognition aims to detect entities with the form of any linguistic constituent under scenarios with limited manually annotated data. Existing studies augment the text through the substitution of same type entities or language modeling, but suffer from the lower quality and the limited entity context patterns within low-resource corpora. In this paper, we propose a novel data augmentation method E2DA from both exogenous and endogenous perspectives. As for exogenous augmentation, we treat the limited manually annotated data as anchors, and leverage the powerful instruction-following capabilities of Large Language Models (LLMs) to expand the anchors by generating data that are highly dissimilar from the original anchor texts in terms of entity mentions and contexts. As regards the endogenous augmentation, we explore diverse semantic directions in the implicit feature space of the original and expanded anchors for effective data augmentation. Our complementary augmentation method from two perspectives not only continuously expands the global text-level space, but also fully explores the local semantic space for more diverse data augmentation. Extensive experiments on 10 diverse datasets across various low-resource settings demonstrate that the proposed method excels significantly over prior state-of-the-art data augmentation methods.

Cross-domain recommendation (CDR) has been proven as a promising way to tackle the user cold-start problem, which aims to make recommendations for users in the target domain by transferring the user preference derived from the source domain. Traditional CDR studies follow the embedding and mapping (EMCDR) paradigm, which transfers user representations from the source to target domain by learning a user-shared mapping function, neglecting the user-specific preference. Recent CDR studies attempt to learn user-specific mapping functions in meta-learning paradigm, which regards each user's CDR as an individual task, but neglects the preference correlations among users, limiting the beneficial information for user representations. Moreover, both of the paradigms neglect the explicit user-item interactions from both domains during the mapping process. To address the above issues, this paper proposes a novel CDR framework with neural process (NP), termed as CDRNP. Particularly, it develops the meta-learning paradigm to leverage user-specific preference, and further introduces a stochastic process by NP to capture the preference correlations among the overlapping and cold-start users, thus generating more powerful mapping functions by mapping the user-specific preference and common preference correlations to a predictive probability distribution. In addition, we also introduce a preference remainer to enhance the common preference from the overlapping users, and finally devises an adaptive conditional decoder with preference modulation to make prediction for cold-start users with items in the target domain. Experimental results demonstrate that CDRNP outperforms previous SOTA methods in three real-world CDR scenarios.

This paper studies the multimodal named entity recognition (MNER) and multimodal relation extraction (MRE), which are important for content analysis and various applications. The core of MNER and MRE lies in incorporating evident visual information to enhance textual semantics, where two issues inherently demand investigations. The first issue is modality-noise, where the task-irrelevant information in each modality may be noises misleading the task prediction. The second issue is modality-gap, where representations from different modalities are inconsistent, preventing from building the semantic alignment between the text and image. To address these issues, we propose a novel method for MNER and MRE byMultiModal representation learning withInformationBottleneck (MMIB). For the first issue, a refinement-regularizer probes the information-bottleneck principle to balance the predictive evidence and noisy information, yielding expressive representations for prediction. For the second issue, an alignment-regularizer is proposed, where a mutual information-based item works in a contrastive manner to regularize the consistent text-image representations. To our best knowledge, we are the first to explore variational IB estimation for MNER and MRE. Experiments show that MMIB achieves the state-of-the-art performances on three public benchmarks.

Cross-domain Named Entity Recognition (NER) transfers knowledge learned from a rich-resource source domain to improve the learning in a low-resource target domain. Most existing works are designed based on the sequence labeling framework, defining entity detection and type prediction as a monolithic process. However, they typically ignore the discrepant transferability of these two sub-tasks: the former locating spans corresponding to entities is largely domain-robust, whereas the latter owns distinct entity types across domains. Combining them into an entangled learning problem may contribute to the complexity of domain transfer. In this work, we propose the novel divide-and-transfer paradigm in which different sub-tasks are learned using separate functional modules for respective cross-domain transfer. To demonstrate the effectiveness of divide-and-transfer, we concretely implement two NER frameworks by applying this paradigm with different cross-domain transfer strategies. Experimental results on 10 different domain pairs show the notable superiority of our proposed frameworks. Experimental analyses indicate that significant advantages of the divide-and-transfer paradigm over prior monolithic ones originate from its better performance on low-resource data and a much greater transferability. It gives us a new insight into cross-domain NER. Our code is available on GitHub. 1

Graph contrastive learning (GCL) has recently achieved substantial advancements. Existing GCL approaches compare two different “views“ of the same graph in order to learn node/graph representations. The underlying assumption of these studies is that the graph augmentation strategy is capable of generating several different graph views such that the graph views are structurally different but semantically similar to the original graphs, and thus the ground-truth labels of the original and augmented graph/nodes can be regarded identical in contrastive learning. However, we observe that this assumption does not always hold. For instance, the deletion of a supernode within a social network can exert a substantial influence on the partitioning of communities for other nodes. Similarly, any perturbation to nodes or edges in a molecular graph will change the labels of the graph. Therefore, we believe that augmenting the graph, accompanied by an adaptation of the labels used for the contrastive loss, will facilitate the encoder to learn a better representation. Based on this idea, we propose ID-MixGCL, which allows the simultaneous interpolation of input nodes and corresponding identity labels to obtain soft-confidence samples, with a controllable degree of change, leading to the capture of fine-grained representations from self-supervised training on unlabeled graphs. Experimental results demonstrate that ID-MixGCL improves performance on graph classification and node classification tasks, as demonstrated by significant improvements on the Cora, IMDB-B, IMDB-M, and PROTEINS datasets compared to state-of-the-art techniques, by 3-29parcent absolute points. The dataset and source code can be found at: https://github.com/zhanggehang/ID-MIXGCL.

Document-level event factuality identification (DocEFI) is an important task in event knowledge acquisition, which aims to detect whether an event actually occurs or not from the perspective of the document. Unlike the sentence-level task, a document can have multiple sentences with different event factualities, leading to event factuality conflicts in DocEFI. Existing studies attempt to aggregate local event factuality by exploiting document structures, but they mostly consider textual components in the document separately, degrading complicated correlations therein. To address the above issues, this paper proposes a novel approach, namely UR-HAT, to improve DocEFI with uncertain relational hypergraph attention networks. Particularly, we reframe a document graph as a hypergraph, and establish beneficial n-ary correlations among textual nodes with relational hyperedges, which helps to globally consider local factuality features to resolve event factuality conflicts. To better discern the importance of event factuality features, we further represent textual nodes with uncertain Gaussian distributions, and propose novel uncertain relational hypergraph attention networks to refine textual nodes with the document hypergraph. In addition, we select factuality-related keywords as nodes to enrich event factuality features. Experimental results demonstrate the effectiveness of our proposed method, and outperforms previous methods on two widely used benchmark datasets.

Abstractive multi-document summarization (MDS) aims to generate a summary for a set of topic-related documents, in which different documents may contain trivial and redundant information and present complementary or contradictory content. It is essential to extract salient information and detect redundancy across documents for abstractive MDS compared with single-document summarization. However, it is challenging for models to exploit salient information and generate concise summaries. In this paper, we propose an Adapt-to-Learn Policy (ALP) network to seamlessly adapt the key sentence selection and word generation over retrieval and generative agents, guided by the multi-document summarization reward. Moreover, our model learns latent dependencies among textual units and explicitly takes advantage of critical information by focusing on semantic similarity or discourse relations. Extensive experiments on WikiSum and Multi-News datasets confirm that our method is superior to prior competitive baselines, and experimental analyses show that higher-quality summaries and more fluent word order can be generated in our ALP network.

Cross-lingual Named Entity Recognition (NER) aims to address the challenge of data scarcity in low-resource languages by leveraging knowledge from high-resource languages. Most current work relies on general multilingual language models to represent text, and then uses classic combined tagging (e.g., B-ORG) to annotate entities; However, this approach neglects the lack of cross-lingual alignment of entity representations in language models, and also ignores the fact that entity spans and types have varying levels of labeling difficulty in terms of transferability. To address these challenges, we propose a novel framework, referred to as DLBri, which addresses the issues of representation and labeling simultaneously. Specifically, the proposed framework utilizes progressive contrastive learning with source-to-target oriented sentence pairs to pre-finetune the language model, resulting in improved cross-lingual entity-aware representations. Additionally, a decomposition-then-combination procedure is proposed, which separately transfers entity span and type, and then combines their information, to reduce the difficulty of cross-lingual entity labeling. Extensive experiments on 13 diverse language pairs confirm the effectiveness of DLBri.

Personalized news recommendation aims to recommend candidate news to the target user, according to the clicked news history. The user-news interaction data exhibits power-law distribution, however, existing works usually learn representations in Euclidean space which makes inconsistent capacities between data space and embedding space, leading to severe representation distortion problem. Besides, the existence of conformity bias, a potential cause of power-law distribution, may introduce biased guidance to learn user representations. In this paper, we propose a novel debiased method based on hyperbolic space, named HDNR, to tackle the above problems. Specifically, first, we employ hyperboloid model with exponential growth capacity to conduct user and news modeling, in order to solve inconsistent space capacities problem and obtain low distortion representations. Second, we design a re-weighting aggregation module to further mitigate conformity bias in data distribution, through considering local importance of the clicked news among contextual history and its global popularity degree simultaneously. Finally, we calculate the relevance score between target user and candidate news representations. We conduct experiments on two real-world news recommendation datasets MIND-Large, MIND-Small and empirical results demonstrate the effectiveness of our approach from multiple perspectives.

In industry, web platforms such as Alibaba and Amazon often provide diverse services for users. Unsurprisingly, some developed services are data-rich, while some newly started services are data-scarce accompanied by severe data sparsity and cold-start problems. To alleviate the above problems and incubate new services easily, cross-domain recommendation (CDR) has attracted much attention from industrial and academic researchers. Generally, CDR aims to transfer rich user-item interaction information from related source domains (e.g., developed services) to boost recommendation quality of target domains (e.g., newly started services). For different scenarios, previous CDR methods can be roughly divided into two branches: (1) Data sparsity CDR fulfills user preference aided by other domain data to make intra-domain recommendations for users with few interactions, (2) Cold-start CDR projects user preference from other domain to make inter-domain recommendations for users with none interactions. In the past years, many outstanding CDR methods are emerged, however, to the best of our knowledge, none of them attempts to solve the two branches simultaneously. In this paper, we provide a unified framework, namely UniCDR, which can universally model different CDR scenarios by transferring the domain-shared information. Extensive experiments under the above 2 branches on 4 CDR scenarios and 6 public and large-scale industrial datasets demonstrate the effectiveness and universal ability of our UniCDR.

Extracting structured information from all manner of webpages is an important problem with the potential to automate many real-world applications. Recent work has shown the effectiveness of leveraging DOM trees and pre-trained language models to describe and encode webpages. However, they typically optimize the model to learn the semantic co-occurrence of elements and labels in the same webpage, thus their effectiveness depends on sufficient labeled data, which is labor-intensive. In this paper, we further observe structural co-occurrences in different webpages of the same website: the same position in the DOM tree usually plays the same semantic role, and the DOM nodes in this position also share similar surface forms. Motivated by this, we propose a novel method, Structor, to effectively incorporate the structural co-occurrences over DOM tree and surface form into pre-trained language models. Such structural co-occurrences help the model learn the task better under low-resource settings, and we study two challenging experimental scenarios: website-level low-resource setting and webpage-level low-resource setting, to evaluate our approach. Extensive experiments on the public SWDE dataset show that Structor significantly outperforms the state-of-the-art models in both settings, and even achieves three times the performance of the strong baseline model in the case of extreme lack of training data.

Yanzeng Li, Jiangxia Cao, Xin Cong, Zhenyu Zhang, Bowen Yu, Hongsong Zhu, Tingwen Liu. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

Cross-Domain Sequential Recommendation (CDSR) aims to predict future interactions based on user's historical sequential interactions from multiple domains. Generally, a key challenge of CDSR is how to mine precise cross-domain user preference based on the intra-sequence and inter-sequence item interactions. Existing works first learn single-domain user preference only with intra-sequence item interactions, and then build a transferring module to obtain cross-domain user preference. However, such a pipeline and implicit solution can be severely limited by the bottleneck of the designed transferring module, and ignores to consider inter-sequence item relationships.

Event Causality Identification (ECI), which aims to detect whether a causality relation exists between two given textual events, is an important task for event causality understanding. However, the ECI task ignores crucial event structure and cause-effect causality component information, making it struggle for downstream applications. In this paper, we introduce a novel task, namely Event Causality Extraction (ECE), aiming to extract the cause-effect event causality pairs with their structured event information from plain texts. The ECE task is more challenging since each event can contain multiple event arguments, posing fine-grained correlations between events to decide the cause-effect event pair. Hence, we propose a method with a dual grid tagging scheme to capture the intra- and inter-event argument correlations for ECE. Further, we devise a event type-enhanced model architecture to realize the dual grid tagging scheme. Experiments demonstrate the effectiveness of our method, and extensive analyses point out several future directions for ECE.

Multi-intent detection and slot filling joint model attracts more and more attention since it can handle multi-intent utterances, which is closer to complex real-world scenarios.Most existing joint models rely entirely on the training procedure to obtain the implicit correlation between intents and slots.However, they ignore the fact that leveraging the rich global knowledge in the corpus can determine the intuitive and explicit correlation between intents and slots.In this paper, we aim to make full use of the statistical co-occurrence frequency between intents and slots as prior knowledge to enhance joint multiple intent detection and slot filling.To be specific, an intent-slot cooccurrence graph is constructed based on the entire training corpus to globally discover correlation between intents and slots.Based on the global intent-slot co-occurrence, we propose a novel graph neural network to model the interaction between the two subtasks.Experimental results on two public multi-intent datasets demonstrate that our approach outperforms the state-of-the-art models.

Document-level Event Extraction (DEE) is particularly tricky due to the two challenges it poses: scattering-arguments and multi-events. The first challenge means that arguments of one event record could reside in different sentences in the document, while the second one reflects that one document may simultaneously contain multiple such event records. Motivated by humans’ reading cognitive to extract information of interests, in this paper, we propose a method called HRE (Human Reading inspired Extractor for Document Events), where DEE is decomposed into these two iterative stages, rough reading and elaborate reading. Specifically, the first stage browses the document to detect the occurrence of events, and the second stage serves to extract specific event arguments. For each concrete event role, elaborate reading hierarchically works from sentences to characters to locate arguments across sentences, thus the scattering-arguments problem is tackled. Meanwhile, rough reading is explored in a multi-round manner to discover undetected events, thus the multi-events problem is handled. Experiment results show the superiority of HRE over prior competitive methods.

Recommender systems have been widely deployed in many real-world applications, but usually suffer from the long-standing user cold-start problem. As a promising way, Cross-Domain Recommendation (CDR) has attracted a surge of inter-est, which aims to transfer the user preferences observed in the source domain to make recommendations in the target domain. Previous CDR approaches mostly achieve the goal by following the Embedding and Mapping (EMCDR) idea which attempts to learn a mapping function to transfer the pre-trained user repre-sentations (embeddings) from the source domain into the target domain. However, they pre-train the user/item representations independently for each domain, ignoring to consider both domain interactions simultaneously. Therefore, the biased pre-trained representations inevitably involve the domain-specific information which may lead to negative impact to transfer information across domains. In this work, we consider a key point of the CDR task: what information needs to be shared across domains? To achieve the above idea, this paper utilizes the information bottleneck (IB) principle, and proposes a novel approach termed as CDRIB to enforce the representations encoding the domain-shared information. To derive the unbiased representations, we devise two IB regularizers to model the cross-domain/in-domain user-item interactions simultaneously and thereby CDRIB could consider both domain interactions jointly for de-biasing. With an additional contrastive information regularizer, CDRIB can also capture cross-domain user-user correlations. In this way, those regularizers encourage the representations to encode the domain-shared information, which has the capability to make recommendations in both domains directly. To the best of our knowledge, this paper is the first work to capture the domain-shared information for cold-start users via variational information bottleneck. Empirical experiments illustrate that CDRIB outperforms the state-of-the-art approaches on four real-world cross-domain datasets, demonstrating the effectiveness of adopting the information bottleneck for CDR.

Neural abstractive text summarization models have gained a lot of popularity. Recent researches on this tasks resort to a single generative model to generate high-quality summaries. However, people adopt adaptive strategies in the process of obtaining text abstracts, such as searching for combinations from textbooks or abstracts from domain knowledge. In this paper, we use dynamic reasoning to address abstractive summarization tasks. To achieve dynamic reasoning for high flexibility when faced with diverse summarizations, we propose a Learning-to-Learn Agent Adaption policy (LLAA-Policy) network to seamlessly adapt the selection over generative agent and retrieval agent, guided by the final summarization reward. Extensive experiments on the TT News Corpus dataset demonstrate the superiority of LLAA-Policy model over previous methods and the automatically learned policy reveals the underlying summarization aspects.

Building document-grounded dialogue systems have received growing interest as documents convey a wealth of human knowledge and commonly exist in enterprises. Wherein, how to comprehend and retrieve information from documents is a challenging research problem. Previous work ignores the visual property of documents and treats them as plain text, resulting in incomplete modality. In this paper, we propose a Layout-aware document-level Information Extraction dataset, LIE, to facilitate the study of extracting both structural and semantic knowledge from visually rich documents (VRDs), so as to generate accurate responses in dialogue systems. LIE contains 62k annotations of three extraction tasks from 4,061 pages in product and official documents, becoming the largest VRD-based information extraction dataset to the best of our knowledge. We also develop benchmark methods that extend the token-based language model to consider layout features like humans. Empirical results show that layout is critical for VRD-based extraction, and system demonstration also verifies that the extracted knowledge can help locate the answers that users care about.

Keyphrase extraction (KPE) aims to obtain a set of phrases from a document that can summarize the main content of the document.Recently, pre-trained language models (LMs), especially BERT and ELMo, have achieved remarkable success, presenting new state-of-the-art results in unsupervised KPE.However, current pre-trained LMs focus on building language modeling objectives to learn a general representation, ignoring the keyphrase-related knowledge.Intuitively, the joint embedding of the keyphrase set should tend to be close to that of the extracted document, and far from those of other documents.In this work, we propose a contrastive learning-based semantic representation task to further improve BERT for unsupervised KPE.Particularly, we design a doc-phrase attention module to generate joint semantic embedding of the keyphrase set as a positive sample and select other semantically similar documents as hard negative samples.In the prediction layer, we further add an accumulated self-attention module to calculate the final scores of candidate phrases.We compare with eight strong baselines, and evaluate our model on three publicly available datasets.Experimental results show that our model is effective and robust on both long and short documents.

Cross-domain Named Entity Recognition (NER) aims to transfer knowledge from the source domain to the target, alleviating expensive labeling costs in the target domain. Most prior studies acquire domain-invariant features under the end-to-end sequence-labeling framework where each token is assigned a compositional label (e.g., B-LOC). However, the complexity of cross-domain transfer may be increased over this complicated labeling scheme, which leads to sub-optimal results, especially when there are significantly distinct entity categories across domains. In this paper, we aim to explore the task decomposition in cross-domain NER. Concretely, we suggest a modular learning approach in which two sub-tasks (entity span detection and type classification) are learned by separate functional modules to perform respective cross-domain transfer with corresponding strategies. Compared with the compositional labeling scheme, the label spaces are smaller and closer across domains especially in entity span detection, leading to easier transfer in each sub-task. And then we combine two sub-tasks to achieve the final result with modular interaction mechanism, and deploy the adversarial regularization for generalized and robust learning in low-resource target domains. Extensive experiments over 10 diverse domain pairs demonstrate that the proposed method is superior to state-of-the-art cross-domain NER methods in an end-to-end fashion (about average 6.4% absolute F1 score increase). Further analyses show the effectiveness of modular task decomposition and its great potential in cross-domain NER.

Real-world web applications such as Amazon and Netflix often provide services in multiple countries and regions (i.e., markets) around the world. Generally, different markets share similar item sets while containing different amounts of interaction data. Some markets are data-scarce and others are data-rich and leveraging those data from similar and data-rich auxiliary markets could enhance the data-scarce markets. In this paper, we explore multi-market recommendation (MMR), and propose a novel model called M$^3$Rec to improve all markets recommendation simultaneously. Since items play the role to bridge different markets, we argue that mining the similarities among items is the key point of MMR. Our M^3Rec preprocess two global item similarities: intra- and inter- market similarities. Specifically, we first learn the second-order intra-market similarity by adopting linear models with closed-form solutions, and then capture the high-order inter-market similarity by the random walk. Afterward, we incorporate the global item similarities for each local market. We conduct extensive experiments on five public available markets and compare with several state-of-the-art methods. Detailed experimental results demonstrate the effectiveness of our proposed method.

Data sparsity is a long-standing problem in recommender systems. To alleviate it, Cross-Domain Recommendation (CDR) has attracted a surge of interests, which utilizes the rich user-item interaction information from the related source domain to improve the performance on the sparse target domain. Recent CDR approaches pay attention to aggregating the source domain information to generate better user representations for the target domain. However, they focus on designing more powerful interaction encoders to learn both domains simultaneously, but fail to model different user preferences of different domains. Particularly, domain-specific preferences of the source domain usually provide useless information to enhance the performance in the target domain, and directly aggregating the domain-shared and domain-specific information together maybe hurts target domain performance. This work considers a key challenge of CDR: How do we transfer shared information across domains? Grounded in the information theory, we propose DisenCDR, a novel model to disentangle the domain-shared and domain-specific information. To reach our goal, we propose two mutual-information-based disentanglement regularizers. Specifically, an exclusive regularizer aims to enforce the user domain-shared representations and domain-specific representations encoding exclusive information. An information regularizer is to encourage the user domain-shared representations encoding predictive information for both domains. Based on them, we further derive a tractable bound of our disentanglement objective to learn desirable disentangled representations. Extensive experiments show that DisenCDR achieves significant improvements over state-of-the-art baselines on four real-world datasets.

In few-shot relational triple extraction (FS-RTE), one seeks to extract relational triples from plain texts by utilizing only few annotated samples. Recent work first extracts all entities and then classifies their relations. Such an entity-then-relation paradigm ignores the entity discrepancy between relations. To address it, we propose a novel task decomposition strategy, Relation-then-Entity, for FS-RTE. It first detects relations occurred in a sentence and then extracts the corresponding head/tail entities of the detected relations. To instantiate this strategy, we further propose a model, RelATE, which builds a dual-level attention to aggregate relation-relevant information to detect the relation occurrence and utilizes the annotated samples of the detected relations to extract the corresponding head/tail entities. Experimental results show that our model outperforms previous work by an absolute gain (18.98%, 28.85% in F1 in two few-shot settings).

Event detection (ED) is a pivotal task for information retrieval, which aims at identifying event triggers and classifying them into pre-defined event types. In real-world applications, events are usually annotated with numerous fine-grained types, which often arises long-tail type nature and co-occurrence event nature. Existing studies explore the event correlations without full utilization, which may limit the capability of event detection. This paper simultaneously incorporates both the type-level and instance-level event correlations, and proposes a novel framework, termed as CorED. Specifically, we devise an adaptive graph-based type encoder to capture instance-level correlations, learning type representations not only from their training data but also from their relevant types, thus leading to more informative type representations especially for the low-resource types. Besides, we devise an instance interactive decoder to capture instance-level correlations, which predicts event instance types conditioned on the contextual typed event instances, leveraging co-occurrence events as remarkable evidence in prediction. We conduct experiments on two public benchmarks, MAVEN and ACE-2005 dataset. Empirical results demonstrate the unity of both type-level and instance-level correlations, and the model achieves effectiveness performance on both benchmarks.

Yucheng Wang, Bowen Yu, Hongsong Zhu, Tingwen Liu, Nan Yu, Limin Sun. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Open Information Extraction (OpenIE) aims to discover textual facts from a given sentence.In essence, the facts contained in plain text are unordered.However, the popular Ope-nIE systems usually output facts sequentially in the way of predicting the next fact conditioned on the previous decoded ones, which enforce an unnecessary order on the facts and involve the error accumulation between autoregressive steps.To break this bottleneck, we propose MacroIE, a novel non-autoregressive framework for OpenIE.MacroIE firstly constructs a fact graph based on the table filling scheme, in which each node denotes a fact element, and an edge links two nodes that belong to the same fact.Then OpenIE can be reformulated as a non-parametric process of finding maximal cliques from the graph.It directly outputs the final set of facts in one go, thus getting rid of the burden of predicting fact order, as well as the error propagation between facts.Experiments conducted on two benchmark datasets show that our proposed model significantly outperforms current state-of-theart methods, beats the previous systems by as much as 5.7 absolute gain in F1 score.

Distantly supervised named entity recognition (DS-NER) efficiently reduces labor costs but meanwhile intrinsically suffers from the label noise due to the strong assumption of distant supervision.Typically, the wrongly labeled instances comprise numbers of incomplete and inaccurate annotation noise, while most prior denoising works are only concerned with one kind of noise and fail to fully explore useful information in the whole training set.To address this issue, we propose a robust learning paradigm named Self-Collaborative Denoising Learning (SCDL), which jointly trains two teacher-student networks in a mutuallybeneficial manner to iteratively perform noisy label refinery.Each network is designed to exploit reliable labels via self denoising, and two networks communicate with each other to explore unreliable annotations by collaborative denoising.Extensive experimental results on five real-world datasets demonstrate that SCDL is superior to state-of-the-art DS-NER denoising methods 1 .

Recent deep learning based methods have achieved impressive performance on paraphrase identification (PI), a fundamental NLP task, judging whether two sentences are semantically equivalent or not. However, their success heavily relies on massive labeled samples, which are time-consuming and expensive to obtain. To alleviate this problem, this study explores the effect of word alignment information (WAI), extracted by existing monolingual alignment tools, on deep PI baseline models. Apart from directly encoding WAI into fixed-size embeddings, we propose a novel auxiliary task so that the baselines can be pre-trained using a large amount of unlabeled in-domain data. Moreover, our proposed auxiliary task can also jointly train with the baselines, aiming to eliminate the overheads of preprocessing WAI at the test period. Experimental results verify that our methods can significantly outperform the deep PI baseline model.

Bipartite graph embedding has recently attracted much attention due to the fact that bipartite graphs are widely used in various application domains. Most previous methods, which adopt random walk-based or reconstruction-based objectives, are typically effective to learn local graph structures. However, the global properties of bipartite graph, including community structures of homogeneous nodes and long-range dependencies of heterogeneous nodes, are not well preserved. In this paper, we propose a bipartite graph embedding called BiGI to capture such global properties by introducing a novel local-global infomax objective. Specifically, BiGI first generates a global representation which is composed of two prototype representations. BiGI then encodes sampled edges as local representations via the proposed subgraph-level attention mechanism. Through maximizing the mutual information between local and global representations, BiGI enables nodes in bipartite graph to be globally relevant. Our model is evaluated on various benchmark datasets for the tasks of top-K recommendation and link prediction. Extensive experiments demonstrate that BiGI achieves consistent and significant improvements over state-of-the-art baselines. Detailed analyses verify the high effectiveness of modeling the global properties of bipartite graph.

Open Information Extraction (OIE), the task aimed at discovering all textual facts organized in the form of (subject, predicate, object) found within a sentence, has gained much attention recently. However, in some knowledge-driven applications such as question answering, we often have a target entity and hope to obtain its structured factual knowledge for better understanding, instead of extracting all possible facts aimlessly from the corpus. In this paper, we define a new task, namely Semi-Open Information Extraction (SOIE), to address this need. The goal of SOIE is to discover domain-independent facts towards a particular entity from general and diverse web text. To facilitate research on this new task, we propose a large-scale human-annotated benchmark called SOIED, consisting of 61,984 facts for 8,013 subject entities annotated on 24,000 Chinese sentences collected from the web search engine.

Extracting relations from plain text is an important task with wide application. Most existing methods formulate it as a supervised problem and utilize one-hot hard labels as the sole target in training, neglecting the rich semantic information among relations. In this paper, we aim to explore the supervision with soft labels in relation extraction, which makes it possible to integrate prior knowledge. Specifically, a bipartite graph is first devised to discover type constraints between entities and relations based on the entire corpus. Then, we combine such type constraints with neural networks to achieve a knowledgeable model. Furthermore, this model is regarded as teacher to generate well-informed soft labels and guide the optimization of a student network via knowledge distillation. Besides, a multi-aspect attention mechanism is introduced to help student mine latent information from text. In this way, the enhanced student inherits the dark knowledge (e.g., type constraints and relevance among relations) from teacher, and directly serves the testing scenarios without any extra constraints. We conduct extensive experiments on the TACRED and SemEval datasets, the experimental results justify the effectiveness of our approach.

Most Chinese pre-trained models take character as the basic unit and learn representation according to character's external contexts, ignoring the semantics expressed in the word, which is the smallest meaningful utterance in Chinese.Hence, we propose a novel wordaligned attention to exploit explicit word information, which is complementary to various character-based Chinese pre-trained language models.Specifically, we devise a pooling mechanism to align the character-level attention to the word level and propose to alleviate the potential issue of segmentation error propagation by multi-source information fusion.As a result, word and character information are explicitly integrated at the fine-tuning procedure.Experimental results on five Chinese NLP benchmark tasks demonstrate that our method achieves significant improvements against BERT, ERNIE and BERT-wwm.

Extracting entities and relations from unstructured text has attracted increasing attention in recent years but remains challenging, due to the intrinsic difficulty in identifying overlapping relations with shared entities.Prior works show that joint learning can result in a noticeable performance gain.However, they usually involve sequential interrelated steps and suffer from the problem of exposure bias.At training time, they predict with the ground truth conditions while at inference it has to make extraction from scratch.This discrepancy leads to error accumulation.To mitigate the issue, we propose in this paper a one-stage joint extraction model, namely, TPLinker, which is capable of discovering overlapping relations sharing one or both entities while immune from the exposure bias.TPLinker formulates joint extraction as a token pair linking problem and introduces a novel handshaking tagging scheme that aligns the boundary tokens of entity pairs under each relation type.Experiment results show that TPLinker performs significantly better on overlapping and multiple relation extraction, and achieves state-of-the-art performance on two public datasets 1 .

Incorporating lexicons into character-level Chinese NER by lattices is proven effective to exploit rich word boundary information.Previous work has extended RNNs to consume lattice inputs and achieved great success.However, due to the DAG structure and the inherently unidirectional sequential nature, this method precludes batched computation and sufficient semantic interaction.In this paper, we propose PLTE, an extension of transformer encoder that is tailored for Chinese NER, which models all the characters and matched lexical words in parallel with batch processing.PLTE augments self-attention with positional relation representations to incorporate lattice structure.It also introduces a porous mechanism to augment localness modeling and maintain the strength of capturing the rich long-term dependencies.Experimental results show that PLTE performs up to 11.4 times faster than state-of-the-art methods while realizing better performance.We also demonstrate that using BERT representations further substantially boosts the performance and brings out the best in PLTE.

Dependency trees have been shown to be effective in capturing long-range relations between target entities.Nevertheless, how to selectively emphasize target-relevant information and remove irrelevant content from the tree is still an open problem.Existing approaches employing predefined rules to eliminate noise may not always yield optimal results due to the complexity and variability of natural language.In this paper, we present a novel architecture named Dynamically Pruned Graph Convolutional Network (DP-GCN), which learns to prune the dependency tree with rethinking in an end-to-end scheme.In each layer of DP-GCN, we employ a selection module to concentrate on nodes expressing the target relation by a set of binary gates and then augment the pruned tree with a pruned semantic graph to ensure the connectivity.After that, we introduce a rethinking mechanism to guide and refine the pruning operation by feeding back the high-level learned features repeatedly.Extensive experimental results demonstrate that our model achieves impressive performance compared to strong competitors.

Document-level relation extraction (RE)poses new challenges over its sentence-level counterpart since it requires an adequate comprehension of the whole document and the multi-hop reasoning ability across multiple sentences to reach the final result.In this paper, we propose a novel graphbased model with Dual-tier Heterogeneous Graph (DHG) for document-level RE.In particular, DHG is composed of a structure modeling layer followed by a relation reasoning layer.The major advantage is that it is capable of not only capturing both the sequential and structural information of documents but also mixing them together to benefit for multi-hop reasoning and final decisionmaking.Furthermore, we employ Graph Neural Networks (GNNs) based message propagation strategy to accumulate information on DHG.Experimental results demonstrate that the proposed method achieves state-of-the-art performance on two widely used datasets, and further analyses suggest that all the modules in our model are indispensable for document-level RE.

Joint extraction of entities and relations aims to detect entity pairs along with their relations using a single model. Prior work typically solves this task in the extract-then-classify or unified labeling manner. However, these methods either suffer from the redundant entity pairs, or ignore the important inner structure in the process of extracting entities and relations. To address these limitations, in this paper, we first decompose the joint extraction task into two interrelated subtasks, namely HE extraction and TER extraction. The former subtask is to distinguish all head-entities that may be involved with target relations, and the latter is to identify corresponding tail-entities and relations for each extracted head-entity. Next, these two subtasks are further deconstructed into several sequence labeling problems based on our proposed span-based tagging scheme, which are conveniently solved by a hierarchical boundary tagger and a multi-span decoding algorithm. Owing to the reasonable decomposition strategy, our model can fully capture the semantic interdependency between different steps, as well as reduce noise from irrelevant entity pairs. Experimental results show that our method outperforms previous work by 5.2%, 5.9% and 21.5% (F1 score), achieving a new state-of-the-art on three public datasets.

Few-shot Knowledge Graph (KG) completion is a focus of current research, where each task aims at querying unseen facts of a relation given its few-shot reference entity pairs.Recent attempts solve this problem by learning static representations of entities and references, ignoring their dynamic properties, i.e., entities may exhibit diverse roles within task relations, and references may make different contributions to queries.This work proposes an adaptive attentional network for few-shot KG completion by learning adaptive entity and reference representations.Specifically, entities are modeled by an adaptive neighbor encoder to discern their task-oriented roles, while references are modeled by an adaptive query-aware aggregator to differentiate their contributions.Through the attention mechanism, both entities and references can capture their fine-grained semantic meanings, and thus render more expressive representations.This will be more predictive for knowledge acquisition in the few-shot scenario.Evaluation in link prediction on two public datasets shows that our approach achieves new state-of-theart results with different few-shot sizes.The source code is available at https://github.

More recently, Named Entity Recognition has achieved great advances aided by pre-training approaches such as BERT.However, current pre-training techniques focus on building language modeling objectives to learn a general representation, ignoring the named entityrelated knowledge.To this end, we propose a NER-specific pre-training framework to inject coarse-to-fine automatically mined entity knowledge into pre-trained models.Specifically, we first warm-up the model via an entity span identification task by training it with Wikipedia anchors, which can be deemed as general-typed entities.Then we leverage the gazetteer-based distant supervision strategy to train the model extract coarse-grained typed entities.Finally, we devise a self-supervised auxiliary task to mine the fine-grained named entity knowledge via clustering.Empirical studies on three public NER datasets demonstrate that our framework achieves significant improvements against several pre-trained baselines, establishing the new state-of-the-art performance on three benchmarks.Besides, we show that our framework gains promising results without using human-labeled training data, demonstrating its effectiveness in labelfew and low-resource scenarios.1

Relation extraction and entity linking are two fundamental procedures to extend knowledge bases. Most existing methods typically treat them separately and ignore the semantic relevance between entities and relations. In this paper, we pioneer a general joint learning framework for relation extraction and entity linking, which allows these two tasks boost each other. Based on the framework, a demonstration model is proposed with neural networks. We conduct experiments on variants of a standard benchmark dataset (NYT-10) to verify the effectiveness of our approach. Experimental results show that our approach significantly outperforms traditional separate methods without reducing efficiency, especially on datasets with many ambiguous entity mentions. Furthermore, various mainstream methods for relation extraction and entity linking can be easily integrated into our loosely-coupled framework due to its flexible architecture.

In this paper, we study the effect of label noise on deep learning models for paraphrase identification. Curriculum learning, a learning paradigm that learns easy samples first and then gradually proceeds with hard ones, has shown excellent results in dealing with label noise for deep neural networks (DNNs). However, most previous studies focus on image classification, and design their curriculum only based on training losses of samples, ignoring domain-specific knowledge. In this paper, we propose a predefined curriculum learning based framework, incorporating both training losses of samples and domain-specific knowledge, to train robust deep models for paraphrase identification (PI) with label noise. Through extensive experiments on two popular PI benchmarks, we show that 1) the performance of the deep paraphrase identification model can drop sharply at the case of severe label noise; 2) our approach can significantly improve generalization performance of deep networks trained on corrupted data especially at extremely high levels of label noise; 3) our method can outperform several state-of-the-art label corruption robust methods.

Even with larger and larger text data available, encoding linguistic knowledge into word embeddings directly from corpora is still difficult. The most intuitive way to incorporate knowledge into word embeddings is to use external resources, such as the largest datasource for describing words - natural language dictionaries. However, previous methods usually neglect the recursive nature of dictionaries and cannot capture the high-order relation. In this paper, we present DRG2vec, a novel and efficient approach for learning word representations, which exploits the inherent recursiveness of dictionaries by modeling the whole dictionary as a homogeneous graph based on the co-occurrence of entry and word in the definition. Moreover, a tailor-made sampling strategy is introduced to generate word sequences from the definition relational graph, and then, the generated sequences are fed to the Skip-gram model with semantic negative sampling for word representation learning. Extensive experiments on sixteen benchmark datasets show that leveraging the recursive dictionary graph indeed achieves better performance than other state-of-the-art methods, especially exhibits a weighted average improvement of 8.6% in the word similarity task.

Self-attention based models like Transformer have achieved great success on kinds of Natural Language Processing tasks. However, the traditional fixed fully-connected structure faces many challenges in practice, such as computing redundancy, fixed granularity, and inexplicable. In this paper, we present BiG-Transformer, which employs attention with bipartite-graph structure to replace the fully-connected self-attention mechanism in Transformer. Specifically, two parts of the graph are designed for integrating hierarchical semantic information, and two types of connection are proposed to fuse information from different positions. Experiments on four tasks show the BiG-Transformer achieves better performance compared to Transformer liked models and Recurrent Neural Networks.

In a website fingerprinting attack, an eavesdropper analyses the traffic between the Tor user and entry node of the Tor network to infer which websites the user has visited. Some recent work apply deep learning algorithms, however, most of them do not fully exploit the packet timing information. In this work, we propose a novel website fingerprinting attack based on a two-channel Temporal Convolutional Networks model that extracts features from both the packet sequences and packet timing information. Our attack is proved to perform better compared to the state-of-the-art attacks. Experiment results also show that the timing information is very useful for classification. Furthermore, we collect our own traffic traces between client and entry node, and transform them into three extraction layers: TCP, TLS and Tor cell layer, and meanwhile record Tor’s cell log at the entry node. The experimental results show that the data of the cell layer is the most divisible among the three layers. Based on the experimental results, we conclude that the adversary at the entry node has an advantage over the one who just listens to traffic between client and entry node.

Relation extraction studies the issue of predicting semantic relations between pairs of entities in sentences. Attention mechanisms are often used in this task to alleviate the inner-sentence noise by performing soft selections of words independently. Based on the observation that information pertinent to relations is usually contained within segments (continuous words in a sentence), it is possible to make use of this phenomenon for better extraction. In this paper, we aim to incorporate such segment information into neural relation extractor. Our approach views the attention mechanism as linear-chain conditional random fields over a set of latent variables whose edges encode the desired structure, and regards attention weight as the marginal distribution of each word being selected as a part of the relational expression. Experimental results show that our method can attend to continuous relational expressions without explicit annotations, and achieve the state-of-the-art performance on the large-scale TACRED dataset.

Functional description information refers to the texts that describe the functionality or performance characteristics of a certain object. This type of information is of great potential value for the field of intelligence discovery. Thus automatically and accurately identifying this information from large amounts of texts on the web is very important. In this paper we reduce the functional description problem to a binary classification task deciding whether the input sentence is a functional description sentence or not. However, there exist lots of comment texts in the web data, which are semantically very similar to description texts, making our task quite difficult. Also, existing methods only provide general sentence representation models, which can't lead to targeted ways to solve our problem. Therefore, to address the problem, we not only exploit contexts, like many other previous work did, but also introduce indicator word information to learn rich representations. And in order to incorporate them both, we propose two models, namely ICNet(multi-tasks) and ICNet(ensemble). ICNet(multitasks) exploits them jointly in a integrated process of learning representations, while ICNet(ensemble) exploits them by two respective but concatenated sub-models. Experimental results on the collected real-world dataset indicate that both ICNet(multitasks) and ICNet(ensemble) achieve higher F1 scores compared with FaxtText, CNN, RNN, LSTM and Bi-LSTM, QuickThought models on this task.

Spoofed URLs are associated with various cyber crimes such as phishing and ransomware etc. Most existing detection approaches design a set of hand-crafted features and feed them to machine learning classifiers. However, designing such features is a time consuming and labor intensive process. This paper proposes an approach named NeuralAS (Neural Anti-Spoofing) by segmenting URLs into word sequences and detecting spoofed URLs with recurrent neural networks. As a result, NeuralAS can perform detection with high-abstract and poor-interpretable features learned automatically, and achieve accurate detection with contextual information in sequences. We also propose a novel method to construct indistinguishable data sets of strong similar samples, which can be used to evaluate the robustness of different approaches. Extensive experimental results show that NeuralAS works well on spoofed URLs detection, and has a significant effectiveness and robustness even on strong similar data sets.

In this paper, we propose the SignalCookie attack which can reveal guard relays of multiple hidden services in parallel. The key insight of our method is utilizing Rendezvous Cookie and circuit watermark to deliver the hidden services' identifiers to our controlled relays. By conducting the attack in parallel, the speed of our method increases about 11.6 times compared with the previous work, so that we can continually monitor the guard relays of 13604 hidden services for 7 months. And we have an interesting finding by analyzing the distribution of hidden services binding on each guard relays. The distribution is quite uneven, that only 20% of guard relays serve about 89.32% hidden services. These guard relays would bind all hidden services at least one time in about 17 months. At last, we analyze several security problems aggravated by the uneven distribution, and find that just one corrupt guard relay may cause hundreds of hidden services being de-anonymized or eclipsed.

With the rapidly evolving landscape of cyber threats, Indicators of Compromise (IOCs) are aggressively exchanged as forensic artifacts to help security professionals quickly identify and response cyber threats. Previous related studies mostly focus on extracting and generating IOCs from some fixed-point monitoring data sources, which are passive and time-consuming. In this paper, we present iMCircle, an innovation system that automatically mines IOCs from the Web by checking suspicious indicators with the help of open-source threat information. Based on the initial input of several suspicious indicators, iMCircle first collects their relevant public threat information from the Web and generates IOCs by checking whether those indicators are threat indicators in the target threat field. Second, it actively extracts new indicators from the search results as new inputs and checks them as described above. In that way, the system works in a circle and generates IOCs continuously. Running this system for almost two months in the real world, it has the appreciable performances on the active checking of suspicious indicators and the automatic generation of IOCs.

Opinion target extraction (OTE) is a fundamental step for sentiment analysis and opinion summarization. We analyze the difference between Chinese and the Indo-European languages family, and reduce Chinese OTE to a character-based sequence tagging task. Then we introduce two novel features for each character by distributing POS differentially and using predefined templates over contexts and dictionaries. We further propose a character-based BiLSTM-CRF model incorporating the two feature sequences aligned with the character sequence. Experimental results on real-world consumer review datasets show that our work significantly outperforms the baseline methods for Chinese OTE.

Expert finding (EF) task has received widespread attention as an important task of information retrieval.One key category of EF is expert finding on the web, which seeks to rank influential public figures from diverse webpage sources with respect to given query.Previous web expert finding approach relies on casting webpages to hypergraph structure and run heat diffusion process to find the top ranking person vertices according to their heat of popularity.Such approach suffer from two major drawbacks:First, previous web expert finding approach (CoDiffusion) suffers from unflexibility of selecting queries.This means that all corresponding queries must be stored as vertices in hypergraph index beforehand, otherwise CoDiffusion cannot run the expert finding process.Such defect make it ungeneric and incapable of handling the newly invented technical terms or phrases in real world scenarios.Second, the performance of previous approach is less satisfying.We incorporate semantic relatedness information with Hypergraph Learning Framework and Affinity Propagation ({HLFAP}) to handle the above drawbacks.In order to overcome the first disadvantage, we distribute initial heat to the related vertices according to their semantic similarity on given query. In order to solve the second disadvantage, we propose semantic labeled hypergraph learning framework and person influence affinity propagation model to make high quality candidates can receive more heat transition. Experimental results shows that our generic methodology achieves more satisfying results than the non-semantics state-of-the-art baseline method.

Cybercriminals use Malicious Uniform Resource Locators (URLs) as the entry to implement a variety of web attacks, such as phishing, spamming, and malware distribution, which may lead to huge finance and data loss. Thus, malicious URLs should be detected as accurately and quickly as possible. Heuristic-based detection approaches are one of the most popular methods to achieve the above goals. The detection results come from the usage of many heuristic features in this approach. However, tremendous new pages and meaningless tokens lead to the explosion of feature sets, and exhaust memory space finally. In this paper, we try to address the problem by selecting some representative members from the initial feature set, which should have the best predictive ability among the same number of selected features. For each feature, we give an evaluation method of O(1) complexity to measure its predictive ability. Then we make the selection based on all the measured values with linear complexity. Experimental results show that our approach can achieve almost the same false negative rate using only 8.3% features for malicious URLs detection, comparing with prior approaches. Moreover, our approach may work efficiently in the big data era, as it can handle 20 thousand URLs per second in our experiments on average.

Email has become widely used in recent years bringing with it new problems. Although this event doesn't happen often, misdirected emails can bring out great information leakage. It is not easy to detect these misdirected emails from legitimate ones since they may be only distinguishable in the sender's perspective. Existing methods discover misdirected emails from user agent or gateway but are not appropriate for varied application environment. This paper proposes a misdirected mail detection method based on multi-attributes which can be deployed on server side. Three type of attributes including email content fingerprinting, social relationship and meta information are considered in this method. Based on SVM classification algorithm, experiments show that it can detect misdirected emails with up to 91.6% accuracy.

Email plays a very important role in our daily life. Much work have been put into practice on email network. Those studies mostly require real email network datasets and reliable models to analyze user information and understand the mechanisms of network evolution. However, much research work is constrained by the absence of real large-scale email datasets. Although email communication is ubiquitous, there are very few large-scale available email datasets satisfied different research purposes. Due to privacy policy and restricted permissions, it is arduous to collect a real large-scale email dataset in a short time. Various social network models are usually used to create synthetic email networks. However, these models focus on modeling several structural properties of network without considering user behaviour patterns. They are not appropriate to generate large-scale realistic synthetic email network datasets. Towards this end, we propose a probabilistic model by which we can construct large-scale synthetic email datasets with a small captured email log. What is more important is that the generated synthetic dataset matches real email network properties and individual communication patterns. Moreover, it has linear complexity, and can be paralleled easily. Experimental results on Enron dataset demonstrate the above benefits of our model.

With the widespread usage of emails, information leakage via misdirected emails becomes a practical and disastrous problem, which should be addressed at all costs. Prior methods have two limitations: privacy issue as relying on email contents to work, and high cost as building too many targeted models. In this paper, we reduce the detection of misdirected emails to a binary classification problem, and build only a universal model to detect misdirected emails. We introduce some representative features that can vividly describe the characteristics of misdirected emails while not infringe users' privacy. Then we design novel algorithms to get these features. The random forest classifier is chosen to perform the detecting task. Experimental results show that our work is able to detect misdirected emails with 89% precision rate and 82% recall rate in average.

Regular Expression (RegEx) matching, as a core operation in many network and security applications, is typically performed on Deterministic Finite Automata (DFA) to process packets at wire speed; however, DFA size is often exponential in the number of RegExes. RegEx grouping is the practical way to address DFA state explosion. Prior RegEx grouping algorithms are extremely slow and memory intensive. In this paper, we first propose DFAestimator, an algorithm that can quickly estimate DFA size for a given RegEx set without building the actual DFA. Second, we propose RegexGrouper, a RegEx grouping algorithm based on DFA size estimation. In terms of speed and memory consumption, our work is orders of magnitude more efficient than prior art because DFA size estimation is much faster and memory efficient than DFA construction. In terms of the resulting size sum of DFAs, our work is significantly more effective than prior art because we use a much finer grained quantification of the degree of interaction between two RegExes. For example, to divide the RegEx set of the L7-filter system into 7 groups, prior art uses 279.3 minutes and the resulting 7 DFAs have a total of 29047 states, whereas RegexGrouper uses 3.2 minutes and the resulting 7 DFAs have a total of 15578 states.

Deep packet inspection plays a increasingly important role in network security devices and applications, which use more regular expressions to depict patterns. DFA engine is usually used as a classical representation for regular expressions to perform pattern matching, because it only need O(1) time to process one input character. However, DFAs of regular expression sets require large amount of memory, which limits the practical application of regular expressions in high-speed networks. Some compression algorithms have been proposed to address this issue in recent literatures. In this paper, we reconsider this problem from a new perspective, namely observing the characteristic of transition distribution inside each state, which is different from previous algorithms that observe transition characteristic among states. Furthermore, we introduce a new compression algorithm which can reduce 95% memory usage of DFA stably without significant impact on matching speed. Moreover, our work is orthogonal to previous compression algorithms, such as D2FA, δFA. Our experiment results show that applying our work to them will have several times memory reduction, and matching speed of up to dozens of times comparing with original δFA in software implementation.

Traffic classification is important to many network applications, such as network monitoring. The classic way to identify flows, e.g., examining the port numbers in the packet headers, becomes ineffective. In this context, deep packet inspection technology, which does not only inspect the packet headers but also the packet payloads, plays a more important role in traffic classification. Meanwhile regular expressions are replacing strings to represent patterns because of their expressive power, simplicity and flexibility. However, regular expressions mathcing technique causes a high memory usage and processing cost, which result in low throughout. In this paper, we analyze the application-level protocol distribution of network traffic and conclude its characteristic. Furthermore, we design a fast and memory-efficient system of a two-layer architecture for traffic classification with the help of regular expressions in multi-core architecture, which is different from previous one-layer architecture. In order to reduce the memory usage of DFA, we use a compression algorithm called CSCA to perform regular expressions matching, which can reduce 95% memory usage of DFA. We also introduce some optimizations to accelerate the matching speed. We use real-world traffic and all L7-filter protocol patterns to make our experiments, and the results show that the system achieves at Gbps level throughout in 4-cores Servers.

Load balancing is critical to the performance of parallel processing systems. It is more difficult for network systems such as NIDS and Web Servers, because they must preserve flow order. But traditional flow-based load balancing schemes of network parallel processing systems, such as LLF, cost much resource and introduce lots of communication overhead. With the rapid popularization of multi-core system, it is a good choice to apply NIDS in CMP architecture to achieve higher performance. Some companies have taken the first step. Their scheduling algorithm operates at a custom NIC based on FPGA technology. LLF algorithm can't be used in such environment because it needs much more memory than NIC owns. In this paper, we propose a scheduling scheme that re-maps the new arrival flows when the system is unbalanced. To make effective adjustments we design a new triggering policy based on waiting lengths and their difference. Compared with LLF, our algorithm costs about 5% memory to get the same performance.