Multi-Hop Question Answering (MHQA) is crucial for evaluating the model's capability to integrate information from diverse sources.However, creating extensive and high-quality MHQA datasets is challenging: (i) manual annotation is expensive, and (ii) current synthesis methods often produce simplistic questions or require extensive manual guidance.This paper introduces HopWeaver, the first cross-document framework synthesizing authentic multi-hop questions without human intervention.HopWeaver synthesizes bridge and comparison questions through an innovative pipeline that identifies complementary documents and constructs authentic reasoning paths to ensure true multi-hop reasoning.We further present a comprehensive system for evaluating the synthesized multi-hop questions.Empirical evaluations demonstrate that the synthesized questions achieve comparable or superior quality to human-annotated datasets at a lower cost.Our framework provides a valuable tool for the research community: it can automatically generate challenging benchmarks from any raw corpus, which opens new avenues for both evaluation and targeted training to improve the reasoning capabilities of advanced question answering models, especially in domains with scarce resources.The code for HopWeaver is publicly available 1 .

Maintaining consistency in long-term dialogues remains a fundamental challenge for LLMs, as standard retrieval mechanisms often fail to capture the temporal evolution of historical states.While memory-augmented frameworks offer a structured alternative, current systems rely on static prompting of closed-source models or suffer from ineffective training paradigms with sparse rewards.We introduce MemBuilder, a reinforcement learning framework that trains models to orchestrate multi-dimensional memory construction with attributed dense rewards.MemBuilder addresses two key challenges: (1) Sparse Trajectory-Level Rewards: we employ synthetic session-level question generation to provide dense intermediate rewards across extended trajectories; and (2) Multi-Dimensional Memory Attribution: we introduce contribution-aware gradient weighting that scales policy updates based on each component's downstream impact.Experimental results show that MemBuilder enables a 4Bparameter model to outperform state-of-the-art closed-source baselines, exhibiting strong generalization across long-term dialogue benchmarks. 1

This article aims to speed up the training of large neural networks with the Mixture-of-Experts (MoE) structure. Training MoE often needs a lot of computing resources due to its large scale. Traditional acceleration methods either degrade prediction performance or rely on dedicated hardware with additional resources, but the resources are usually limited in real applications.One solution is to resort to new optimization strategies, such as learning from easy to hard by multiple stages. However, existing strategies are designed mainly for networks with a serial structure, but MoE has multiple expert networks working in parallel. They employ an identical learning plan for all experts, ignoring that each expert's learning domain and speed differ, resulting in some experts being over-learned while others being under-learned. This mismatch will make it hard for experts to train together, harming training efficiency. To address this problem, we propose a new training acceleration framework. It can customize an effective learning plan for each expert by considering their training progress, avoiding blindly searching in a huge parameter space. In detail, we first design a multi-stage planner that starts with optimizing a subpart of the network and then scales it up to retrain until it expands to an entire network. It uses the density function to assess the knowledge gained by the expert in each stage, giving priority to the experts who learn faster to increase the training scale, so as to boost convergence. Afterward, we exploit the growth operator to add the expert training scale of the next stage. In each stage, the network would converge to some locally optimal values. That can provide a better initialization to train the next stage more easily, since the time and data required for training from scratch are greatly reduced. To alleviate the gradient vanishing problem caused by network growth, we develop a scheduler to dynamically adjust the learning rate. Extensive experiments are conducted to validate the effectiveness of our method. The results show that we can obtain more than 25% training acceleration on average.

Single-cell annotation is a fundamental task in the analysis of single-cell data, and one promising research direction relies on the marker gene information accumulated in biology. Recently, self-training strategies have been introduced into the field, which significantly improve the annotation accuracy by iteratively optimizing the model. However, existing methods have not yet systematically explored how to construct self-training frameworks that are more applicable to single-cell data. To this end, we propose a context-aware self-training model named CSSTA. Firstly, the contextual information of marker genes is introduced to enhance the compatibility of marker genes with different single-cell datasets to generate high-quality pseudo-labels. Secondly, high- and low-confidence pseudo-labels recognition and supervision strategies more applicable to single-cell data are designed that can better guide the optimization of the model. Finally, the insight of the single-cell foundation model on cell-cell association information is introduced. Experiments demonstrate that the introduction of marker gene contextual information significantly improves the ability to recognize cell-cell type associations with heuristic-based strategies. Benchmark experiments show that CSSTA significantly outperforms state-of-the-art baselines. Notably, we demonstrate the potential of CSSTA for hierarchical cellular annotation by extending it to hierarchies.

Fan Li, Jianxing Yu, Jielong Tang, Wenqing Chen, Hanjiang Lai, Yanghui Rao, Jian Yin. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Emotion Recognition in Conversations (ERC) is crucial for machines to understand dynamic human emotions. While Large Language Models (LLMs) show promise, their performance is often limited by challenges in interpreting complex conversational streams. We introduce a Clue of Emotion (CoE) framework, which progressively integrates key conversational clues to enhance the ERC task. Building on CoE, we implement a multi-stage auxiliary learning strategy that incorporates role-playing, speaker identification, and emotion reasoning tasks, each targeting different aspects of conversational emotion understanding and enhancing the model’s ability to interpret emotional contexts. Our experiments on EmoryNLP, MELD, and IEMOCAP demonstrate that CoE consistently outperforms state-of-the-art methods, achieving a 2.92% improvement on EmoryNLP. These results underscore the effectiveness of clues and multi-stage auxiliary learning for ERC, offering valuable insights for future research.

This paper focuses on sarcasm detection, which aims to identify whether given statements convey criticism, mockery, or other negative sentiment opposite to the literal meaning. To detect sarcasm, humans often require a comprehensive understanding of the semantics in the statement and even resort to external commonsense to infer the fine-grained incongruity. However, existing methods lack commonsense inferential ability when they face complex real-world scenarios, leading to unsatisfactory performance. To address this problem, we propose a novel framework for sarcasm detection, which conducts incongruity reasoning based on commonsense augmentation, called EICR. Concretely, we first employ retrieval-augmented large language models to supplement the missing but indispensable commonsense background knowledge. To capture complex contextual associations, we construct a dependency graph and obtain the optimized topology via graph refinement. We further introduce an adaptive reasoning skeleton that integrates prior rules to extract sentiment-inconsistent subgraphs explicitly. To eliminate the possible spurious relations between words and labels, we employ adversarial contrastive learning to enhance the robustness of the detector. Experiments conducted on five datasets demonstrate the effectiveness of EICR.

This paper studies the task of generating commonsense reasoning questions (QG) with desired difficulty levels. Compared to traditional shallow questions that can be solved by simple term matching, ours are more challenging. Our answering process requires reasoning over multiple contextual and commonsense clues. That involves advanced comprehension skills, such as abstract semantics learning and missing knowledge inference. Existing work mostly learns to map the given text into questions, lacking a mechanism to control results with the desired complexity. To address this problem, we propose a novel controllable framework. We first derive contextual and commonsense clues involved in reasoning questions from the text. These clues are used to create simple sub-questions. We then aggregate multiple sub-questions to compose complex ones under the guidance of prior reasoning structures. By iterating this process, we can compose a complex QG task based on a series of smaller and simpler QG subtasks. Each subtask serves as a building block for a larger one. Each composition corresponds to an increase in the reasoning step. Moreover, we design a voting verifier to ensure results’ validity from multiple views, including answer consistency, reasoning difficulty, and context correlation. Finally, we can learn the optimal QG model to yield thought-provoking results. Evaluations on two typical datasets validate our method.

Disagreement detection is a crucial task in natural language processing (NLP), particularly in analyzing online discussions and social media content.Large language models (LLMs) have demonstrated significant advancements across various NLP tasks.However, the performance of LLMs in disagreement detection is limited by two issues: conceptual gap and reasoning gap.In this paper, we propose a novel two-stage framework, Concept Alignment and Reasoning Enhancement (CARE), to tackle the issues.The first stage, Concept Alignment, addresses the gap between expert and model by performing sub-concept taxonomy extraction, aligning the model's comprehension with human experts.The second stage, Reasoning Enhancement, improves the model's reasoning capabilities by introducing curriculum learning workflow, which includes rationale to critique and counterfactual to detection for reducing spurious association.Extensive experiments on disagreement detection task demonstrate the effectiveness of our framework, showing superior performance in zero-shot and supervised learning settings, both within and across domains.

Topic modeling is a powerful unsupervised tool for knowledge discovery.However, existing work struggles with generating limited-quality topics that are uninformative and incoherent, which hinders interpretable insights from managing textual data.In this paper, we improve the original variational autoencoder framework by incorporating contextual and graph information to address the above issues.First, the encoder utilizes topic fusion techniques to combine contextual and bag-of-words information well, and meanwhile exploits the constraints of topic alignment and topic sharpening to generate informative topics.Second, we develop a simple word co-occurrence graph information fusion strategy that efficiently increases topic coherence.On three benchmark datasets, our new framework generates more coherent and diverse topics compared to various baselines, and achieves strong performance on both automatic and manual evaluations.We make our code available for reproduction 1 .

The isomorphism problem is a key challenge in both graph and hypergraph domains, crucial for applications like protein design, chemical pathways, and community detection. Hypergraph isomorphism, which models high-order relationships in real-world scenarios, remains underexplored compared to the graph isomorphism. Current algorithms for hypergraphs, like the 1-dimensional generalized Weisfeiler-Lehman test (1-GWL), lag behind advancements in graph isomorphism tests, limiting most hypergraph neural networks to 1-GWL's expressive power. To address this, we propose the high-dimensional GWL (k-GWL), generalizing k-WL from graphs to hypergraphs. We prove that k-GWL reduces to k-WL for simple graphs, and thus develop a unified isomorphism method for both graphs and hypergraphs. We also successfully establish a clear and complete understanding of the GWL hierarchy of expressivity, showing that (k+1)-GWL is more expressive than k-GWL with illustrative examples. Based on k-GWL, we develop a hypergraph neural network model named k-HNN with improved expressive power of k-GWL, which achieves superior performance on real-world datasets, including a 6\% accuracy improvement on the Steam-Player dataset over the runner-up. Our code is available at https://github.com/talence-zcq/KGWL.

Anomaly detection on text-rich graphs is widely prevalent in real life, such as detecting incorrectly assigned academic papers to authors and detecting bots in social networks. The remarkable capabilities of large language models (LLMs) pave a new revenue by utilizing rich-text information for effective anomaly detection. However, simply introducing rich texts into LLMs can obscure essential detection cues and introduce high fine-tuning costs. Moreover, LLMs often overlook the intrinsic structural bias of graphs which is vital for distinguishing normal from abnormal node patterns. To this end, this paper introduces GuARD, a text-rich and graph-informed language model that combines key structural features from graph-based methods with fine-grained semantic attributes extracted via small language models for effective anomaly detection on text-rich graphs. GuARD is optimized with the progressive multimodal multi-turn instruction tuning framework in the task-guided instruction tuning regime tailed to incorporate both rich-text and structural modalities. Extensive experiments on four datasets reveal that GuARD outperforms graph-based and LLM-based anomaly detection methods, while offering up to 5× speedup in training and 10× speedup in inference over vanilla long-context LLMs on the large-scale WhoIsWho dataset.

Abstract Neural topic modeling is a scalable automated technique for text data mining. In various downstream tasks of topic modeling, it is preferred that the discovered topics well align with labels. However, due to the lack of guidance from labels, unsupervised neural topic models are less powerful in this situation. Existing supervised neural topic models often adopt a label-free prior to generate the latent document-topic distributions and use them to predict the labels and thus achieve label-topic alignment indirectly. Such a mechanism faces the following issues: 1) The label-free prior leads to topics blending the latent patterns of multiple labels; and 2) One is unable to intuitively identify the explicit relationships between labels and the discovered topics. To tackle these problems, we develop a novel supervised neural topic model which utilizes a chain-structured graphical model with a label-conditioned prior. Soft indicators are introduced to explicitly construct the label-topic relationships. To obtain well-organized label-topic relationships, we formalize an entropy-regularized optimal transport problem on the embedding space and model them as the transport plan. Moreover, our proposed method can be flexibly integrated with most existing unsupervised neural topic models. Experimental results on multiple datasets demonstrate that our model can greatly enhance the alignment between labels and topics while maintaining good topic quality.

Complex Query Answering (CQA) on knowledge graphs is a fundamental yet challenging task, which can be formalized as answering a subset of first-order logic queries containing logical conjunction, disjunction, negation, and existential quantifiers. Recent research reveals that Link Predictors (LPs) trained on 1-hop queries can generalize to various types of complex queries. However, existing methods neglect crucial characteristics of LPs' outputs, including the effects of highly relevant entities and uncertainty. What's worse, as they model logical operations by fuzzy set operations, these methods suffer from problems like inflexibility, sensitivity to noise, and inconsistency with priority in human cognition, which limits their performance, especially on queries with negation. To address these challenges, we propose an efficient fuzzy system for CQA that requires no extra training overheads and is plug-and-play with existing LP-based methods. Firstly, we expand the output of LPs by two complementary membership functions of weak and strong relevance, which help to distinguish the target entities from highly relevant and irrelevant entities. Subsequently, we model logical operations through fuzzy rule bases and infer the final predictions via defuzzification, providing a flexible and tractable scheme for modeling logical operations. Finally, the effectiveness of the proposed fuzzy system is validated by its outstanding performance on benchmark datasets when compared to state-of-theart methods. The source code of our proposed method is available at https://anonymous.4open.science/r/FuzzSys-CQA-E285.

Yufeng Zhang, Jianxing Yu, Yanghui Rao, Libin Zheng, Qinliang Su, Huaijie Zhu, Jian Yin. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Hierarchical topic modeling, which can mine implicit semantics in the corpus and automatically construct topic hierarchical relationships, has received considerable attention recently. However, the current hierarchical topic models are mainly based on Euclidean space, which cannot well retain the implicit hierarchical semantic information in the corpus, leading to irrational structure of the generated topics. On the other hand, the existing Generative Adversarial Network (GAN) based neural topic models perform satisfactorily, but they remain constrained by pattern collapse due to the discontinuity of latent space. To solve the above problems, with the hypothesis of hyperbolic space, we propose a novel GAN-based hierarchical topic model to mine high-quality topics by introducing contrastive learning to capture information from documents. Furthermore, the distinct tree-like property of hyperbolic space preserves the implicit hierarchical semantics of documents in topic embeddings, which are projected into the hyperbolic space. Finally, we use a multi-head self-attention mechanism to learn implicit hierarchical semantics of topics and mine topic structure information. Experiments on real-world corpora demonstrate the remarkable performance of our model on topic coherence and topic diversity, as well as the rationality of the topic hierarchy.

This paper focuses on detecting clickbait posts on the Web.These posts often use eye-catching disinformation in mixed modalities to mislead users to click for profit.That affects the user experience and thus would be blocked by content provider.To escape detection, malicious creators use tricks to add some irrelevant nonbait content into bait posts, dressing them up as legal to fool the detector.This content often has biased relations with non-bait labels, yet traditional detectors tend to make predictions based on simple co-occurrence rather than grasping inherent factors that lead to malicious behavior.This spurious bias would easily cause misjudgments.To address this problem, we propose a new debiased method based on causal inference.We first employ a set of features in multiple modalities to characterize the posts.Considering these features are often mixed up with unknown biases, we then disentangle three kinds of latent factors from them, including the invariant factor that indicates intrinsic bait intention; the causal factor which reflects deceptive patterns in a certain scenario, and non-causal noise.By eliminating the noise that causes bias, we can use invariant and causal factors to build a robust model with good generalization ability.Experiments on three popular datasets show the effectiveness of our approach.

Recent advances in microfluidics and sequencing technologies allow researchers to explore cellular heterogeneity at single-cell resolution. In recent years, deep learning frameworks, such as generative models, have brought great changes to the analysis of transcriptomic data. Nevertheless, relying on the potential space of these generative models alone is insufficient to generate biological explanations. In addition, most of the previous work based on generative models is limited to shallow neural networks with one to three layers of latent variables, which may limit the capabilities of the models. Here, we propose a deep interpretable generative model called d-scIGM for single-cell data analysis. d-scIGM combines sawtooth connectivity techniques and residual networks, thereby constructing a deep generative framework. In addition, d-scIGM incorporates hierarchical prior knowledge of biological domains to enhance the interpretability of the model. We show that d-scIGM achieves excellent performance in a variety of fundamental tasks, including clustering, visualization, and pseudo-temporal inference. Through topic pathway studies, we found that d-scIGM-learned topics are better enriched for biologically meaningful pathways compared to the baseline models. Furthermore, the analysis of drug response data shows that d-scIGM can capture drug response patterns in large-scale experiments, which provides a promising way to elucidate the underlying biological mechanisms. Lastly, in the melanoma dataset, d-scIGM accurately identified different cell types and revealed multiple melanin-related driver genes and key pathways, which are critical for understanding disease mechanisms and drug development.

MOTIVATION: Recent advances in sequencing technology provide opportunities to study biological processes at a higher resolution. Cell type annotation is an important step in scRNA-seq analysis, which often relies on established marker genes. However, most of the previous methods divide the identification of cell types into two stages, clustering and assignment, whose performances are susceptible to the clustering algorithm, and the marker information cannot effectively guide the clustering process. Furthermore, their linear heuristic-based cell assignment process is often insufficient to capture potential dependencies between cells and types. RESULTS: Here, we present Interpretable Cell Type Annotation based on self-training (sICTA), a marker-based cell type annotation method that combines the self-training strategy with pseudo-labeling and the nonlinear association capturing capability of Transformer. In addition, we incorporate biological priori knowledge of genes and pathways into the classifier through an attention mechanism to enhance the transparency of the model. A benchmark analysis on 11 publicly available single-cell datasets demonstrates the superiority of sICTA compared to state-of-the-art methods. The robustness of our method is further validated by evaluating the prediction accuracy of the model on different cell types for each single-cell data. Moreover, ablation studies show that self-training and the ability to capture potential dependencies between cells and cell types, both of which are mutually reinforcing, work together to improve model performance. Finally, we apply sICTA to the pancreatic dataset, exemplifying the interpretable attention matrix captured by sICTA. AVAILABILITY AND IMPLEMENTATION: The source code of sICTA is available in public at https://github.com/nbnbhwyy/sICTA. The processed datasets can be found at https://drive.google.com/drive/folders/1jbqSxacL_IDIZ4uPjq220C9Kv024m9eL. The final version of the model will be permanently available at https://doi.org/10.5281/zenodo.13474010.

Abstract Topic taxonomy discovery aims at uncovering topics of different abstraction levels and constructing hierarchical relations between them. Unfortunately, most prior work can hardly model semantic scopes of words and topics by holding the Euclidean embedding space assumption. What’s worse, they infer asymmetric hierarchical relations by symmetric distances between topic embeddings. As a result, existing methods suffer from problems of low-quality topics at high abstraction levels and inaccurate hierarchical relations. To alleviate these problems, this paper develops a Box embedding-based Topic Model (BoxTM) that maps words and topics into the box embedding space, where the asymmetric metric is defined to properly infer hierarchical relations among topics. Additionally, our BoxTM explicitly infers upper-level topics based on correlation between specific topics through recursive clustering on topic boxes. Finally, extensive experiments validate high-quality of the topic taxonomy learned by BoxTM.

Hierarchical topic models, which can extract semantically meaningful topics from a text corpus in an unsupervised manner and automatically organise them into a topic hierarchy, have been widely used to discover the underlying semantic structure of documents.However, the existing models often assume in the prior that the topic hierarchy is a tree structure, ignoring symmetrical dependencies between topics at the same level.Moreover, the sparsity of text data often complicate the analysis.To address these issues, we propose NSEM-GMHTM as a deep topic model, with a Gaussian mixture prior distribution to improve the model's ability to adapt to sparse data, which explicitly models hierarchical and symmetric relations between topics through the dependency matrices and nonlinear structural equations.Experiments on widely used datasets show that our NSEM-GMHTM generates more coherent topics and a more rational topic structure when compared to state-of-theart baselines.Our code is available at https: //github.com/nbnbhwyy/NSEM-GMHTM. W3 W2 W1

Multi-hop QA requires reasoning over multiple supporting facts to answer the question.However, the existing QA models always rely on shortcuts, e.g., providing the true answer by only one fact, rather than multi-hop reasoning, which is referred to as disconnected reasoning problem.To alleviate this issue, we propose a novel counterfactual multihop QA, a causaleffect approach that enables to reduce the disconnected reasoning.It builds upon explicitly modeling of causality: 1) the direct causal effects of disconnected reasoning and 2) the causal effect of true multi-hop reasoning from the total causal effect.With the causal graph, a counterfactual inference is proposed to disentangle the disconnected reasoning from the total causal effect, which provides us a new perspective and technology to learn a QA model that exploits the true multi-hop reasoning instead of shortcuts.Extensive experiments have been conducted on the benchmark HotpotQA dataset, which demonstrate that the proposed method can achieve notable improvement on reducing disconnected reasoning.For example, our method achieves 5.8% higher points of its Supp s score on HotpotQA through true multihop reasoning.The code is available at https://github.com/guowzh/CFMQA.

The out-of-vocabulary (OOV) words are difficult to represent while critical to the performance of embedding-based downstream models.Prior OOV word embedding learning methods failed to model complex word formation well.In this paper, we propose a novel graph-based relation mining method, namely GRM, for OOV word embedding learning.We first build a Word Relationship Graph (WRG) based on word formation and associate OOV words with their semantically relevant words, which can mine the relational information inside word structures.Subsequently, our GRM can infer high-quality embeddings for OOV words through passing and aggregating semantic attributes and relational information in the WRG, regardless of contextual richness.Extensive experiments demonstrate that our model significantly outperforms state-of-theart baselines on both intrinsic and downstream tasks when faced with OOV words.

In terms of the generative process, the Gamma-Gamma-Poisson Process (G2PP) is equivalent to the nonparametric topic model of Hierarchical Dirichlet Process (HDP). Considering the high computational cost of estimating parameters in HDP, a parallel G2PP was developed to generate topics efficiently via multi-threading. Unfortunately, the above model needs to predefine the number of topics. To address this issue, we first propose a Topic Self-Adaptive Model (TSAM) for nonparametric and parallel topic discovery. In TSAM, a monitor-executor mechanism is developed to manage the global topic information using a hierarchical structure of threads. Based on the apparatus of copulas, we further extend our TSAM to TSAMcop for coherent topic modeling by exploiting a copula guided parallel Gibbs sampling algorithm. Extensive experiments validate the effectiveness of both TSAM and TSAMcop.

Combination therapy, which can improve therapeutic efficacy and reduce side effects, plays an important role in the treatment of complex diseases. Yet, a large number of possible combinations among candidate compounds limits our ability to identify effective combinations. Though many studies have focused on predicting potential drug combinations, the existing methods are not entirely satisfactory in terms of performance and scalability. In this study, we propose a new computational pipeline, called DCMGCN, which integrates diverse drug-related information, to predict novel drug combinations. Specifically, DCMGCN first learns low-dimensional representations of drugs from the drug attributes and similarity networks. Then, by quantifying the degree of the nodes in the known drug-drug network and the similarity between connected nodes, we found the drug-drug network has heterophily and sparseness, which may limit the effectiveness of the graph convolutional network (GCN). Therefore, we introduce two designs to modify GCN. Finally, the drug representations are optimized using modified GCN (MGCN) and used to predict drug combinations. The tests on multiple drug combination datasets show that DCMGCN achieved substantial improvements over state-of-the-art methods. Importantly, our model may embed the mechanism of ground-truth drug pairs into the low-dimensional representation of each drug, which may help to further clarify the understanding of mechanisms of drug action.

In this study, sentiment classification and emotion distribution learning across domains are both formulated as a semi-supervised domain adaptation problem, which utilizes a small amount of labeled documents in the target domain for model training. By introducing a shared matrix that captures the stable association between document clusters and word clusters, non-negative matrix tri-factorization (NMTF) is robust to the labeled target domain data and has shown remarkable performance in cross-domain text classification. However, the existing NMTF-based models ignore the incompatible relationship of sentiment polarities and the relatedness among emotions. Besides, their applications on large-scale datasets are limited by the high computation complexity. To address these issues, we propose a semi-supervised NMTF framework for sentiment classification and emotion distribution learning across domains. Based on a many-to-many mapping between document clusters and sentiment polarities (or emotions), we first incorporate the prior information of label dependency to improve the model performance. Then, we develop a parallel algorithm based on message passing interface (MPI) to further enhance the model scalability. Extensive experiments on real-world datasets validate the effectiveness of our method.

As a novel paradigm for data mining and dimensionality reduction, Non-negative Matrix Tri-Factorization (NMTF) has attracted much attention due to its notable performance and elegant mathematical derivation, and it has been applied to a plethora of real-world applications, such as text data co-clustering. However, the existing NMTF-based methods usually involve intensive matrix multiplications, which exhibits a major limitation of high computational complexity. With the explosion at both the size and the feature dimension of texts, there is a growing need to develop a parallel and scalable NMTF-based algorithm for text data co-clustering. To this end, we first show in this paper how to theoretically derive the original optimization problem of NMTF by introducing the Lagrangian multipliers. Then, we propose to solve the Lagrange dual objective function in parallel through an efficient distributed implementation. Extensive experiments on five benchmark corpora validate the effectiveness, efficiency, and scalability of our distributed parallel update algorithm for an NMTF-based text data co-clustering method.

Aiming at mining high quality topics by accumulating and utilizing semantic knowledge for a stream of documents, lifelong topic modeling (LTM) has attracted more and more attentions recently. However, the permutation of topics may change over time, resulting in asemantic misalignmentbetween the topic representations of document chunks across the stream. Such a misalignment deteriorates the model performances of various downstream tasks, while it has been overlooked by the existing lifelong topic models. Towards addressing the misalignment of semantics, we formulate LTM as a problem of non-negative matrix tri-factorization (NMTF) and propose a consolidation framework (i.e., NMTF-LTM) to enforce an alignment in a mapped topic space. In addition, a distributed parallel algorithm, namely PNMTF-LTM, is developed to meet the real-time requirement for large-scale stream processing. Empirical results show that our method can not only obtain a superior alignment of semantics without loss of topic quality, but also achieve effective speedup when deployed to a high performance computing cluster.

Aspect extraction is one of the key tasks in fine-grained sentiment analysis. This task aims to identify explicit opinion targets from user-generated documents. Currently, the mainstream methods for aspect extraction are built on recurrent neural networks (RNNs), which are difficult to parallelize. To accelerate the training/testing process, convolutional neural network (CNN)-based methods are introduced. However, such models usually utilize the same set of filters to convolve all input documents, and hence, the unique information inherent in each document may not be fully captured. To alleviate this issue, we propose a CNN-based model that employs a set of dynamic filters. Specifically, the proposed model extracts the aspects in a document using the filters generated from the aspect information intrinsic in the document. With the dynamically generated filters, our model is capable of learning more important features concerning aspects, thus promoting the effectiveness of aspect extraction. Furthermore, considering that aspects can be grouped into certain topics that conversely indicate the target words that need to be extracted, we naturally introduce a neural topic model (NTM) and integrate latent topics into the CNN-based module to help identify aspects. Experiments on two benchmark datasets demonstrate that the joint model is able to effectively identify aspects and produce interpretable topics.

Neural topic models have been widely used in discovering the latent semantics from a corpus. Recently, there are several researches on hierarchical neural topic models since the relationships among topics are valuable for data analysis and exploration. However, the existing hierarchical neural topic models are limited to generate a single topic tree. In this study, we present a nonparametric forest-structured neural topic model by firstly applying the self-attention mechanism to capture parent-child topic relationships, and then build a sparse directed acyclic graph to form a topic forest. Experiments indicate that our model can automatically learn a forest-structured topic hierarchy with indefinite numbers of trees and leaves, and significantly outperforms the baseline models on topic hierarchical rationality and affinity.

As time goes by, language evolves with word semantics changing. Unfortunately, traditional word embedding methods neglect the evolution of language and assume that word representations are static. Although contextualized word embedding models can capture the diverse representations of polysemous words, they ignore temporal information as well. To tackle the aforementioned challenges, we propose a graph-based dynamic word embedding (GDWE) model, which focuses on capturing the semantic drift of words continually. We introduce word-level knowledge graphs (WKGs) to store short-term and long-term knowledge. WKGs can provide rich structural information as supplement of lexical information, which help enhance the word embedding quality and capture semantic drift quickly. Theoretical analysis and extensive experiments validate the effectiveness of our GDWE on dynamic word embedding learning.

With the growing availability and popularity of sentiment-rich resources like blogs and online reviews, new opportunities and challenges have emerged regarding the identification, extraction, and organization of sentiments from user-generated documents or sentences. Recently, many studies have exploited lexicon-based methods or supervised learning algorithms to conduct sentiment analysis tasks separately; however, the former approaches ignore contextual information of sentences and the latter ones do not take sentiment information embedded in sentiment words into consideration. To tackle these limitations, we propose a new model named Sentiment Convolutional Neural Network (SentiCNN) to analyze the sentiments of sentences with both contextual and sentiment information of sentiment words, in which, contextual information is captured from word embeddings and sentiment information is identified using existing lexicons. We incorporate a Highway Network into our model to adaptively combine sentiment and contextual information from sentences by strengthening the connection between features of both sentences and their sentiment words. Furthermore, we propose three lexicon-based attention mechanisms (LBAMs) for our SentiCNN model to find the most important indicators of sentiments and make predictions more effectively. Experiments over two well-known datasets indicate that sentiment words, the Highway Network, and LBAMs contribute to sentiment analysis.

Hierarchical Dirichlet Process (HDP) has attracted much attention in the research community of natural language processing. Given a corpus, HDP is able to determine the number of topics automatically, possessing an important feature dubbed nonparametric that overcomes the challenging issue of manually specifying a suitable topic number in parametric topic models, such as Latent Dirichlet Allocation (LDA). Nevertheless, HDP requires a much higher computational cost than LDA for parameter estimation. By taking the advantage of multi-threading, a parallel Gibbs sampling algorithm is proposed to estimate parameters for HDP based on the equivalence between HDP and Gamma-Gamma Poisson Process (G2PP) in terms of the generative process. Unfortunately, the above parallel Gibbs sampling algorithm requires to apply the finite approximation on the number of topics manually (i.e., predefine the topic number), thus can not retain the nonparametric feature of HDP. Another drawback of the above models is the lack of capturing the semantic dependencies between words, because the topic assignment of words is independent with each other. Although some works have been done in phrase-based topic modelling, these existing methods are still limited by either enforcing the entire phrase to share a common topic or requiring much complex and time-consuming phrase mining methods. In this paper, we aim to develop a copula guided parallel Gibbs sampling algorithm for HDP which can adjust the number of topics dynamically and capture the latent semantic dependencies between words that compose a coherent segment. Extensive experiments on real-world datasets indicate that our method achieves low perplexities and high topic coherence scores with a small time cost. In addition, we validate the effectiveness of our method on the modelling of word semantic dependencies by comparing the extracted topical phrases with those learned by state-of-the-art phrase-based baselines.

Ziye Chen, Cheng Ding, Zusheng Zhang, Yanghui Rao, Haoran Xie. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Yuhao Feng, Yanghui Rao, Yuyao Tang, Ninghua Wang, He Liu. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

With the development of social network platforms, discussion forums, and question answering websites, a huge number of short messages that typically contain a few words for an individual document are posted by online users. In these short messages, emotions are frequently embedded for communicating opinions, expressing friendship, and promoting influence. It is quite valuable to detect emotions from short messages, but the corresponding task suffers from the sparsity of feature space. In this article, we first generate term groups co-occurring in the same context to enrich the number of features. Then, two basic supervised topic models are proposed to associate emotions with topics accurately. To reduce the time cost of parameter estimation, we further propose an accelerated algorithm for our basic models. Extensive evaluations using three short corpora validate the efficiency and effectiveness of the accelerated models for predicting the emotions of unlabeled documents, in addition to generate the topic-level emotion lexicons.

Mixed counting models that use the negative binomial distribution as the prior can well model over-dispersed and hierarchically dependent random variables; thus they have attracted much attention in mining dispersed document topics. However, the existing parameter inference method like Monte Carlo sampling is quite time-consuming. In this paper, we propose two efficient neural mixed counting models, i.e., the Negative Binomial-Neural Topic Model (NB-NTM) and the Gamma Negative Binomial-Neural Topic Model (GNB-NTM) for dispersed topic discovery. Neural variational inference algorithms are developed to infer model parameters by using the reparameterization of Gamma distribution and the Gaussian approximation of Poisson distribution. Experiments on real-world datasets indicate that our models outperform state-of-the-art baseline models in terms of perplexity and topic coherence. The results also validate that both NB-NTM and GNB-NTM can produce explainable intermediate variables by generating dispersed proportions of document topics.

Extracting the topical information from documents is important for public opinion analysis, text classification, and information retrieval tasks. Compared with identifying a wide variety of topics from long documents, it is challenging to generate a concentrated topic distribution for each short message. Although this problem can be tackled by adjusting the hyper-parameters in traditional topic models such as Latent Dirichlet Allocation, it remains an open problem in neural topic modelling. In this paper, we focus on adapting the popular Auto-Encoding Variational Bayes based neural topic models to short texts, by exploring the Archimedean copulas to guide the estimated topic distributions derived from linear projected samples of re-parameterized posterior distributions. Experimental results show the superiority of our method when compared with existing neural topic models in terms of perplexity, topic coherence, and classification accuracy.

This paper introduce a novel Position-aware Transformation Network (PTNet) for aspect-level sentiment classification. On the one hand, attention mechanisms have been employed to model the relationship between aspect and context. However, the position information of aspect words is rarely emphasized for sentiment prediction. The truth is that we should pay more attention to the word which is close to the aspect, since the words with closer distance may have a greater impact on the sentiment polarity of a sentence toward the aspect. On the other hand, existing approaches often adopt the average of aspect vectors or context vectors to calculate the attention weights, which may cause information loss if the aspect and context is not a single word. Therefore, this paper propose a position-aware layer and a context transformation layer in our model to solve the above two issues respectively. Moreover, several convolution kernels are also used to extract the n-gram information for prediction. We examine the performance of our model on three datasets: the first two are from SemEval2014 including the reviews of restaurants and laptops, and the third is a tweet collection. Experimental results show that our model consistently outperforms the state-of-the-art methods on all three datasets.

Traditional methods of annotating the sentiment of an unlabeled document are based on sentiment lexicons or machine learning algorithms, which have shown low computational cost or competitive performance. However, these methods ignore the semantic composition problem displaying in several ways such as negative reversing and intensification. In this paper, we propose a new method for sentiment classification using negative and intensive sentiment supplementary information, so as to exploit the linguistic feature of negative and intensive words in conjunction with the context information. Particularly, our method can solve the domain-specific problem without relying on the external sentiment lexicons. Experimental results on two real-world datasets demonstrate the effectiveness of our proposed method.

In the field of sentiment analysis and emotion detection in social media, or other tasks such as text classification involving supervised learning, researchers rely more heavily on large and accurate labelled training datasets. However, obtaining large-scale labelled datasets is time-consuming and high-quality labelled datasets are expensive and scarce. To deal with these problems, online crowdsourcing systems provide us an efficient way to accelerate the process of collecting training data via distributing the enormous tasks to various annotators to help create large amounts of labelled data at an affordable cost. Nowadays, these crowdsourcing platforms are heavily needed in dealing with social media text, since the social network platforms (e.g., Twitter) generate huge amounts of data in textual form everyday. However, people from different social and knowledge backgrounds have different views on various texts, which may lead to noisy labels. The existing noisy label aggregation/refinement algorithms mostly focus on aggregating labels from noisy annotations, which would not guarantee their effectiveness on the subsequent classification/ranking tasks. In this article, we propose a noise-aware classification framework that integrates the steps of noisy label aggregation and classification. The aggregated noisy crowd labels are fed into a classifier for training, while the predicted labels are employed as feedback for adjusting the parameters at the label aggregating stage. The classification framework is suitable for directly running on crowdsourcing datasets and applies to various kinds of classification algorithms. The feedback strategy makes it possible for us to find optimal parameters instead of using known data for parameter selection. Simulation experiments demonstrate that our method provide significant label aggregation performance for both binary and multiple classification tasks under various noisy environments. Experimenting on real-world data validates the feasibility of our framework in real noise data and helps us verify the reasonableness of the simulated experiment settings.

Label-specific topics can be widely used for supporting personality psychology, aspectlevel sentiment analysis, and cross-domain sentiment classification.To generate labelspecific topics, several supervised topic models which adopt likelihood-driven objective functions have been proposed.However, it is hard for them to get a precise estimation on both topic discovery and supervised learning.In this study, we propose a supervised topic model based on the Siamese network, which can trade off label-specific word distributions with document-specific label distributions in a uniform framework.Experiments on realworld datasets validate that our model performs competitive in topic discovery quantitatively and qualitatively.Furthermore, the proposed model can effectively predict categorical or real-valued labels for new documents by generating word embeddings from a labelspecific topical space.

Sentiment classification is a key task of natural language processing and a long-term goal of artificial intelligence. In this paper, we propose a new approach, which combines the Attention Mechanism, Bidirectional Long Short-Term Memory Network, and Lexicons for sentiment classification. Experiments on the Movie Review and Stanford Sentiment Treebank datasets demonstrate that the performance of our approach is better than other approaches.

Cross-domain sentiment classification aims to tag sentiments for a target domain by labeled data from a source domain. Due to the difference between domains, the accuracy of a trained classifier may be very low. In this paper, we propose a boosting-based learning framework named TR-TrAdaBoost for cross-domain sentiment classification. We firstly explore the topic distribution of documents, and then combine it with the unigram TrAdaBoost. The topic distribution captures the domain information of documents, which is valuable for cross-domain sentiment classification. Experimental results indicate that TR-TrAdaBoost represents documents well and boost the performance and robustness of TrAdaBoost.

This paper addresses the task of cross-domain social emotion classification of online documents. The cross-domain task is formulated as using abundant labeled documents from a source domain and a small amount of labeled documents from a target domain, to predict the emotion of unlabeled documents in the target domain. Although several cross-domain emotion classification algorithms have been proposed, they require that feature distributions of different domains share a sufficient overlapping, which is hard to meet in practical applications. This paper proposes a novel framework, which uses the emotion distribution of training documents at the cluster level, to alleviate the aforementioned issue. Experimental results on two datasets show the effectiveness of our proposed model on cross-domain social emotion classification.

Social emotion classification aims to predict the aggregation of emotional responses embedded in online comments contributed by various users. Such a task is inherently challenging because extracting relevant semantics from free texts is a classical research problem. Moreover, online comments are typically characterized by a sparse feature space, which makes the corresponding emotion classification task very difficult. On the other hand, though deep neural networks have been shown to be effective for speech recognition and image analysis tasks because of their capabilities of transforming sparse low-level features to dense high-level features, their effectiveness on emotion classification requires further investigation. The main contribution of our work reported in this paper is the development of a novel model of semantically rich hybrid neural network (HNN) which leverages unsupervised teaching models to incorporate semantic domain knowledge into the neural network to bootstrap its inference power and interpretability. To our best knowledge, this is the first successful work of incorporating semantics into neural networks to enhance social emotion classification and network interpretability. Through empirical studies based on three real-world social media datasets, our experimental results confirm that the proposed hybrid neural networks outperform other state-of-the-art emotion classification methods.

With the development of Web 2.0, many users express their opinions online. This paper is concerned with the classification of social emotions on varied-scale datasets. Different from traditional models which weight training documents equally, the concept of emotional entropy is proposed to estimate the weight and tackle the issue of noisy documents. The topic assignment is also used to distinguish different emotional senses of the same word. Experimental evaluations using different data sets validate the effectiveness of the proposed social emotion classification model.

Short text is prevalent on the Web, but it brings challenges to content analysis methods for the lack of contextual information. Biterm topic model (BTM) is a variant of latent Dirichlet allocation, which effectively infers the latent topic distribution of short text by modeling the generation of biterms in the whole corpus. However, it needs fine-tuning from labels to reduce noise when applied to supervised learning. Motivated by the transfer learning approach, we propose the hybrid neural networks based on BTM and conventional neural networks, which first make the hidden layer of neural networks approximate the inference of BTM. Following this initial pre-training phase, we then use the simple back-propagation algorithm to fine-tune the topic distribution learned from BTM, so as to improve the performance of supervised learning. Our experiment on two diverse collections of short text validates the effectiveness of the proposed hybrid neural networks for social emotion detection.

With the development of the Internet, user-generated data has been growing tremendously in Web 2.0 era. Facing such a big volume of resources in folksonomy, people need a method of fast exploration and indexing to find their demanded data. To achieve this goal, contextual information is indispensable and valuable to understand user preference and purpose. In sociolinguistics, context can be mainly categorized as verbal context and social context. Comparing with verbal context, social context not only requires domain knowledge to pre-define contextual attributes but also acquires additional data from users. However, there is no research of addressing irrelevant contextual factors for verbal context model so far. The dominating set from verbal context proposed in this paper is to fill this blank. We present the verbal context in folksonomy to capture the user intention, and propose a dominating set discovering method for this verbal context model to prune the irrelevant contextual factors and keep the major characteristics at the same time. Furthermore, the experiments, which are conducted on a public data set, show that the proposed method gives convincing results.

As an event-based task, Emerging Event Detection (EED) faces the problems of multiple events on the same subject and the evolution of events. Current term weighting schemes for EED exploiting Named Entity, temporal information and Topic Modeling all have their limited utility. In this paper, a new term weighting scheme, which models the sparse aspect, global weight and local weight of each story, is proposed. Then, an unsupervised algorithm based on the new scheme is applied to EED. We evaluate our approach on two datasets from TDT5, and compare it with TFIDF and existing two schemes exploiting Topic Modeling. Experiments on Retrospective and On-line EED show that our scheme yields better results.