Recent advances in Speech Large Language Models (Speech LLMs) have led to great progress in speech understanding tasks such as Automatic Speech Recognition (ASR) and Speech Emotion Recognition (SER). However, whether these models can achieve human-level auditory perception, particularly in terms of their ability to comprehend latent intentions and implicit emotions in real-world spoken language, remains underexplored. To this end, we introduce the Human-level Perception in Spoken Speech Understanding (HPSU), a new benchmark for fully evaluating the human-level perceptual and understanding capabilities of Speech LLMs. HPSU comprises over 20,000 expert-validated spoken language understanding samples in English and Chinese. It establishes a comprehensive evaluation framework by encompassing a spectrum of tasks, ranging from basic speaker attribute recognition to complex inference of latent intentions and implicit emotions. To address the issues of data scarcity and high cost of manual annotation in real-world scenarios, we developed a semi-automatic annotation process. This process fuses audio, textual, and visual information to enable precise speech understanding and labeling, thus enhancing both annotation efficiency and quality. We systematically evaluate various open-source and proprietary Speech LLMs. The results demonstrate that even top-performing models still fall considerably short of human capabilities in understanding genuine spoken interactions. Consequently, HPSU will be useful for guiding the development of Speech LLMs toward human-level perception and cognition.

Understanding human attitudes, preferences, and behaviors through social surveys is essential for academic research and policymaking. Yet traditional surveys face persistent challenges, including fixed-question formats, high costs, limited adaptability, and difficulties ensuring cross-cultural equivalence. While recent studies explore large language models (LLMs) to simulate survey responses, most are limited to structured questions, overlook the entire survey process, and risks under-representing marginalized groups due to training data biases. We introduce AlignSurvey, the first benchmark that systematically replicates and evaluates the full social survey pipeline using LLMs. It defines four tasks aligned with key survey stages: social role modeling, semi-structured interview modeling, attitude stance modeling and survey response modeling. It also provides task-specific evaluation metrics to assess alignment fidelity, consistency, and fairness at both individual and group levels, with a focus on demographic diversity. To support AlignSurvey, we construct a multi-tiered dataset architecture: (i) the Social Foundation Corpus, a cross-national resource with 44K+ interview dialogues and 400K+ structured survey records; and (ii) a suite of Entire-Pipeline Survey Datasets, including the expert-annotated AlignSurvey-Expert (ASE) and two nationally representative surveys for cross-cultural evaluation. We release the SurveyLM family, obtained through two-stage fine-tuning of open-source LLMs, and offer reference models for evaluating domain-specific alignment. All datasets, models, and tools are available at github and huggingface to support transparent and socially responsible research.

Recommending event schedules is a key issue in Event-based Social Networks (EBSNs) in order to maintain user activity. An effective recommendation is required to maximize the user's preference, subjecting to both time and geographical constraints. Existing methods face an inherent trade-off among efficiency, effectiveness, and generalization, due to the NP-hard nature of the problem. This paper proposes the Chain-of-Scheduling (CoS) framework, which activates the event scheduling capability of Large Language Models (LLMs) through a guided, efficient scheduling process. CoS enhances LLM by formulating the schedule task into three atomic stages, i.e., exploration, verification and integration. Then we enable the LLMs to generate CoS autonomously via Knowledge Distillation (KD). Experimental results show that CoS achieves near-theoretical optimal effectiveness with high efficiency on three real-world datasets in a interpretable manner. Moreover, it demonstrates strong zero-shot learning ability on out-of-domain data.

Hang Hu, Ziyan Liu, Rujie Wen, Ruihui Hou, Xueyan Wu, Mu Zhang, Jianxing Yu, Tong Ruan, Jingping Liu. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Multi-Hop Question Answering (MHQA) is crucial for evaluating the model's capability to integrate information from diverse sources.However, creating extensive and high-quality MHQA datasets is challenging: (i) manual annotation is expensive, and (ii) current synthesis methods often produce simplistic questions or require extensive manual guidance.This paper introduces HopWeaver, the first cross-document framework synthesizing authentic multi-hop questions without human intervention.HopWeaver synthesizes bridge and comparison questions through an innovative pipeline that identifies complementary documents and constructs authentic reasoning paths to ensure true multi-hop reasoning.We further present a comprehensive system for evaluating the synthesized multi-hop questions.Empirical evaluations demonstrate that the synthesized questions achieve comparable or superior quality to human-annotated datasets at a lower cost.Our framework provides a valuable tool for the research community: it can automatically generate challenging benchmarks from any raw corpus, which opens new avenues for both evaluation and targeted training to improve the reasoning capabilities of advanced question answering models, especially in domains with scarce resources.The code for HopWeaver is publicly available 1 .

This article aims to speed up the training of large neural networks with the Mixture-of-Experts (MoE) structure. Training MoE often needs a lot of computing resources due to its large scale. Traditional acceleration methods either degrade prediction performance or rely on dedicated hardware with additional resources, but the resources are usually limited in real applications.One solution is to resort to new optimization strategies, such as learning from easy to hard by multiple stages. However, existing strategies are designed mainly for networks with a serial structure, but MoE has multiple expert networks working in parallel. They employ an identical learning plan for all experts, ignoring that each expert's learning domain and speed differ, resulting in some experts being over-learned while others being under-learned. This mismatch will make it hard for experts to train together, harming training efficiency. To address this problem, we propose a new training acceleration framework. It can customize an effective learning plan for each expert by considering their training progress, avoiding blindly searching in a huge parameter space. In detail, we first design a multi-stage planner that starts with optimizing a subpart of the network and then scales it up to retrain until it expands to an entire network. It uses the density function to assess the knowledge gained by the expert in each stage, giving priority to the experts who learn faster to increase the training scale, so as to boost convergence. Afterward, we exploit the growth operator to add the expert training scale of the next stage. In each stage, the network would converge to some locally optimal values. That can provide a better initialization to train the next stage more easily, since the time and data required for training from scratch are greatly reduced. To alleviate the gradient vanishing problem caused by network growth, we develop a scheduler to dynamically adjust the learning rate. Extensive experiments are conducted to validate the effectiveness of our method. The results show that we can obtain more than 25% training acceleration on average.

Many unsupervised visual anomaly detection methods train an auto-encoder to reconstruct normal samples and then leverage the reconstruction error map to detect and localize the anomalies. However, due to the powerful modeling and generalization ability of neural networks, some anomalies can also be well reconstructed, resulting in unsatisfactory detection and localization accuracy. In this paper, a small coarsely-labeled anomaly dataset is first collected. Then, a coarse-knowledge-aware adversarial learning method is developed to align the distribution of reconstructed features with that of normal features. The alignment can effectively suppress the auto-encoder's reconstruction ability on anomalies and thus improve the detection accuracy. Considering that anomalies often only occupy very small areas in anomalous images, a patch-level adversarial learning strategy is further developed. Although no patch-level anomalous information is available, we rigorously prove that by simply viewing any patch features from anomalous images as anomalies, the proposed knowledge-aware method can also align the distribution of reconstructed patch features with the normal ones. Experimental results on four medical datasets and two industrial datasets demonstrate the effectiveness of our method in improving the detection and localization performance.

Grounded Multimodal Named Entity Recognition (GMNER) is an emerging information extraction (IE) task, aiming to simultaneously extract entity spans, types, and corresponding visual regions of entities from given sentence-image pairs data. Recent unified methods employing machine reading comprehension or sequence generation-based frameworks show limitations in this difficult task. The former, utilizing human-designed type queries, struggles to differentiate ambiguous entities, such as Jordan (Person) and off-White x Jordan (Shoes). The latter, following the one-by-one decoding order, suffers from exposure bias issues. We maintain that these works misunderstand the relationships of multimodal entities. To tackle these, we propose a novel unified framework named Multi-grained Query-guided Set Prediction Network (MQSPN) to learn appropriate relationships at intra-entity and inter-entity levels. Specifically, MQSPN explicitly aligns textual entities with visual regions by employing a set of learnable queries to strengthen intra-entity connections. Based on distinct intra-entity modeling, MQSPN reformulates GMNER as a set prediction, guiding models to establish appropriate inter-entity relationships from a optimal global matching perspective. Additionally, we incorporate a query-guided Fusion Net (QFNet) as a glue network to boost better alignment of two-level relationships. Extensive experiments demonstrate that our approach achieves state-of-the-art performances in widely used benchmarks.

Fan Li, Jianxing Yu, Jielong Tang, Wenqing Chen, Hanjiang Lai, Yanghui Rao, Jian Yin. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Emotion Recognition in Conversations (ERC) is crucial for machines to understand dynamic human emotions. While Large Language Models (LLMs) show promise, their performance is often limited by challenges in interpreting complex conversational streams. We introduce a Clue of Emotion (CoE) framework, which progressively integrates key conversational clues to enhance the ERC task. Building on CoE, we implement a multi-stage auxiliary learning strategy that incorporates role-playing, speaker identification, and emotion reasoning tasks, each targeting different aspects of conversational emotion understanding and enhancing the model’s ability to interpret emotional contexts. Our experiments on EmoryNLP, MELD, and IEMOCAP demonstrate that CoE consistently outperforms state-of-the-art methods, achieving a 2.92% improvement on EmoryNLP. These results underscore the effectiveness of clues and multi-stage auxiliary learning for ERC, offering valuable insights for future research.

This paper focuses on sarcasm detection, which aims to identify whether given statements convey criticism, mockery, or other negative sentiment opposite to the literal meaning. To detect sarcasm, humans often require a comprehensive understanding of the semantics in the statement and even resort to external commonsense to infer the fine-grained incongruity. However, existing methods lack commonsense inferential ability when they face complex real-world scenarios, leading to unsatisfactory performance. To address this problem, we propose a novel framework for sarcasm detection, which conducts incongruity reasoning based on commonsense augmentation, called EICR. Concretely, we first employ retrieval-augmented large language models to supplement the missing but indispensable commonsense background knowledge. To capture complex contextual associations, we construct a dependency graph and obtain the optimized topology via graph refinement. We further introduce an adaptive reasoning skeleton that integrates prior rules to extract sentiment-inconsistent subgraphs explicitly. To eliminate the possible spurious relations between words and labels, we employ adversarial contrastive learning to enhance the robustness of the detector. Experiments conducted on five datasets demonstrate the effectiveness of EICR.

This paper studies the task of generating commonsense reasoning questions (QG) with desired difficulty levels. Compared to traditional shallow questions that can be solved by simple term matching, ours are more challenging. Our answering process requires reasoning over multiple contextual and commonsense clues. That involves advanced comprehension skills, such as abstract semantics learning and missing knowledge inference. Existing work mostly learns to map the given text into questions, lacking a mechanism to control results with the desired complexity. To address this problem, we propose a novel controllable framework. We first derive contextual and commonsense clues involved in reasoning questions from the text. These clues are used to create simple sub-questions. We then aggregate multiple sub-questions to compose complex ones under the guidance of prior reasoning structures. By iterating this process, we can compose a complex QG task based on a series of smaller and simpler QG subtasks. Each subtask serves as a building block for a larger one. Each composition corresponds to an increase in the reasoning step. Moreover, we design a voting verifier to ensure results’ validity from multiple views, including answer consistency, reasoning difficulty, and context correlation. Finally, we can learn the optimal QG model to yield thought-provoking results. Evaluations on two typical datasets validate our method.

Grounded Multimodal Named Entity Recognition (GMNER) is a new information extraction task.It requires models to extract named entities and ground them to real-world visual objects.Previous methods, relying on domainspecific fine-tuning, struggle with unseen multimodal entities due to limited knowledge and generalization.Recently, multimodal large language models (MLLMs) have demonstrated strong open-set abilities.However, their performance is hindered by the lack of in-domain knowledge due to costly training for GMNER datasets.To address these limitations, we propose UnCo, a two-stage Uncertainty-driven Collaborative framework that leverages the complementary strengths of small fine-tuned models and MLLMs.Specifically, in stage one, we equip the small model with a unified uncertainty estimation (UE) for multimodal entities.This enables the small model to express "I do not know" when recognizing unseen entities beyond its capabilities.Predictions with high uncertainty are then filtered and delegated to the MLLM.In stage two, an Uncertainty-aware Hierarchical Correction mechanism guides the MLLM to refine uncertain predictions using its open-domain knowledge.Ultimately, UnCo effectively retains the in-domain knowledge of small models while utilizing the capabilities of MLLMs to handle unseen samples.Extensive experiments demonstrate UnCo's effectiveness on two GMNER benchmarks.

This paper focuses on generating speech with the acoustic style that meets users' needs based on their open-domain instructions.To control the style, early work mostly relies on predefined rules or templates.The control types and formats are fixed in a closed domain, making it hard to meet users' diverse needs.One solution is to resort to instructions in free text to guide the generation.Current work mainly studies the instructions that clearly specify the acoustic styles, such as low pitch and fast speed.However, the instructions are complex, some even vague and abstract, such as "Generate a voice of a woman who is heartbroken due to a breakup."It is hard to infer this implicit style by traditional matching-based methods.To address this problem, we propose a new controllable model.It first utilizes multimodal LLMs with knowledge-augmented techniques to infer the desired speech style from the instructions.The powerful language understanding ability of LLMs can help us better elicit the implicit style factors from the instruction.By using these factors as a control condition, we design a diffusion-based generator adept at finely adjusting speech details, enabling higher flexibility to meet complex users' needs.Next, we verify the output speech from three aspects, i.e., consistency of decoding state, mel-spectrogram, and instruction style.This verified feedback can inversely optimize the generator.Extensive experiments are conducted on three popular datasets.The results show the effectiveness and good controllability of our approach.

Top-N recommendation systems are recognized as highly effective for delivering personalized services that cater to the varied interests of users. Nonetheless, current state-of-the-art (SOTA) analyses reveal a marked variability in their performance across users with differing levels of activity, which substantially undermines the quality of personalized recommendation services. Prevailing research tends to overlook this discrepancy, often presuming a uniform probability distribution in user preferences and employing a static model (such as a single latent vector) for user representation. This oversimplification impedes the adaptability of existing models to accommodate the spectrum of user activity levels. In our research, we introduce the variational kernel density estimation (VKDE) approach, an innovative nonparametric method designed to accurately capture the unique preference distributions of individual users. The VKDE framework integrates multiple local distributions to construct a comprehensive global preference profile for each user. We have developed a novel variational kernel function that delineates user-specific interests and constructs each local distribution accordingly. Additionally, we present a tailored sampling strategy that simplifies the complexity of the training process while preserving the efficacy of the recommendations. Empirical evaluations conducted on four widely recognized public datasets demonstrate that our VKDE model achieves superior performance over the SOTA alternatives, significantly enhancing accuracy for users with a broad range of activity levels.

Complex Query Answering (CQA) on knowledge graphs is a fundamental yet challenging task, which can be formalized as answering a subset of first-order logic queries containing logical conjunction, disjunction, negation, and existential quantifiers. Recent research reveals that Link Predictors (LPs) trained on 1-hop queries can generalize to various types of complex queries. However, existing methods neglect crucial characteristics of LPs' outputs, including the effects of highly relevant entities and uncertainty. What's worse, as they model logical operations by fuzzy set operations, these methods suffer from problems like inflexibility, sensitivity to noise, and inconsistency with priority in human cognition, which limits their performance, especially on queries with negation. To address these challenges, we propose an efficient fuzzy system for CQA that requires no extra training overheads and is plug-and-play with existing LP-based methods. Firstly, we expand the output of LPs by two complementary membership functions of weak and strong relevance, which help to distinguish the target entities from highly relevant and irrelevant entities. Subsequently, we model logical operations through fuzzy rule bases and infer the final predictions via defuzzification, providing a flexible and tractable scheme for modeling logical operations. Finally, the effectiveness of the proposed fuzzy system is validated by its outstanding performance on benchmark datasets when compared to state-of-theart methods. The source code of our proposed method is available at https://anonymous.4open.science/r/FuzzSys-CQA-E285.

With the development of blockchain technology, smart contracts have become an important component of blockchain applications. Despite their crucial role, the development of smart contracts may introduce vulnerabilities and potentially lead to severe consequences, such as financial losses. Meanwhile, large language models, represented by ChatGPT, have gained great attention, showcasing great capabilities in code analysis tasks. In this article, we presented an empirical study to investigate the performance of ChatGPT in identifying smart contract vulnerabilities. Initially, we evaluated ChatGPT’s effectiveness using a publicly available smart contract dataset. Our findings discover that while ChatGPT achieves a high recall rate, its precision in pinpointing smart contract vulnerabilities is limited. Furthermore, ChatGPT’s performance varies when detecting different vulnerability types. We delved into the root causes for the false positives generated by ChatGPT, and categorized them into four groups. Second, by comparing ChatGPT with other state-of-the-art smart contract vulnerability detection tools, we found that ChatGPT’s F-score is lower than others for 3 out of the 7 vulnerabilities. In the case of the remaining 4 vulnerabilities, ChatGPT exhibits a slight advantage over these tools. Finally, we analyzed the limitation of ChatGPT in smart contract vulnerability detection, revealing that the robustness of ChatGPT in this field needs to be improved from two aspects: its uncertainty in answering questions; and the limited length of the detected code. In general, our research provides insights into the strengths and weaknesses of employing large language models, specifically ChatGPT, for the detection of smart contract vulnerabilities.

Yufeng Zhang, Jianxing Yu, Yanghui Rao, Libin Zheng, Qinliang Su, Huaijie Zhu, Jian Yin. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

BERT and TFIDF features excel in capturing rich semantics and important words, respectively.Since most existing clustering methods are solely based on the BERT model, they often fall short in utilizing keyword information, which, however, is very useful in clustering short texts.In this paper, we propose a CO-Training Clustering (COTC) framework to make use of the collective strengths of BERT and TFIDF features.Specifically, we develop two modules responsible for the clustering of BERT and TFIDF features, respectively.We use the deep representations and cluster assignments from the TFIDF module outputs to guide the learning of the BERT module, seeking to align them at both the representation and cluster levels.Reversely, we also use the BERT module outputs to train the TFIDF module, thus leading to the mutual promotion.We then show that the alternating co-training framework can be placed under a unified joint training objective, which allows the two modules to be connected tightly and the training signals to be propagated efficiently.Experiments on eight benchmark datasets show that our method outperforms current SOTA methods significantly.

This paper focuses on detecting clickbait posts on the Web.These posts often use eye-catching disinformation in mixed modalities to mislead users to click for profit.That affects the user experience and thus would be blocked by content provider.To escape detection, malicious creators use tricks to add some irrelevant nonbait content into bait posts, dressing them up as legal to fool the detector.This content often has biased relations with non-bait labels, yet traditional detectors tend to make predictions based on simple co-occurrence rather than grasping inherent factors that lead to malicious behavior.This spurious bias would easily cause misjudgments.To address this problem, we propose a new debiased method based on causal inference.We first employ a set of features in multiple modalities to characterize the posts.Considering these features are often mixed up with unknown biases, we then disentangle three kinds of latent factors from them, including the invariant factor that indicates intrinsic bait intention; the causal factor which reflects deceptive patterns in a certain scenario, and non-causal noise.By eliminating the noise that causes bias, we can use invariant and causal factors to build a robust model with good generalization ability.Experiments on three popular datasets show the effectiveness of our approach.

Multi-party conversation modeling plays a vital role in emotion recognition in conversation (ERC). Aside from the intra- and inter-speaker dependencies between different speakers, the difficulty also lies in the fact that each conversation may contain several to many utterances that compose a long text sequence. In this article, we present two approaches to effective multi-party conversation modeling. First, to encode long sequences and capture long-range dependency between utterances, we introduce a dialog-oriented language model, DialogXL, with enhanced memory to store longer conversation sequences and dialog-aware self-attention to deal with multi-party dependencies. Second, we present a directed acyclic neural network, namely DAG-ERC, to encode the utterances with a directed acyclic graph (DAG) to better capture the intrinsic structure within a conversation. DAG-ERC combines the advantages of recurrent models and graph models and provides a more intuitive way to model information flow between sequential utterances. Extensive experiments are conducted on four ERC benchmarks with state-of-the-art models employed for comparison, and empirical results demonstrate the superiority of the two models in multi-party conversation modeling.

Due to the easy access, implicit feedback is often used for recommender systems. Compared with point-wise learning and pair-wise learning methods, list-wise rank learning methods have superior performance for top- \(N\) recommendation. Recent solutions, especially the list-wise methods, simply treat all interacted items of a user as equally important positives and annotate all no-interaction items of a user as negatives. For the list-wise approaches, we argue that this annotation scheme of implicit feedback is over-simplified due to the sparsity and missing fine-grained labels of the feedback data. To overcome this issue, we revisit the so-called positive and negative samples. First, considering the loss function of list-wise ranking, we analyze the impact of false positives and negatives theoretically. Second, based on the observation, we propose a self-adjusting credibility weight mechanism to re-weigh the positive samples and exploit the higher-order relation based on item–item matrix to sample the critical negative samples. In order to prevent the introduction of noise, we design a pruning strategy for critical negatives. Besides, to combine the reconstruction loss function for the positive samples and critical negative samples, we develop a simple yet effective VAEs framework with linear structure, which abandons the complex non-linear structure. Extensive experiments are conducted on six public real-world datasets. The results demonstrate that, our VAE* outperforms other VAE-based models by a large margin. Besides, we also verify the effect of denoising positives and exploring critical negatives by ablation study.

Generative adversarial networks (GANs) are known for their strong abilities on capturing the underlying distribution of training instances. Since the seminal work of GAN, many variants of GAN have been proposed. However, existing GANs are almost established on the assumption that the training dataset is clean. But in many real-world applications, this may not hold, that is, the training dataset may be contaminated by a proportion of undesired instances. When training on such datasets, existing GANs will learn a mixture distribution of desired and contaminated instances, rather than the desired distribution of desired data only (target distribution). To learn the target distribution from contaminated datasets, two purified generative adversarial networks (PuriGAN) are developed, in which the discriminators are augmented with the capability to distinguish between target and contaminated instances by leveraging an extra dataset solely composed of contamination instances. We prove that under some mild conditions, the proposed PuriGANs are guaranteed to converge to the distribution of desired instances. Experimental results on several datasets demonstrate that the proposed PuriGANs are able to generate much better images from the desired distribution than comparable baselines when trained on contaminated datasets. In addition, we also demonstrate the usefulness of PuriGAN on downstream applications by applying it to the tasks of semi-supervised anomaly detection on contaminated datasets and PU-learning. Experimental results show that PuriGAN is able to deliver the best performance over comparable baselines on both tasks.

This paper proposes a new task in the field of Answering Subjective Induction Question on Products (SUBJPQA). The answer to this kind of question is non-unique, but can be interpreted from many perspectives. For example, the answer to ‘whether the phone is heavy’ has a variety of different viewpoints. A satisfied answer should be able to summarize these subjective opinions from multiple sources and provide objective knowledge, such as the weight of a phone. That is quite different from the traditional QA task, in which the answer to a factoid question is unique and can be found from a single data source. To address this new task, we propose a three-steps method. We first retrieve all answer-related clues from multiple knowledge sources on facts and opinions. The implicit commonsense facts are also collected to supplement the necessary but missing contexts. We then capture their relevance with the questions by interactive attention. Next, we design a reinforcement-based summarizer to aggregate all these knowledgeable clues. Based on a template-controlled decoder, we can output a comprehensive and multi-perspective answer. Due to the lack of a relevant evaluated benchmark set for the new task, we construct a large-scale dataset, named SupQA, consisting of 48,352 samples across 15 product domains. Evaluation results show the effectiveness of our approach.

This article focuses on the topic of multi-hop question generation (QG), which aims to generate the questions requiring multi-hop reasoning skills from the given text. These questions are not only syntactically valid but also logically correlated with the answers. Concretely, we first design a basic QG model and customize several techniques to ensure results' syntactic validity. In order to promote the logical correlations, we use a reasoning chain extracted from the text to regularize the results. Considering that different samples have their own characteristics on the aspects of text contextual structure, the type of question, and logical correlation, we propose a new adaptive meta-learner to optimize the basic QG model. Each case and its similar samples are viewed as a pseudo-QG task. The similar structural contexts contained in the same task are used as guidance to fine-tune the model. To measure the similarity of samples' structured inputs, we propose a data-driven multi-level recognizer. The experimental results on two typical data sets in various domains show the effectiveness of the proposed approach. Moreover, we apply the generated results to the task of machine reading comprehension and achieve significant performance improvements. That demonstrates the capacity of multi-hop QG in facilitating real-world applications.

Efficient document retrieval heavily relies on the technique of semantic hashing, which learns a binary code for every document and employs Hamming distance to evaluate document distances.However, existing semantic hashing methods are mostly established on outdated TFIDF features, which obviously do not contain lots of important semantic information about documents.Furthermore, the Hamming distance can only be equal to one of several integer values, significantly limiting its representational ability for document distances.To address these issues, in this paper, we propose to leverage BERT embeddings to perform efficient retrieval based on the product quantization technique, which will assign for every document a real-valued codeword from the codebook, instead of a binary code as in semantic hashing.Specifically, we first transform the original BERT embeddings via a learnable mapping and feed the transformed embedding into a probabilistic product quantization module to output the assigned codeword.The refining and quantizing modules can be optimized in an end-to-end manner by minimizing the probabilistic contrastive loss.A mutual information maximization based method is further proposed to improve the representativeness of codewords, so that documents can be quantized more accurately.Extensive experiments conducted on three benchmarks demonstrate that our proposed method significantly outperforms current state-of-the-art baselines 1 .

In spite of its wide usage in various applications, existing construction methods for Knowledge Base (KB) are still on their way to obtaining 100% correct facts. Thus, employing crowd workers to validate a KB has been proposed to improve its reliability. Most of the existing works focus on devising games with proper incentives to engage workers in validating more facts, but rarely consider matching facts with proper workers. Facts have diverse domains (topics), which naturally require workers of different expertise. In addition, they also generally have different utilities, i.e., some are more heavily used than others. Thus, distinguishing the facts in terms of utility to give them different validation priorities is meaningful, especially when the budget is limited. To this end, we study the crowdsourced fact validation problem which considers worker domains and fact utilities, and find that with some reductions, it can be solved by the existing minimum cost network flow method. However, directly employing that method requires a huge time cost. We thereby propose an optimized network flow method which reduces the network complexity to save the time cost by properly grouping the facts. Furthermore, we propose an incremental validation method, which utilizes the previous results for validating an evolving KB. We finally conduct extensive experiments to demonstrate the effectiveness of the proposed methods.

Personality detection aims to identify the personality traits implied in social media posts. The core of this task is to put together information in multiple scattered posts to depict an overall personality profile for each user. Existing approaches either encode each post individually or assemble posts arbitrarily into a new document that can be encoded sequentially or hierarchically. While the first approach ignores the connection between posts, the second tends to introduce unnecessary post-order bias into posts. In this paper, we propose a multi-document Transformer, namely Transformer-MD, to tackle the above issues. When encoding each post, Transformer-MD allows access to information in the other posts of the user through Transformer-XL’s memory tokens which share the same position embedding.Besides, personality is usually defined along different traits and each trait may need to attend to different post information, which has rarely been touched by existing research. To address this concern, we propose a dimension attention mechanism on top of Transformer-MD to obtain trait-specific representations for multi-trait personality detection. We evaluate the proposed model on the Kaggle and Pandora MBTI datasets and the experimental results show that it compares favorably with baseline methods.

Zijing Ou, Qinliang Su, Jianxing Yu, Bang Liu, Jingwen Wang, Ruihui Zhao, Changyou Chen, Yefeng Zheng. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Many unsupervised hashing methods are implicitly established on the idea of reconstructing the input data, which basically encourages the hashing codes to retain as much information of original data as possible. However, this requirement may force the models spending lots of their effort on reconstructing the unuseful background information, while ignoring to preserve the discriminative semantic information that is more important for the hashing task. To tackle this problem, inspired by the recent success of contrastive learning in learning continuous representations, we propose to adapt this framework to learn binary hashing codes. Specifically, we first propose to modify the objective function to meet the specific requirement of hashing and then introduce a probabilistic binary representation layer into the model to facilitate end-to-end training of the entire model. We further prove the strong connection between the proposed contrastive-learning-based hashing method and the mutual information, and show that the proposed model can be considered under the broader framework of the information bottleneck (IB). Under this perspective, a more general hashing model is naturally obtained. Extensive experimental results on three benchmark image datasets demonstrate that the proposed hashing method significantly outperforms existing baselines.

Abstract The task of question generation (QG) aims to create valid questions and correlated answers from the given text. Despite the neural QG approaches have achieved promising results, they are typically developed for languages with rich annotated training data. Because of the high annotation cost, it is difficult to deploy to other low-resource languages. Besides, different samples have their own characteristics on the aspects of text contextual structure, question type and correlations. Without capturing these diversified characteristics, the traditional one-size-fits-all model is hard to generate the best results. To address this problem, we study the task of cross-lingual QG from an adaptive learning perspective. Concretely, we first build a basic QG model on a multilingual space using the labelled data. In this way, we can transfer the supervision from the high-resource language to the language lacking labelled data. We then design a task-specific meta-learner to optimize the basic QG model. Each sample and its similar instances are viewed as a pseudo-QG task. The asking patterns and logical forms contained in the similar samples can be used as a guide to fine-tune the model fitly and produce the optimal results accordingly. Considering that each sample contains the text, question and answer, with unknown semantic correlations among them, we propose a context-dependent retriever to measure the similarity of such structured inputs. Experimental results on three languages of three typical data sets show the effectiveness of our approach.

Generating fluent and informative responses is of critical importance for task-oriented dialogue systems.Existing pipeline approaches generally predict multiple dialogue acts first and use them to assist response generation.There are at least two shortcomings with such approaches.First, the inherent structures of multi-domain dialogue acts are neglected.Second, the semantic associations between acts and responses are not taken into account for response generation.To address these issues, we propose a neural co-generation model that generates dialogue acts and responses concurrently.Unlike those pipeline approaches, our act generation module preserves the semantic structures of multi-domain dialogue acts and our response generation module dynamically attends to different acts as needed.We train the two modules jointly using an uncertainty loss to adjust their task weights adaptively.Extensive experiments are conducted on the largescale MultiWOZ dataset and the results show that our model achieves very favorable improvement over several state-of-the-art models in both automatic and human evaluations.

This paper focuses on generating multi-hop reasoning questions from the raw text in a low resource circumstance.Such questions have to be syntactically valid and need to logically correlate with the answers by deducing over multiple relations on several sentences in the text.Specifically, we first build a multi-hop generation model and guide it to satisfy the logical rationality by the reasoning chain extracted from a given text.Since the labeled data is limited and insufficient for training, we propose to learn the model with the help of a large scale of unlabeled data that is much easier to obtain.Such data contains rich expressive forms of the questions with structural patterns on syntax and semantics.These patterns can be estimated by the neural hidden semi-Markov model using latent variables.With latent patterns as a prior, we can regularize the generation model and produce the optimal results.Experimental results on the HotpotQA data set demonstrate the effectiveness of our model.Moreover, we apply the generated results to the task of machine reading comprehension and achieve significant performance improvements.

Network embedding has recently emerged as a promising technique to embed nodes of a network into low-dimensional vectors.While fairly successful, most existing works focus on the embedding techniques for static networks.But in practice, there are many networks that are evolving over time and hence are dynamic, e.g., the social networks.To address this issue, a high-order spatio-temporal embedding model is developed to track the evolutions of dynamic networks.Specifically, an activeness-aware neighborhood embedding method is first proposed to extract the high-order neighborhood information at each given timestamp.Then, an embedding prediction framework is further developed to capture the temporal correlations, in which the attention mechanism is employed instead of recurrent neural networks (RNNs) for its efficiency in computing and flexibility in modeling.Extensive experiments are conducted on four realworld datasets from three different areas.It is shown that the proposed method outperforms all the baselines by a substantial margin for the tasks of dynamic link prediction and node classification, which demonstrates the effectiveness of the proposed methods on tracking the evolutions of dynamic networks.

This paper focuses on the topic of multi-hop question generation, which aims to generate questions needed reasoning over multiple sentences and relations to derive answers. In particular, we first build an entity graph to integrate various entities scattered over text based on their contextual relations. We then heuristically extract the sub-graph by the evidential relations and type, so as to obtain the reasoning chain and textual related contents for each question. Guided by the chain, we propose a holistic generator-evaluator network to form the questions, where such guidance helps to ensure the rationality of generated questions which need multi-hop deduction to correspond to the answers. The generator is a sequence-to-sequence model, designed with several techniques to make the questions syntactically and semantically valid. The evaluator optimizes the generator network by employing a hybrid mechanism combined of supervised and reinforced learning. Experimental results on HotpotQA data set demonstrate the effectiveness of our approach, where the generated samples can be used as pseudo training data to alleviate the data shortage problem for neural network and assist to learn the state-of-the-arts for multi-hop machine comprehension.

This paper focuses on the topic of inferential machine comprehension, which aims to fully understand the meanings of given text to answer generic questions, especially the ones needed reasoning skills.In particular, we first encode the given document, question and options in a context aware way.We then propose a new network to solve the inference problem by decomposing it into a series of attentionbased reasoning steps.The result of the previous step acts as the context of next step.To make each step can be directly inferred from the text, we design an operational cell with prior structure.By recursively linking the cells, the inferred results are synthesized together to form the evidence chain for reasoning, where the reasoning direction can be guided by imposing structural constraints to regulate interactions on the cells.Moreover, a termination mechanism is introduced to dynamically determine the uncertain reasoning depth, and the network is trained by reinforcement learning.Experimental results on 3 popular data sets, including MCTest, RACE and MultiRC, demonstrate the effectiveness of our approach.

Numerous consumer reviews of products are now available on the Internet. Consumer reviews contain rich and valuable knowledge for both firms and users. However, the reviews are often disorganized, leading to difficulties in information navigation and knowledge acquisition. This article proposes a product aspect ranking framework, which automatically identifies the important aspects of products from online consumer reviews, aiming at improving the usability of the numerous reviews. The important product aspects are identified based on two observations: 1) the important aspects are usually commented on by a large number of consumers and 2) consumer opinions on the important aspects greatly influence their overall opinions on the product. In particular, given the consumer reviews of a product, we first identify product aspects by a shallow dependency parser and determine consumer opinions on these aspects via a sentiment classifier. We then develop a probabilistic aspect ranking algorithm to infer the importance of aspects by simultaneously considering aspect frequency and the influence of consumer opinions given to each aspect over their overall opinions. The experimental results on a review corpus of 21 popular products in eight domains demonstrate the effectiveness of the proposed approach. Moreover, we apply product aspect ranking to two real-world applications, i.e., document-level sentiment classification and extractive review summarization, and achieve significant performance improvements, which demonstrate the capacity of product aspect ranking in facilitating real-world applications.