Recently, there has been a heightened interest in building chatbots based on Large Language Models (LLMs) to emulate human-like qualities in multi-turn conversations. Despite having access to commonsense knowledge to better understand the psychological aspects and causality of dialogue context, even these powerful LLMs struggle to achieve the goals of empathy and emotional support. Current commonsense knowledge derived from dialogue contexts is inherently limited and often fails to adequately anticipate the future course of a dialogue. This lack of foresight can mislead LLMs and hinder their ability to provide effective support. In response to this challenge, we present an innovative framework named Sensible and Visionary Commonsense Knowledge (Sibyl). Designed to concentrate on the immediately succeeding dialogue, this paradigm equips LLMs with the capability to uncover the implicit requirements of the conversation, aiming to elicit more empathetic responses. Experimental results demonstrate that incorporating our paradigm for acquiring commonsense knowledge into LLMs comprehensively enhances the quality of their responses.

Xinyu Liu, Bei Li, Jiahao Liu, Junhao Ruan, Kechen Jiao, Hongyin Tang, Jingang Wang, Tong Xiao, JingBo Zhu. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Processing structured tabular data, particularly large and lengthy tables, constitutes a fundamental yet challenging task for large language models (LLMs). However, existing long-context benchmarks like Needle-in-a-Haystack primarily focus on unstructured text, neglecting the challenge of diverse structured tables. Meanwhile, previous tabular benchmarks mainly consider downstream tasks that require high-level reasoning abilities, and overlook models' underlying fine-grained perception of individual table cells, which is crucial for practical and robust LLM-based table applications. To address this gap, we introduce \textsc{NeedleInATable} (NIAT), a new long-context tabular benchmark that treats each table cell as a ``needle'' and requires models to extract the target cell based on cell locations or lookup questions. Our comprehensive evaluation of various LLMs and multimodal LLMs reveals a substantial performance gap between popular downstream tabular tasks and the simpler NIAT task, suggesting that they may rely on dataset-specific correlations or shortcuts to obtain better benchmark results but lack truly robust long-context understanding towards structured tables. Furthermore, we demonstrate that using synthesized NIAT training data can effectively improve performance on both NIAT task and downstream tabular tasks, which validates the importance of NIAT capability for LLMs' genuine table understanding ability. Our data, code and models will be released to facilitate future research.

Most previous sequential labeling models are task-specific, while recent years have witnessed the rise of generative models due to the advantage of unifying all named entity recognition (NER) tasks into the encoder-decoder framework. Although achieving promising performance, our pilot studies demonstrate that existing generative models are ineffective at detecting entity boundaries and estimating entity types. In this paper, we propose a multi-task Transformer, which incorporates an entity boundary detection task into the named entity recognition task. More concretely, we achieve entity boundary detection by classifying the relations between tokens within the sentence. To improve the accuracy of entity-type mapping during decoding, we adopt an external knowledge base to calculate the prior entity-type distributions and then incorporate the information into the model via the self- and cross-attention mechanisms. We perform experiments on extensive NER benchmarks, including flat, nested, and discontinuous NER datasets involving long entities. It substantially increases nearly$+0.3 \sim +1.5\;{F_1}$scores across a broad spectrum or performs closely to the best generative NER model. Experimental results show that our approach improves the performance of the generative NER model considerably.

Named entity recognition (NER) is an important research problem in natural language processing. There are three types of NER tasks, including flat, nested and discontinuous entity recognition. Most previous sequential labeling models are task-specific, while recent years have witnessed the rising of generative models due to the advantage of unifying all NER tasks into the seq2seq model framework. Although achieving promising performance, our pilot studies demonstrate that existing generative models are ineffective at detecting entity boundaries and estimating entity types. This paper proposes a multi-task Transformer, which incorporates an entity boundary detection task into the named entity recognition task. More concretely, we achieve entity boundary detection by classifying the relations between tokens within the sentence. To improve the accuracy of entity-type mapping during decoding, we adopt an external knowledge base to calculate the prior entity-type distributions and then incorporate the information into the model via the self and cross-attention mechanisms. We perform experiments on an extensive set of NER benchmarks, including two flat, three nested, and three discontinuous NER datasets. Experimental results show that our approach considerably improves the generative NER model’s performance.

Pre-trained Language Models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding. Various Chinese PLMs have been successively proposed for learning better Chinese language representation. However, most current models use Chinese characters as inputs and are not able to encode semantic information contained in Chinese words. While recent pre-trained models incorporate both words and characters simultaneously, they usually suffer from deficient semantic interactions and fail to capture the semantic relation between words and characters. To address the above issues, we propose a simple yet effective PLM CLOWER, which adopts the Contrastive Learning Over Word and charactER representations. In particular, CLOWER implicitly encodes the coarse-grained information (i.e., words) into the fine-grained representations (i.e., characters) through contrastive learning on multi-grained information. CLOWER is of great value in realistic scenarios since it can be easily incorporated into any existing fine-grained based PLMs without modifying the production pipelines. Extensive experiments conducted on a range of downstream tasks demonstrate the superior performance of CLOWER over several state-of-the-art baselines.

Text matching is a fundamental research problem in natural language understanding.Interaction-based approaches treat the text pair as a single sequence and encode it through cross encoders, while representation-based models encode the text pair independently with siamese or dual encoders.Interactionbased models require dense computations and thus are impractical in real-world applications.Representation-based models have become the mainstream paradigm for efficient text matching.However, these models suffer from severe performance degradation due to the lack of interactions between the pair of texts.To remedy this, we propose a Virtual InteRacTion mechanism (VIRT) for improving representation-based text matching while maintaining its efficiency.In particular, we introduce an interactive knowledge distillation module that is only applied during training.It enables deep interaction between texts by effectively transferring knowledge from the interaction-based model.A light interaction strategy is designed to fully leverage the learned interactive knowledge.Experimental results on six text matching benchmarks demonstrate the superior performance of our method over several state-of-the-art representationbased models.We further show that VIRT can be integrated into existing methods as plugins to lift their performances.

Recently, a zero-shot entity linking task is introduced to challenge the generalization ability of entity linking models. In this task, mentions must be linked to unseen entities and only the textual information is available. In order to make full use of the documents, previous work has proposed a BERT-based model which can only take fixed length of text as input. However, the key information for entity linking may exist in nearly everywhere of the documents thus the proposed model cannot capture them all. To leverage more textual information and enhance text understanding capability, we propose a bidirectional multi-paragraph reading model for the zero-shot entity linking task. Firstly, the model treats the mention context as a query and matches it with multiple paragraphs of the entity description documents. Then, the mention-aware entity representation obtained from the first step is used as a query to match multiple paragraphs in the document containing the mention through an entity-mention attention mechanism. In particular, a new pre-training strategy is employed to strengthen the representative ability. Experimental results show that our bidirectional model can capture long-range context dependencies and outperform the baseline model by 3-4% in terms of accuracy.

Hongyin Tang, Xingwu Sun, Beihong Jin, Jingang Wang, Fuzheng Zhang, Wei Wu. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

In this paper, we propose a new ranking model DR-BERT, which improves the Document Retrieval (DR) task by a task-adaptive training process and a Segmented Token Recovery Mechanism (STRM).In the task-adaptive training, we first pre-train DR-BERT to be domain-adaptive and then make the two-phase fine-tuning.In the first-phase fine-tuning, the model learns query-document matching patterns regarding different query types in a pointwise way.Next, in the second-phase finetuning, the model learns document-level ranking features and ranks documents with regard to a given query in a listwise manner.Such pointwise plus listwise fine-tuning enables the model to minimize errors in the document ranking by incorporating ranking-specific supervisions.Meanwhile, the model derived from pointwise fine-tuning is also used to reduce noise in the training data of the listwise fine-tuning.On the other hand, we present STRM which can compute OOV word representation and contextualization more precisely in BERT-based models.As an effective strategy in DR-BERT, STRM improves the matching perfromance of OOV words between a query and a document.Notably, our DR-BERT model keeps in the top three on the MS MARCO leaderboard since May 20, 2020.

For autonomous driving in extremely complex scenarios, existing research utilizes deep reinforcement learning or imitation learning to obtain the decision-making capability of agents. However, due to the incomplete information nature of such driving scenarios, existing techniques usually suffer issues such as the incorrect rewards or unstable training which would impact the learning quality seriously. In this paper, we propose a new approach named Columba which trains the agent to learn from expert trajectory data and abnormal trajectory data instead of relying on any manually-set reward functions. In particular, Columba designs a positive and negative feedback regulator to reduce the dangerous or bad states of the car agent at the beginning of training. Further, Columba generates the rewards by coordinating with the discriminator, the random distillation network and the regulator, enhancing the accuracy of rewards. We conduct extensive experiments on the Torcs simulation platform. Experimental results show that the agent trained by Columba outperforms the agents trained by DDPG and GAIL, which are strong baselines in the deep reinforcement learning and the imitation learning, respectively.

Xingwu Sun, Yanling Cui, Hongyin Tang, Qiuyu Zhu, Fuzheng Zhang, Beihong Jin. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Document retrieval (DR) is a crucial task in NLP. Recently, the pre-trained BERT-like language models have achieved remarkable success, obtaining a state-of-the-art result in DR. In this paper, we come up with a new BERT-based ranking model for DR task, named TABLE. In the pre-training stage of TABLE, we present a domain-adaptive strategy. More essentially, in the fine-tuning stage, we develop a two-phase task-adaptive process, i.e., type-adaptive pointwise fine-tuning and listwise fine-tuning. In the type-adaptive pointwise fine-tuning phase, the model can learn different matching patterns regarding different query types. In the listwise fine-tuning phase, the model matches documents with regard to a given query in a listwise fashion. This task-adaptive process makes the model more robust. In addition, a simple but effective exact matching feature is introduced in fine-tuning, which can effectively compute matching of out-of-vocabulary (OOV) words between a query and a document. As far as we know, we are the first who propose a listwise ranking model with BERT. This work can explore rich matching features between queries and documents. Therefore it substantially improves model performance in DR. Notably, our TABLE model shows excellent performance on the MS MARCO leaderboard.

In the deep reinforcement learning, the sparse reward problem directly impacts the quality of agent training. Existing methods have not been satisfactory, especially for the scenarios with high-dimensional state information. In this paper, we propose a new approach Monoceros to training a game agent. Monoceros can work for the scenarios with high-dimensional state information and alleviate the sparse reward problem during the agent training. Specifically, we present a composite reward function which combines both the knowledge implied in expert trajectories and manually-set reward functions. Moreover, we design a specific policy network to adapt to the high-dimensional information scenarios, and adopt the behavior clone as a pre-training strategy to accelerate the training process. Technically, Monoceros can be applied to train the agents to play First Person Shooter (FPS) games. We conduct extensive experiments on three scenarios in the VIZDoom platform. Experimental results show that in all the scenarios, the agent trained by Monoceros outperforms the agents trained by Arnold and GAIL, which are representative methods in the deep reinforcement learning and the imitation learning, respectively.

Hongyin Tang, Miao Li, Beihong Jin. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Text clustering is one of the fundamental tasks in natural language processing and text data mining. It remains challenging because texts have complex internal structure besides the sparsity in the high-dimensional representation. In the paper, we propose a new Neural Variational model with mixture-of-Gaussians prior for Text Clustering (abbr. NVTC) to reveal the underlying textual manifold structure and cluster documents effectively. NVTC is a deep latent variable model built on the basis of the neural variational inference. In NVTC, the stochastic latent variable, which is modeled as one obeying a Gaussian mixture distribution, plays an important role in establishing the association of documents and document labels. On the other hand, by joint learning, NVTC simultaneously learns text encoded representations and cluster assignments. Experimental results demonstrate that NVTC is able to learn clustering-friendly representations of texts. It significantly outperforms several baselines including VAE+GMM, VaDE, LCK-NFC, GSDPMM and LDA on four benchmark text datasets in terms of ACC, NMI, and AMI. Furthermore, NVTC learns effective latent embeddings of texts which are interpretable by topics of texts, where each dimension of latent embeddings corresponds to a specific topic.

In large and medium-sized cities, detecting unusual changes of crowds of people on the streets is needed for public security, transportation management, emergency control, and terrorism prevention. As public transportation has the capability to bring a large number of people to an area in a short amount of time, real-time discovery of anomalies in passenger numbers is an effective way to detect crowd anomalies. In this paper, we devise an approach called Kochab. Kochab adopts a generative model and combines the prior knowledge about passenger flows. Hence, it can detect anomalies in the numbers of incoming and outgoing passengers within a certain time and spatial area, including anomalous events along with their durations and severities. Through well-designed inference algorithms, Kochab requires only a moderate amount of historical data to be sample data. As such, Kochab shows good performance in real time and makes prompt responses to user' s interactive analysis requests. In particular, based on the recognized anomalous events, we capture event patterns which give us hints to link to activities or status in cities. In addition, for the convenience of method evaluation and comparison, we create an open Stream Anomaly Benchmark on the basis of large-scale real-world data. This benchmark will prove useful for other researchers too. Using this benchmark, we compare Kochab with four other methods. The experimental results show that Kochab is sensitive to population flow anomalies and has superior accuracy in detecting anomalies in terms of precision, recall and the F1 score.

With the advent of Web 2.0 application, and the increasing number of browsers and platforms on which the applications can be executed, cross-browser incompatibilities (XBIs) are becoming a serious problem for organizations to develop web-based software. Although some techniques and tools have been proposed to identify XBIs, a number of false positives and false negatives still exist as they cannot assure the same execution when the application runs across different browsers. To address this limitation, leveraging existing record/replay technique, we developed X-Check, a novel cross-browser testing technique and tool, which supports automated XBIs detection with high accuracy. Our empirical evaluation shows that X-Check is effective and improves the state of the art.

With the advent of Web 2.0 application, and the increasing number of browsers and platforms on which the applications can be executed, cross-browser incompatibilities (XBIs) are becoming a serious problem for organizations to develop web-based software. Although some techniques and tools have been proposed to identify XBIs, they cannot assure the same execution when the application runs across different browsers as only explicit user activity is considered, and thus prone to generating both false positives and false negatives. To address this limitation, this paper describes X-Check, a platform that enables cross-browser testing as a service by leveraging record/replay technique. Comparing to existing techniques and tools, X-Check supports to detect cross-browser issues with high accuracy. It also provides useful support to developers for diagnosis and (eventually) elimination of XBIs. Our empirical evaluation shows that X-Check is effective, improves the state of the art.

Mobile systems usually support an event-based model of concurrent programming. This model, although advantageous to maintain responsive user interfaces, may lead to subtle concurrency errors due to unforeseen threads interleaving coupled with non-deterministic reordering of asynchronous events. These bugs are very difficult to reproduce even by the same user action sequences that trigger them, due to the undetermined schedules of underlying events and threads. In this paper, we proposed RacerDroid, a novel technique that aims to expose concurrency bugs in android applications by actively controlling event schedule and thread interleaving, given the test cases that have potential data races. By exploring the state model of the application constructed dynamically, our technique starts first to generate a test case that has potential data races based on the results obtained from existing static or dynamic race detection technique. Then it reschedules test cases execution by actively controlling event dispatching and thread interleaving to determine whether such potential races really lead to thrown exceptions or assertion violations. Our preliminary experiments show that RacerDroid is effective, and it confirms real data races, while at the same time eliminates false warnings for Android apps found in the wild.

With the advent of Web 2.0 application, and the increasing number of browsers and platforms on which the applications can be executed, cross-browser incompatibilities (XBIs) are becoming a serious problem for organizations to develop web-based software with good user experience. Although some techniques and tools have been proposed to identify XBIs, some XBIs are still missed as only partial state space is explored (by the crawler) in the testing environment. To address this limitation, based on record/replay technique, this paper proposed a crowdsourcing framework to detect cross-browser issues for Web application deployed in the field. Our empirical evaluation shows that the proposed technique is effective and efficient, improves on the state of the art.