Yumo Xu, Peng Qi, Jifan Chen, Kunlun Liu, Rujun Han, Lan Liu, Bonan Min, Vittorio Castelli, Arshit Gupta, Zhiguo Wang. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Rujun Han, Yuhao Zhang, Peng Qi, Yumo Xu, Jenyuan Wang, Lan Liu, William Yang Wang, Bonan Min, Vittorio Castelli. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Modern language models (LMs) need to follow human instructions while being faithful; yet, they often fail to achieve both.Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow open-ended instructions) and faithfulness (i.e., ground responses in given context) when training LMs with these objectives.For instance, fine-tuning LLaMA-7B on instruction following datasets renders it less faithful.Conversely, instructiontuned Vicuna-7B shows degraded performance at following instructions when further optimized on tasks that require contextual grounding.One common remedy is multi-task learning (MTL) with data mixing, yet it remains far from achieving a synergic outcome.We propose a simple yet effective method that relies on Rejection Sampling for Continued Selfinstruction Tuning (RESET), which significantly outperforms vanilla MTL.Surprisingly, we find that less is more, as training RESET with high-quality, yet substantially smaller data (three-fold less) yields superior results.Our findings offer a better understanding of objective discrepancies in alignment training of LMs.

Alexander Hanbo Li, Mingyue Shang, Evangelia Spiliopoulou, Jie Ma, Patrick Ng, Zhiguo Wang, Bonan Min, William Yang Wang, Kathleen McKeown, Vittorio Castelli, Dan Roth, Bing Xiang. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

This paper explores utilizing Large Language Models (LLMs) to perform Cross-Document Event Coreference Resolution (CDEC) annotations and evaluates how they fare against human annotators with different levels of training.Specifically, we formulate CDEC as a multiclass classification problem on pairs of events that are represented as decontextualized sentences, and compare the predictions of GPT-4 with the judgment of fully trained annotators and crowdworkers on the same dataset.Our study indicates that GPT-4 with zero-shot learning outperformed crowd-workers by a large margin and exhibits a level of performance comparable to trained annotators.Upon closer analysis, GPT-4 also exhibits tendencies of being overly confident, and forcing annotation decisions even when such decisions are not warranted due to insufficient information.Our results have implications on how to perform complicated annotations such as CDEC in the age of LLMs, and show that the best way to acquire such annotations might be to combine the strengths of LLMs and trained human annotators in the annotation process, and using untrained or undertrained crowdworkers is no longer a viable option to acquire high-quality data to advance the state of the art for such problems.We make our source and data publicly available.1

Extracting entities, events, event arguments, and relations (i.e., task instances) from text represents four main challenging tasks in information extraction (IE), which have been solved jointly (JointIE) to boost the overall performance for IE.As such, previous work often leverages two types of dependencies between the tasks, i.e., cross-instance and cross-type dependencies representing relatedness between task instances and correlations between information types of the tasks.However, the crosstask dependencies in prior work are not optimal as they are only designed manually according to some task heuristics.To address this issue, we propose a novel model for JointIE that aims to learn cross-task dependencies from data.In particular, we treat each task instance as a node in a dependency graph where edges between the instances are inferred through information from different layers of a pretrained language model (e.g., BERT).Furthermore, we utilize the Chow-Liu algorithm to learn a dependency tree between information types for JointIE by seeking to approximate the joint distribution of the types from data.Finally, the Chow-Liu dependency tree is used to generate cross-type patterns, serving as anchor knowledge to guide the learning of representations and dependencies between instances for JointIE.Experimental results show that our proposed model significantly outperforms strong JointIE baselines over four datasets with different languages.

Minh Van Nguyen, Bonan Min, Franck Dernoncourt, Thien Nguyen. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

The task of modal dependency parsing aims to parse a text into its modal dependency structure, which is a representation for the factuality of events in the text.We design a modal dependency parser that is based on priming pre-trained language models, and evaluate the parser on two data sets.Compared to baselines, we show an improvement of 2.6% in F-score for English and 4.6% for Chinese.To the best of our knowledge, this is also the first work on Chinese modal dependency parsing.

Jiarui Yao, Haoling Qiu, Jin Zhao, Bonan Min, Nianwen Xue. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Previous work on crosslingual Relation and Event Extraction (REE) suffers from the monolingual bias issue due to the training of models on only the source language data.An approach to overcome this issue is to use unlabeled data in the target language to aid the alignment of crosslingual representations, i.e., via fooling a language discriminator.However, as this approach does not condition on class information, a target language example of a class could be incorrectly aligned to a source language example of a different class.To address this issue, we propose a novel crosslingual alignment method that leverages class information of REE tasks for representation learning.In particular, we propose to learn two versions of representation vectors for each class in an REE task based on either source or target language examples.Representation vectors for corresponding classes will then be aligned to achieve class-aware alignment for crosslingual representations.In addition, we propose to further align representation vectors for languageuniversal word categories (i.e., parts of speech and dependency relations).As such, a novel filtering mechanism is presented to facilitate the learning of word category representations from contextualized representations on input texts based on adversarial learning.We conduct extensive crosslingual experiments with English, Chinese, and Arabic over REE tasks.The results demonstrate the benefits of the proposed method that significantly advances the state-of-the-art performance in these settings.

Event Detection (ED) aims to recognize and classify trigger words of events in text.The recent progress has featured advanced transformer-based language models (e.g., BERT) as a critical component in stateof-the-art models for ED.However, the length limit for input texts is a barrier for such ED models as they cannot encode long-range document-level context that has been shown to be beneficial for ED.To address this issue, we propose a novel method to model documentlevel context with BERT for ED that dynamically selects relevant sentences in the document for the event prediction of the target sentence.The target sentence will be then augmented with the selected sentences and consumed entirely by BERT for improved representation learning for ED.To this end, the RE-INFORCE algorithm is employed to train the relevant sentence selection for ED.Several information types are then introduced to form the reward function for the training process, including ED performance, sentence similarity, and discourse relations.Our extensive experiments on multiple benchmark datasets reveal the effectiveness of the proposed model, leading to new state-of-the-art performance.

Many researchers have explored type inference for dynamic languages. However, traditional type inference computes most general types which, for complex type systems—which are often needed to type dynamic languages—can be verbose, complex, and difficult to understand. In this paper, we introduce SimTyper, a Ruby type inference system that aims to infer usable types—specifically, nominal and generic types—that match the types programmers write. SimTyper builds on InferDL, a recent Ruby type inference system that soundly combines standard type inference with heuristics. The key novelty of SimTyper is type equality prediction , a new, machine learning-based technique that predicts when method arguments or returns are likely to have the same type. SimTyper finds pairs of positions that are predicted to have the same type yet one has a verbose, overly general solution and the other has a usable solution. It then guesses the two types are equal, keeping the guess if it is consistent with the rest of the program, and discarding it if not. In this way, types inferred by SimTyper are guaranteed to be sound. To perform type equality prediction, we introduce the deep similarity (DeepSim) neural network. DeepSim is a novel machine learning classifier that follows the Siamese network architecture and uses CodeBERT, a pre-trained model, to embed source tokens into vectors that capture tokens and their contexts. DeepSim is trained on 100,000 pairs labeled with type similarity information extracted from 371 Ruby programs with manually documented, but not checked, types. We evaluated SimTyper on eight Ruby programs and found that, compared to standard type inference, SimTyper finds 69% more types that match programmer-written type information. Moreover, DeepSim can predict rare types that appear neither in the Ruby standard library nor in the training data. Our results show that type equality prediction can help type inference systems effectively produce more usable types.

We present a system which allows a user to create event-event relation extractors on-demand with a small amount of effort. The system provides a suite of algorithms, flexible workflows, and a user interface (UI), to allow rapid customization of event-event relation extractors for new types and domains of interest. Experiments show that it enables users to create extractors for 6 types of causal and temporal relations, with less than 20 minutes of effort per type. Our system (source code, UI) is available at https://github.com/BBN-E/LearnIt. A demonstration video is available at https://vimeo.com/329950144.

Extracting temporal relations between events and time expressions has many applications such as constructing event timelines and timerelated question answering.It is a challenging problem which requires syntactic and semantic information at sentence or discourse levels, which may be captured by deep contextualized language models (LMs) such as BERT (Devlin et al., 2019).In this paper, we develop several variants of BERT-based temporal dependency parser, and show that BERT significantly improves temporal dependency parsing (Zhang and Xue, 2018a).We also present a detailed analysis on why deep contextualized neural LMs help and where they may fall short.

Subevents elaborate an event and widely exist in event descriptions.Subevent knowledge is useful for discourse analysis and event-centric applications.Acknowledging the scarcity of subevent knowledge, we propose a weakly supervised approach to extract subevent relation tuples from text and build the first large scale subevent knowledge base.We first obtain the initial set of event pairs that are likely to have the subevent relation, by exploiting two observations that 1) subevents are temporally contained by the parent event, and 2) the definitions of the parent event can be used to further guide the identification of subevents.Then, we collect rich weak supervision using the initial seed subevent pairs to train a contextual classifier using BERT and apply the classifier to identify new subevent pairs.The evaluation showed that the acquired subevent tuples (239K) are of high quality (90.1% accuracy) and cover a wide range of event types.The acquired subevent knowledge has been shown useful for discourse analysis and identifying a range of event-event relations 1 .

We present the construction of a corpus of 500 Wikinews articles annotated with temporal dependency graphs (TDGs) that can be used to train systems to understand temporal relations in text.We argue that temporal dependency graphs, built on previous research on narrative times and temporal anaphora, provide a representation scheme that achieves a good balance between completeness and practicality in temporal annotation.We also provide a crowdsourcing strategy to annotate TDGs, and demonstrate the feasibility of this approach with an evaluation of the quality of the annotation, and the utility of the resulting data set by training a machine learning model on this data set.This data set is publicly available 1 .

Automatically analyzing events in a large amount of text is crucial for situation awareness and decision making. Previous approaches treat event extraction as “one size fits all” with an ontology defined a priori. The resulted extraction models are built just for extracting those types in the ontology. These approaches cannot be easily adapted to new event types nor new domains of interest. To accommodate personalized event-centric information needs, this paper introduces the few-shot Event Mention Retrieval (EMR) task: given a user-supplied query consisting of a handful of event mentions, return relevant event mentions found in a corpus. This formulation enables “query by example”, which drastically lowers the bar of specifying event-centric information needs. The retrieval setting also enables fuzzy search. We present an evaluation framework leveraging existing event datasets such as ACE. We also develop a Siamese Network approach, and show that it performs better than ad-hoc retrieval models in the few-shot EMR setting.

Bonan Min, Yee Seng Chan, Haoling Qiu, Joshua Fasching. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Bonan Min, Xiaoxi Zhao. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Building knowledge bases (KB) automatically from text corpora is crucial for many applications such as question answering and web search.The problem is very challenging and has been divided into sub-problems such as mention and named entity recognition, entity linking and relation extraction.However, combining these components has shown to be under-constrained and often produces KBs with supersize entities and common-sense errors in relations (a person has multiple birthdates).The errors are difficult to resolve solely with IE tools but become obvious with world knowledge at the corpus level.By analyzing Freebase and a large text collection, we found that per-relation cardinality and the popularity of entities follow the power-law distribution favoring flat long tails with lowfrequency instances.We present a probabilistic joint inference algorithm to incorporate this world knowledge during KB construction.Our approach yields stateof-the-art performance on the TAC Cold Start task, and 42% and 19.4% relative improvements in F1 over our baseline on Cold Start hop-1 and all-hop queries respectively.

Typically, relation extraction models are trained to extract instances of a relation ontology using only training data from a single language. However, the concepts represented by the relation ontology (e.g. ResidesIn, EmployeeOf) are language independent. The numbers of annotated examples available for a given ontology vary between languages. For example, there are far fewer annotated examples in Spanish and Japanese than English and Chinese. Furthermore, using only language-specific training data results in the need to manually annotate equivalently large amounts of training for each new language a system encounters. We propose a deep neural network to learn transferable, discriminative bilingual representation. Experiments on the ACE 2005 multilingual training corpus demonstrate that the joint training process results in significant improvement in relation classification performance over the monolingual counterparts. The learnt representation is discriminative and transferable between languages. When using 10% (25K English words, or 30K Chinese characters) of the training data, our approach results in doubling F1 compared to a monolingual baseline. We achieve comparable performance to the monolingual system trained with 250K English words (or 300K Chinese characters) With 50% of training data.

The Web brings an open-ended set of semantic relations. Discovering the significant types is very challenging. Unsupervised algorithms have been developed to extract relations from a corpus without knowing the relation types in advance, but most rely on tagging arguments of predefined types. One recently reported system is able to jointly extract relations and their argument semantic classes, taking a set of relation instances extracted by an open IE (Information Extraction) algorithm as input. However, it cannot handle polysemy of relation phrases and fails to group many similar (“synonymous”) relation instances because of the sparseness of features. In this paper, the authors present a novel unsupervised algorithm that provides a more general treatment of the polysemy and synonymy problems. The algorithm incorporates various knowledge sources which they will show to be very effective for unsupervised relation extraction. Moreover, it explicitly disambiguates polysemous relation phrases and groups synonymous ones. While maintaining approximately the same precision, the algorithm achieves significant improvement on recall compared to the previous method. It is also very efficient. Experiments on a real-world dataset show that it can handle 14.7 million relation instances and extract a very large set of relations from the Web.

Peer-to-Peer (P2P) applications have been consuming an increasingly significant fraction of Internet bandwidth. They are becoming a financial burden to Internet Service Providers (ISPs), creating hot spots in the Internet, and causing potential performance degradation to other applications. As a result, there has been increasing tensions between P2P applications and network service providers. In this paper, we propose a framework called SODA for P2P applications to be self-adaptive and optimize their demands in order to more efficiently utilize network resources. Through preliminary experiments using two representative P2P applications, we demonstrate that SODA can effectively reduce bandwidth consumption by adapting the demands among peers.