Discourse coherence theories posit relations between text spans as a key feature of coherent texts.However, existing work on coherence modeling has paid little attention to discourse relations.In this paper, we provide empirical evidence to demonstrate that relation features are correlated with text coherence.Then, we investigate a novel fusion model that uses position-aware attention and a visible matrix to combine text-and relation-based features for coherence assessment.Experimental results on two benchmarks show that our approaches can significantly improve baselines, demonstrating the importance of relation features for coherence modeling.

In linguistics, coherence can be achieved by different means, such as by maintaining reference to the same set of entities across sentences and by establishing discourse relations between them.However, most existing work on coherence modeling focuses exclusively on either entity features or discourse relation features, with little attention given to combining the two.In this study, we explore two methods for jointly modeling entities and discourse relations for coherence assessment.Experiments on three benchmark datasets show that integrating both types of features significantly enhances the performance of coherence models, highlighting the benefits of modeling both simultaneously for coherence evaluation.

Despite the predominance of contextualized embeddings in NLP, approaches to detect semantic change relying on these embeddings and clustering methods underperform simpler counterparts based on static word embeddings.This stems from the poor quality of the clustering methods to produce sense clusters-which struggle to capture word senses, especially those with low frequency.This issue hinders the next step in examining how changes in word senses in one language influence another.To address this issue, we propose a graph-based clustering approach to capture nuanced changes in both high-and low-frequency word senses across time and languages, including the acquisition and loss of these senses over time.Our experimental results show that our approach substantially surpasses previous approaches in the SemEval2020 binary classification task across four languages.Moreover, we showcase the ability of our approach as a versatile visualization tool to detect semantic changes in both intra-language and inter-language setups.We make our code and data available 1 .

Wei Liu, Stephen Wan, Michael Strube. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Automating Cross-lingual Science Journalism (CSJ) aims to generate popular science summaries from English scientific texts for nonexpert readers in their local language.We introduce CSJ as a downstream task of text simplification and cross-lingual scientific summarization to facilitate science journalists' work.We analyze the performance of possible existing solutions as baselines for the CSJ task.Based on these findings, we propose to combine the three components -SELECT, SIMPLIFY and REWRITE (SSR) to produce cross-lingual simplified science summaries for non-expert readers.Our empirical evaluation on the WIKIPEDIA dataset shows that SSR significantly outperforms the baselines for the CSJ task and can serve as a strong baseline for future work.We also perform an ablation study investigating the impact of individual components of SSR.Further, we analyze the performance of SSR on a high-quality, real-world CSJ dataset with human evaluation and in-depth analysis, demonstrating the superior performance of SSR for CSJ. A Scientific and News StructureFigure A.1 presents the difference between a scientific text discourse and a news text discourse.

Implicit discourse relation classification is a challenging task due to the absence of discourse connectives.To overcome this issue, we design an end-to-end neural model to explicitly generate discourse connectives for the task, inspired by the annotation process of PDTB.Specifically, our model jointly learns to generate discourse connectives between arguments and predict discourse relations based on the arguments and the generated connectives.To prevent our relation classifier from being misled by poor connectives generated at the early stage of training while alleviating the discrepancy between training and inference, we adopt Scheduled Sampling to the joint learning.We evaluate our method on three benchmarks, PDTB 2.0, PDTB 3.0, and PCC.Results show that our joint model significantly outperforms various baselines on three datasets, demonstrating its superiority for the task.

Coherence is an important aspect of text quality, and various approaches have been applied to coherence modeling.However, existing methods solely focus on a single document's coherence patterns, ignoring the underlying correlation between documents.We investigate a GCN-based coherence model that is capable of capturing structural similarities between documents.Our model first creates a graph structure for each document, from where we mine different subgraph patterns.We then construct a heterogeneous graph for the training corpus, connecting documents based on their shared subgraphs.Finally, a GCN is applied to the heterogeneous graph to model the connectivity relationships.We evaluate our method on two tasks, assessing discourse coherence and automated essay scoring.Results show that our GCN-based model outperforms all baselines, achieving a new state-of-the-art on both tasks.

Recently, there has been a growing interest in designing text generation systems from a discourse coherence perspective, e.g., modeling the interdependence between sentences.Still, recent BERT-based evaluation metrics are weak in recognizing coherence, and thus are not reliable in a way to spot the discourselevel improvements of those text generation systems.In this work, we introduce DiscoScore, a parametrized discourse metric, which uses BERT to model discourse coherence from different perspectives, driven by Centering theory.Our experiments encompass 16 non-discourse and discourse metrics, including DiscoScore and popular coherence models, evaluated on summarization and document-level machine translation (MT).We find that (i) the majority of BERT-based metrics correlate much worse with human rated coherence than early discourse metrics, invented a decade ago; (ii) the recent state-of-the-art BARTScore is weak when operated at system level-which is particularly problematic as systems are typically compared in this manner.DiscoScore, in contrast, achieves strong system-level correlation with human ratings, not only in coherence but also in factual consistency and other aspects, and surpasses BARTScore by over 10 correlation points on average.Further, aiming to understand DiscoScore, we provide justifications to the importance of discourse coherence for evaluation metrics, and explain the superiority of one variant over another.Our code is available at https://github.com/AIPHES/ DiscoScore.

In this paper, we propose an entity-based neural local coherence model which is linguistically more sound than previously proposed neural coherence models.Recent neural coherence models encode the input document using large-scale pretrained language models.Hence their basis for computing local coherence are words and even sub-words.An analysis of their output shows that these models frequently compute coherence on the basis of connections between (sub-)words which, from a linguistic perspective, should not play a role.Still, these models achieve state-of-the-art performance in several end applications.In contrast to these models, we compute coherence on the basis of entities by constraining the input to noun phrases and proper names.This provides us with an explicit representation of the most important items in sentences leading to the notion of focus.This brings our model linguistically in line with pre-neural models of computing coherence.It also gives us better insight into the behaviour of the model thus leading to better explainability.Our approach is also in accord with a recent study (O'Connor and Andreas, 2021), which shows that most usable information is captured by nouns and verbs in transformer-based language models.We evaluate our model on three downstream tasks showing that it is not only linguistically more sound than previous models but also that it outperforms them in end applications 1 .

In recent years, transformer-based coreference resolution systems have achieved remarkable improvements on the CoNLL dataset.However, how coreference resolvers can benefit from discourse coherence is still an open question.In this paper, we propose to incorporate centering transitions derived from centering theory in the form of a graph into a neural coreference model.Our method improves the performance over the SOTA baselines, especially on pronoun resolution in long documents, formal well-structured text, and clusters with scattered mentions.1

Learning faithful graph representations as sets of vertex embeddings has become a fundamental intermediary step in a wide range of machine learning applications. We propose the systematic use of symmetric spaces in representation learning, a class encompassing many of the previously used embedding targets. This enables us to introduce a new method, the use of Finsler metrics integrated in a Riemannian optimization scheme, that better adapts to dissimilar structures in the graph. We develop a tool to analyze the embeddings and infer structural properties of the data sets. For implementation, we choose Siegel spaces, a versatile family of symmetric spaces. Our approach outperforms competitive baselines for graph reconstruction tasks on various synthetic and real-world datasets. We further demonstrate its applicability on two downstream tasks, recommender systems and node classification.

We propose the use of the vector-valued distance to compute distances and extract geometric information from the manifold of symmetric positive definite matrices (SPD), and develop gyrovector calculus, constructing analogs of vector space operations in this curved space. We implement these operations and showcase their versatility in the tasks of knowledge graph completion, item recommendation, and question answering. In experiments, the SPD models outperform their equivalents in Euclidean and hyperbolic space. The vector-valued distance allows us to visualize embeddings, showing that the models learn to disentangle representations of positive samples from negative ones.

Pretrained language models, neural models pretrained on massive amounts of data, have established the state of the art in a range of NLP tasks.They are based on a modern machine-learning technique, the Transformer which relates all items simultaneously to capture semantic relations in sequences.However, it differs from what humans do.Humans read sentences one-by-one, incrementally.Can neural models benefit by interpreting texts incrementally as humans do?We investigate this question in coherence modeling.We propose a coherence model which interprets sentences incrementally to capture lexical relations between them.We compare the state of the art in each task, simple neural models relying on a pretrained language model, and our model in two downstream tasks.Our findings suggest that interpreting texts incrementally as humans could be useful to design more advanced models.

Previous neural coherence models have focused on identifying semantic relations between adjacent sentences.However, they do not have the means to exploit structural information.In this work, we propose a coherence model which takes discourse structural information into account without relying on human annotations.We approximate a linguistic theory of coherence, Centering theory, which we use to track the changes of focus between discourse segments.Our model first identifies the focus of each sentence, recognized with regards to the context, and constructs the structural relationship for discourse segments by tracking the changes of the focus.The model then incorporates this structural information into a structure-aware transformer.We evaluate our model on two tasks, automated essay scoring and assessing writing quality.Our results demonstrate that our model, built on top of a pretrained language model, achieves state-of-the-art performance on both tasks.We next statistically examine the identified trees of texts assigned to different quality scores.Finally, we investigate what our model learns in terms of theoretical claims 1 .

Metonymy is a figure of speech in which an entity is referred to by another related entity. The existing datasets of metonymy are either too small in size or lack sufficient coverage. We propose a new, labelled, high-quality corpus of location metonymy called WiMCor, which is large in size and has high coverage. The corpus is harvested semi-automatically from English Wikipedia. We use different labels of varying granularity to annotate the corpus. The corpus can directly be used for training and evaluating automatic metonymy resolution systems. We construct benchmarks for metonymy resolution, and evaluate baseline methods using the new corpus.

Pretrained contextual and non-contextual subword embeddings have become available in over 250 languages, allowing massively multilingual NLP.However, while there is no dearth of pretrained embeddings, the distinct lack of systematic evaluations makes it difficult for practitioners to choose between them.In this work, we conduct an extensive evaluation comparing non-contextual subword embeddings, namely FastText and BPEmb, and a contextual representation method, namely BERT, on multilingual named entity recognition and part-of-speech tagging.We find that overall, a combination of BERT, BPEmb, and character representations works well across languages and tasks.A more detailed analysis reveals different strengths and weaknesses: Multilingual BERT performs well in medium-to high-resource languages, but is outperformed by non-contextual subword embeddings in a low-resource setting.* Work done while at HITS. 1 While language-agnostic, these approaches are not language-independent.See Appendix B for a discussion.

This is a repository copy of Using automatically extracted minimum spans to disentangle coreference evaluation from boundary detection.

Recent work has validated the importance of subword information for word representation learning. Since subwords increase parameter sharing ability in neural models, their value should be even more pronounced in low-data regimes. In this work, we therefore provide a comprehensive analysis focused on the usefulness of subwords for word representation learning in truly low-resource scenarios and for three representative morphological tasks: fine-grained entity typing, morphological tagging, and named entity recognition. We conduct a systematic study that spans several dimensions of comparison: 1) type of data scarcity which can stem from the lack of task-specific training data, or even from the lack of unannotated data required to train word embeddings, or both; 2) language type by working with a sample of 16 typologically diverse languages including some truly low-resource ones (e.g. Rusyn, Buryat, and Zulu); 3) the choice of the subword-informed word representation method. Our main results show that subword-informed models are universally useful across all language types, with large gains over subword-agnostic embeddings. They also suggest that the effective use of subwords largely depends on the language (type) and the task at hand, as well as on the amount of available data for training the embeddings and task-based models, where having sufficient in-task data is a more critical requirement.

We propose a local coherence model that captures the flow of what semantically connects adjacent sentences in a text.We represent the semantics of a sentence by a vector and capture its state at each word of the sentence.We model what relates two adjacent sentences based on the two most similar semantic states, each of which is in one of the sentences.We encode the perceived coherence of a text by a vector, which represents patterns of changes in salient information that relates adjacent sentences.Our experiments demonstrate that our approach is beneficial for two downstream tasks: Readability assessment, in which our model achieves new state-of-the-art results; and essay scoring, in which the combination of our coherence vectors and other taskdependent features significantly improves the performance of a strong essay scorer.

This is a repository copy of Using linguistic features to improve the generalization capability of neural coreference resolvers.

In contrast to identity anaphors, which indicate coreference between a noun phrase and its antecedent, bridging anaphors link to their antecedent(s) via lexico-semantic, frame, or encyclopedic relations. Bridging resolution involves recognizing bridging anaphors and finding links to antecedents. In contrast to most prior work, we tackle both problems. Our work also follows a more wide-ranging definition of bridging than most previous work and does not impose any restrictions on the type of bridging anaphora or relations between anaphor and antecedent. We create a corpus (ISNotes) annotated for information status (IS), bridging being one of the IS subcategories. The annotations reach high reliability for all categories and marginal reliability for the bridging subcategory. We use a two-stage statistical global inference method for bridging resolution. Given all mentions in a document, the first stage, bridging anaphora recognition, recognizes bridging anaphors as a subtask of learning fine-grained IS. We use a cascading collective classification method where (i) collective classification allows us to investigate relations among several mentions and autocorrelation among IS classes and (ii) cascaded classification allows us to tackle class imbalance, important for minority classes such as bridging. We show that our method outperforms current methods both for IS recognition overall as well as for bridging, specifically. The second stage, bridging antecedent selection, finds the antecedents for all predicted bridging anaphors. We investigate the phenomenon of semantically or syntactically related bridging anaphors that share the same antecedent, a phenomenon we call sibling anaphors. We show that taking sibling anaphors into account in a joint inference model improves antecedent selection performance. In addition, we develop semantic and salience features for antecedent selection and suggest a novel method to build the candidate antecedent list for an anaphor, using the discourse scope of the anaphor. Our model outperforms previous work significantly.

We introduce automatic verification as a post-processing step for entity linking (EL).The proposed method trusts EL system results collectively, by assuming entity mentions are mostly linked correctly, in order to create a semantic profile of the given text using geospatial and temporal information, as well as fine-grained entity types.This profile is then used to automatically verify each linked mention individually, i.e., to predict whether it has been linked correctly or not.Verification allows leveraging a rich set of global and pairwise features that would be prohibitively expensive for EL systems employing global inference.Evaluation shows consistent improvements across datasets and systems.In particular, when applied to state-of-theart systems, our method yields an absolute improvement in linking performance of up to 1.7 F 1 on AIDA/CoNLL'03 and up to 2.4 F 1 on the English TAC KBP 2015 TEDL dataset.* The majority of this work was done during an internship at Microsoft Research Asia. 1 We use entity to refer to both real-word entities and to their corresponding entries in the KB.

Selectional preferences have long been claimed to be essential for coreference resolution.However, they are mainly modeled only implicitly by current coreference resolvers.We propose a dependencybased embedding model of selectional preferences which allows fine-grained compatibility judgments with high coverage.We show that the incorporation of our model improves coreference resolution performance on the CoNLL dataset, matching the state-of-the-art results of a more complex system.However, it comes with a cost that makes it debatable how worthwhile such improvements are.

In this paper we investigate the performance of event argument identification. We show that the performance is tied to syntactic complexity. Based on this finding, we propose a novel and effective system for event argument identification. Recurrent Neural Networks learn to produce meaningful representations of long and short dependency paths. Convolutional Neural Networks learn to decompose the lexical context of argument candidates. They are combined into a simple system which outperforms a feature-based, state-of-the-art event argument identifier without any manual feature engineering.

This is a repository copy of Which coreference evaluation metric do you trust?A proposal for a link-based entity aware metric.

Event extraction is a difficult information extraction task. Li et al. (2014) explore the benefits of modeling event extraction and two related tasks, entity mention and relation extraction, jointly. This joint system achieves state-of-the-art performance in all tasks. However, as a system operating only at the sentence level, it misses valuable information from other parts of the document. In this paper, we present an incremental easy-first approach to make the global context of the entire document available to the intra-sentential, state-of-the-art event extractor. We show that our method robustly increases performance on two datasets, namely ACE 2005 and TAC 2015.

Previous work on automatic summarization does not thoroughly consider coherence while generating the summary.We introduce a graph-based approach to summarize scientific articles.We employ coherence patterns to ensure that the generated summaries are coherent.The novelty of our model is twofold: we mine coherence patterns in a corpus of abstracts, and we propose a method to combine coherence, importance and non-redundancy to generate the summary.We optimize these factors simultaneously using Mixed Integer Programming.Our approach significantly outperforms baseline and state-of-the-art systems in terms of coherence (summary coherence assessment) and relevance (ROUGE scores).

Coherence is established by semantic connections between sentences of a text which can be modeled by lexical relations.In this paper, we introduce the lexical coherence graph (LCG), a new graph-based model to represent lexical relations among sentences.The frequency of subgraphs (coherence patterns) of this graph captures the connectivity style of sentence nodes in this graph.The coherence of a text is encoded by a vector of these frequencies.We evaluate the LCG model on the readability ranking task.The results of the experiments show that the LCG model obtains higher accuracy than state-of-the-art coherence models.Using larger subgraphs yields higher accuracy, because they capture more structural information.However, larger subgraphs can be sparse.We adapt Kneser-Ney smoothing to smooth subgraphs' frequencies.Smoothing improves performance.

This is a repository copy of Search space pruning: a simple solution for better coreference resolvers.

We present an approach for extractive single-document summarization. Our ap-proach is based on a weighted graphical representation of documents obtained by topic modeling. We optimize importance, coherence and non-redundancy simulta-neously using ILP. We compare ROUGE scores of our system with state-of-the-art results on scientific articles from PLOS Medicine and on DUC 2002 data. Hu-man judges evaluate the coherence of sum-maries generated by our system in com-parision to two baselines. Our approach obtains competitive performance. 1

We present a toolkit for coreference resolution error analysis.It implements a recently proposed analysis framework and contains rich components for analyzing and visualizing recall and precision errors.1 Short for coreference resolution toolkit. 2 http://smartschat.de/software

Machine learning approaches to coreference resolution vary greatly in the modeling of the problem: while early approaches operated on the mention pair level, current research focuses on ranking architectures and antecedent trees. We propose a unified representation of different approaches to coreference resolution in terms of the structure they operate on. We represent several coreference resolution approaches proposed in the literature in our framework and evaluate their performance. Finally, we conduct a systematic analysis of the output of these approaches, highlighting differences and similarities.

This paper takes a discourse-oriented perspective for disambiguating common and proper noun mentions with respect to Wikipedia. Our novel approach models the relationship between disambiguation and aspects of cohesion using Markov Logic Networks with latent variables. Considering cohesive aspects consistently improves the disambiguation results on various commonly used data sets.

Bridging resolution plays an important role in establishing (local) entity coherence.This paper proposes a rule-based approach for the challenging task of unrestricted bridging resolution, where bridging anaphors are not limited to definite NPs and semantic relations between anaphors and their antecedents are not restricted to meronymic relations.The system consists of eight rules which target different relations based on linguistic insights.Our rule-based system significantly outperforms a reimplementation of a previous rule-based system (Vieira and Poesio, 2000).Furthermore, it performs better than a learning-based approach which has access to the same knowledge resources as the rule-based system.Additionally, incorporating the rules and more features into the learning-based system yields a minor improvement over the rule-based system.

We present a novel method for coreference resolution error analysis which we apply to perform a recall error analysis of four state-of-the-art English coreference resolution systems.Our analysis highlights differences between the systems and identifies that the majority of recall errors for nouns and names are shared by all systems.We characterize this set of common challenging errors in terms of a broad range of lexical and semantic properties.

Recognizing bridging anaphora is difficult due to the wide variation within the phenomenon, the resulting lack of easily identifiable surface markers and their relative rarity.We develop linguistically motivated discourse structure, lexico-semantic and genericity detection features and integrate these into a cascaded minority preference algorithm that models bridging recognition as a subtask of learning finegrained information status (IS).We substantially improve bridging recognition without impairing performance on other IS classes.

We present a taxonomy automatically generated from the system of categories in Wikipedia. Categories in the resource are identified as either classes or instances and included in a large subsumption, i.e. isa, hierarchy. The taxonomy is made available in RDFS format to the research community, e.g. for direct use within AI applications or to bootstrap the process of manual ontology creation.

Wikipedia provides a semantic network for computing semantic relatedness in a more structured fashion than a search engine and with more coverage than WordNet. We present experiments on using Wikipedia for computing semantic relatedness and compare it to WordNet on various benchmarking datasets. Existing relatedness measures perform better using Wikipedia than a baseline given by Google counts, and we show that Wikipedia outperforms WordNet on some datasets. We also address the question whether and how Wikipedia can be integrated into NLP applications as a knowledge base. Including Wikipedia improves the performance of a machine learning based coreference resolution system, indicating that it represents a valuable resource for NLP applications. Finally, we show that our method can be easily used for languages other than English by computing semantic relatedness for a German dataset.

We apply a decision tree based approach to pronoun resolution in spoken dialogue.Our system deals with pronouns with NPand non-NP-antecedents.We present a set of features designed for pronoun resolution in spoken dialogue and determine the most promising features.We evaluate the system on twenty Switchboard dialogues and show that it compares well to Byron's (2002) manually tuned system.

This paper describes the handling of multimodality in the Embassi system. Here, multimodality is treated in two modules. Firstly, a modality fusion component merges speech, video traced pointing gestures, and input from a graphical user interface. Secondly, a presentation planning component decides upon the modality to be used for the output, i.e., speech, an animated life-like character (ALC) and/or the graphical user interface, and ensures that the presentation is coherent and cohesive. We describe how these two components work and emphasize one particular feature of our system architecture: All modality analysis components generate output in a common semantic description format and all render components process input in a common output language. This makes it particularly easy to add or remove modality analyzers or renderer components, even dynamically while the system is running. This plug and play of modalities can be used to adjust the system's capabilities to different demands of users and their situative context. In this paper we give details about the implementations of the models, protocols and modules that are necessary to realize those features.

In this paper, we investigate the practical applicability of Co-Training for the task of building a classifier for reference resolution. We are concerned with the question if Co-Training can significantly reduce the amount of manual labeling work and still produce a classifier with an acceptable performance.

We report on experiments in reference resolution using a decision tree approach. We started with a standard feature set used in previous work, which led to moderate results. A closer examination of the performance of the features for different forms of anaphoric expressions showed good results for pronouns, moderate results for proper names, and poor results for definite noun phrases. We then included a cheap, language and domain independent feature based on the minimum edit distance between strings. This feature yielded a significant improvement for data sets consisting of definite noun phrases and proper names, respectively. When applied to the whole data set the feature produced a smaller but still significant improvement.

We specify an algorithm that builds up a hierarchy of referential discourse segments from local centering data. The spatial extension and nesting of these discourse segments constrain the reachability of potential antecedents of an anaphoric expression beyond the local level of adjacent center pairs. Thus, the centering model is scaled up to the level of the global referential structure of discourse. An empirical evaluation of the algorithm is supplied.

We extend the centering model for the resolution of intra-sentential anaphora and specify how to handle complex sentences. An empirical evaluation indicates that the functional information structure guides the search for an antecedent within the sentence.

Based on empirical evidence from a free word order language (German) we propose a fundamental revision of the principles guiding the ordering of discourse entities in the forward-looking centers within the centering model. We claim that grammatical role criteria should be replaced by indicators of the functional information structure of the utterances, i.e., the distinction between context-bound and unbound discourse elements. This claim is backed up by an empirical evaluation of functional centering.