Natural Language Inference (NLI) remains an important benchmark task for LLMs.NLI datasets are a springboard for transfer learning to other semantic tasks, and NLI models are standard tools for identifying the faithfulness of model-generated text.There are several large scale NLI datasets today, and models have improved greatly by hill-climbing on these collections.Yet their realistic performance on out-of-distribution/domain data is less well-understood.We explore the opportunity for synthetic high-quality datasets to adapt NLI models for zero-shot use in downstream applications across new and unseen text domains.We demonstrate a new approach for generating NLI data in diverse domains and lengths, so far not covered by existing training sets.The resulting examples have meaningful premises, the hypotheses are formed in creative ways rather than simple edits to a few premise tokens, and the labels have high accuracy.We show that models trained on this data (685K synthetic examples) have the best generalization to completely new downstream test settings.On the TRUE benchmark, a T5small model trained with our data improves around 7% on average compared to training on the best alternative dataset.The improvements are more pronounced for smaller models, while still meaningful on a T5 XXL model.We also demonstrate gains on test sets when in-domain training data is augmented with our domain-general synthetic data. * Equal contribution.1 https://gluebenchmark.com/leaderboard

Previous work has demonstrated the effectiveness of planning for story generation exclusively in a monolingual setting focusing primarily on English. We consider whether planning brings advantages to automatic story generation across languages. We propose a new task of crosslingual story generation with planning and present a new dataset for this task. We conduct a comprehensive study of different plans and generate stories in several languages, by leveraging the creative and reasoning capabilities of large pretrained language models. Our results demonstrate that plans which structure stories into three acts lead to more coherent and interesting narratives, while allowing to explicitly control their content and structure.

Recent advances in language modeling have enabled new conversational systems.In particular, it is often desirable for people to make choices among specified options when using such systems.We address this problem of reference resolution, when people use natural expressions to choose between the entities.For example, given the choice 'Should we make a Simnel cake or a Pandan cake?' a natural response from a dialog participant may be indirect: 'let's make the green one'.Such natural expressions have been little studied for reference resolution.We argue that robustly understanding such language has large potential for improving naturalness in dialog, recommendation, and search systems.We create AltEntities 1 (Alternative Entities), a new public dataset of 42K entity pairs and expressions (referring to one entity in the pair), and develop models for the disambiguation problem.Consisting of indirect referring expressions across three domains, our corpus enables for the first time the study of how language models can be adapted to this task.We find they achieve 82%-87% accuracy in realistic settings, which while reasonable also invites further advances.

Jeremiah Milbauer, Annie Louis, Mohammad Javad Hosseini, Alex Fabrikant, Donald Metzler, Tal Schuster. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Abstract The ability to convey relevant and faithful information is critical for many tasks in conditional generation and yet remains elusive for neural seq-to-seq models whose outputs often reveal hallucinations and fail to correctly cover important details. In this work, we advocate planning as a useful intermediate representation for rendering conditional generation less opaque and more grounded. We propose a new conceptualization of text plans as a sequence of question-answer (QA) pairs and enhance existing datasets (e.g., for summarization) with a QA blueprint operating as a proxy for content selection (i.e., what to say) and planning (i.e., in what order). We obtain blueprints automatically by exploiting state-of-the-art question generation technology and convert input-output pairs into input-blueprint-output tuples. We develop Transformer-based models, each varying in how they incorporate the blueprint in the generated output (e.g., as a global plan or iteratively). Evaluation across metrics and datasets demonstrates that blueprint models are more factual than alternatives which do not resort to planning and allow tighter control of the generation output.

In a text, entities mentioned earlier can be referred to in later discourse by a more general description. For example, Celine Dion and Justin Bieber can be referred to by Canadian singers or celebrities. In this work, we study this phenomenon in the context of summarization, where entities from a source text are generalized in the summary. We call such instances source-summary entity aggregations. We categorize these aggregations into two types and analyze them in the Cnn/Dailymail corpus, showing that they are reasonably frequent. We then examine how well three state-of-the-art summarization systems can generate such aggregations within summaries. We also develop techniques to encourage them to generate more aggregations. Our results show that there is significant room for improvement in producing semantically correct aggregations.

Human-written texts contain frequent generalizations and semantic aggregation of content. In a document, they may refer to a pair of named entities such as ‘London’ and ‘Paris’ with different expressions: “the major cities”, “the capital cities” and “two European cities”. Yet generation, especially, abstractive summarization systems have so far focused heavily on paraphrasing and simplifying the source content, to the exclusion of such semantic abstraction capabilities. In this paper, we present a new dataset and task aimed at the semantic aggregation of entities. TESA contains a dataset of 5.3K crowd-sourced entity aggregations of Person, Organization, and Location named entities. The aggregations are document-appropriate, meaning that they are produced by annotators to match the situational context of a given news article from the New York Times. We then build baseline models for generating aggregations given a tuple of entities and document context. We finetune on TESA an encoder-decoder language model and compare it with simpler classification methods based on linguistically informed features. Our quantitative and qualitative evaluations show reasonable performance in making a choice from a given list of expressions, but free-form expressions are understandably harder to generate and evaluate.

We revisit a pragmatic inference problem in dialog: Understanding indirect responses to questions.Humans can interpret 'I'm starving.' in response to 'Hungry?', even without direct cue words such as 'yes' and 'no'.In dialog systems, allowing natural responses rather than closed vocabularies would be similarly beneficial.However, today's systems are only as sensitive to these pragmatic moves as their language model allows.We create and release 1 the first large-scale English language corpus 'Circa' with 34,268 (polar question, indirect answer) pairs to enable progress on this task.The data was collected via elaborate crowdsourcing, and contains utterances with yes/no meaning, as well as uncertain, middle-ground, and conditional responses.We also present BERT-based neural models to predict such categories for a question-answer pair.We find that while transfer learning from entailment works reasonably, performance is not yet sufficient for robust dialog.Our models reach 82-88% accuracy for a 4-class distinction, and 74-85% for 6 classes.* * Work done at Google "Want to get some dinner together?""I know a restaurant we could get a reservation at." "I have already eaten recently.""I hope to make it home by supper but I'm not sure I can." "Dinner would be lovely.""I'd rather just go to bed." "There's a few new restaurants we could go to.""I would like that.""We could do dinner this weekend.""I would like to go somewhere casual.""I'd like to try the new Italian place."

Matt Grenander, Yue Dong, Jackie Chi Kit Cheung, Annie Louis. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

When a reader is first introduced to an entity, its referring expression must describe the entity.For entities that are widely known, a single word or phrase often suffices.This paper presents the first study of how expressions that refer to the same entity develop over time.We track thousands of person and organization entities over 20 years of New York Times (NYT).As entities move from hearernew (first introduction to the NYT audience) to hearer-old (common knowledge) status, we show empirically that the referring expressions along this trajectory depend on the type of the entity, and exhibit linguistic properties related to becoming common knowledge (e.g., shorter length, less use of appositives, more definiteness).These properties can also be used to build a model to predict how long it will take for an entity to reach hearer-old status.Our results reach 10-30% absolute improvement over a majority-class baseline.

Today, social media refers to a wide range of Web sites and Internet-based services that allow users to create content and interact with other users. Some of these tools, such as multi-party chats, discussion forums, blogs, and online reviews, have been a focus of natural language processing (NLP) research for quite some time now. But within the last decade, NLP work has expanded rapidly to cover an immense variety of new social media content—microblogs such as Twitter, social networks such as Facebook, comments on news articles, captions on user-contributed images such as on Flickr, and forums dedicated to specialized topics and needs (e.g., health and online education). Simultaneously, many other research communities are carrying out work using social media data—information science, information retrieval, network science, social media analytics, social science, psychology, and corpus linguistics. Today, a large number of businesses are also centered on, or benefit from, analytics performed on social media. Given these myriad research and commercial interests in the social media domain, we are at a time where we should seek to clearly understand what role NLP has in the field of social media analysis, both in terms of the key and interesting language questions, as well as contributions NLP can make to the research carried out in other fields. In this context, this short book by Farzindar and Inkpen is timely and exciting, filling an obvious gap.This book is targeted towards researchers who have a background in natural language processing and machine learning, and want to learn about research questions, tasks, and techniques related to processing of social media texts. Chapter 1 introduces social media, and highlights its large scale and continuous growth. This chapter also describes the challenges associated with processing data from social media (large volume, non-standard language, short length of the texts, and a need to separate useful information from extraneous content), and overviews applications that benefit from social media analysis. This chapter is brief; computer science researchers will already have a general understanding of these issues, and can use the chapter to peruse some of the details. The rest of the chapters can be logically divided into two parts containing two chapters each, and the book ends with a conclusion chapter and a short case study.Chapters 2 and 3 form the larger part of the book's content, and are also the core sections where NLP methods are discussed. Chapter 2 focuses on how preprocessing of the text can be done. This problem includes tasks such as tokenizing the text into words, normalizing non-dictionary words, part-of-speech (POS) tagging, and syntactic parsing. The need for each task is discussed, together with an overview of work that has developed specialized methods for social media–type texts. Also included are useful discussions about how each of these modules is typically evaluated. The methods are discussed rather briefly; they are pointers to work rather than descriptions of the models or techniques. An exception is a detailed example given for dialect identification, where the model, data, experiments, and comprehensive results are presented. Similar detailed work is presented at a few other chosen points in the book. Newcomers to NLP technology will appreciate a listing of NLP tools in this chapter, and those new to social media processing will find pointers to tools developed specifically for social media texts. Chapter 3 focuses on information extraction problems such as geolocation identification, opinion mining, event extraction, and NLP applications such as machine translation and automatic summarization. Although this chapter is called “semantic analysis” and the term “semantics” is used at several points in the book, the authors acknowledge that the term refers to any intelligent processing of the text, rather than an attempt to create a representation of the text's meaning. Again, detailed examples are presented for geolocation, opinion classification, and machine translation. The usefulness of these examples is mixed, and will be discussed later in this review. Otherwise, the methods and techniques are motivated and described briefly, similar to the previous chapter.The second part of the book, Chapters 4 and 5, discusses applications that benefit from processing of social media texts, and how data can be obtained for research. A wide spectrum of applications are described including financial, political, and defense interests. Chapter 5 summarizes existing data sets and evaluation methods together with discussions about how useful data can be gleaned from large volumes, and how privacy can be maintained during social media research. The conclusion chapter reiterates the key challenges of social media processing, and brings up questions that are yet to be addressed by the NLP community. The book ends with a brief case study on a social media–based event-monitoring application that combines many of the tasks and technologies discussed earlier in the book.The authors should be commended for taking up a subject in which NLP has a huge impact. At the same time, as a nascent area, this topic is worthy of discussion within our community to identify the main challenges, the techniques that are consistently useful, as well as the scope for further work. In this regard, this book provides a valuable introduction to the field. For a short book, it covers a number of dimensions to social media language analysis—preprocessing, information extraction, as well as applications. Newcomers to the social media domain will find the book a good reference point to research done in the area. The highlighting of challenges in this domain and differences from NLP work typically carried out on news and other single-author and edited text are very useful. Another handy element is a continuous focus on explaining how methods for various NLP problems are evaluated.At the same time, although a broad set of topics is discussed, some key areas of social media processing are ignored. One is the use of social media as a tool for language understanding. On one side, researchers are excited about using social media as a large-scale source of conversation data on which questions about language can be asked. Recent work in this area has created new insights into language variation (Doyle 2014; Eisenstein 2015b), phonology (Eisenstein 2013, 2015a), and entrainment (Danescu-Niculescu-Mizil, Gamon, and Dumais 2011; Doyle, Yurovsky, and Frank 2016), to name a few. Another line of work is richer conversation understanding through language processing beyond lexical and syntactic levels. Work on dialog act tagging (Kim, Cavedon, and Baldwin 2010; Ritter, Cherry, and Dolan 2010), topic structure (Ramage, Dumais, and Liebling 2010; Peng, Wang, and Dredze 2014), as well as phrasing (Niculae and Danescu-Niculescu-Mizil 2014; Tan, Lee, and Pang 2014) on social media conversations fall under this category and yet are not covered in this book. Even on the word- and sentence-level, the book does not cover research carried out in the related web, social media, and social network communities. Many of the models in these areas carry out similar word-level processing but ask different sets of research questions. There are some references to this line of work but the majority of the content remains unexplored. Finally, there is heavy focus on Twitter or microblogs in comparison with other forms of social media. Some sections such as summarization or machine translation entirely focus on Twitter. Although it may be possible that there are logical divisions of social media text types and that tweets represent a class of short texts, such reflections and explanations are not provided in the book.Going beyond coverage, a more fundamental question that newcomers as well as current NLP researchers want to ask is what is the role of natural language processing for social media. It is clear that the tasks involved in preprocessing content (normalization, POS tagging, and parsing) are clearly in the realm of natural language processing, and so are tasks such as named entity recognition, opinion mining, and event extraction. However, the book does not do a good job of distinguishing what new techniques are required due to distinguishing properties of the social media domain. Some of the tasks appear to be solvable by adding annotated data from this domain to existing NLP corpora, and retraining the models. Other problems appear to be solved by supervised techniques with word-level features, sometimes combined with social media and social network metadata. It is not even clear from the book whether syntactic parsing, and so forth, are useful in current NLP work on social media. A reflection on the types of techniques that work well so far, and insights into new types of models proposed for social media, and an emphasis on methods addressing special properties, for example, streaming algorithms, would have been worthwhile additions. Clarifying these questions would have been an important desired property of a book on NLP and social media, and this is the main place where the book falls short.The book is also not self-sufficient for understanding the details of social media processing. There is no clear discussion of particular models, especially the ones specific to social media. For example, normalization of non-dictionary words is a task clearly new and motivated by social media texts, but only a very brief section is given to its discussion and devoid of any models or techniques. Often multiple paragraph descriptions of a study are given but without enough details, equations, or technical explanations to actually understand the approach. As a result, not much assimilation of the topic is possible at the end of many sections of the book. Rather, a reader may only use these descriptions as a pointer for finding work to read further and understand. Another undesirable property of this book is a heavy focus on research carried out by the authors of the book themselves. The detailed examples in the book where comprehensive explanations of data, models, and results are given are all the authors' own work with their collaborators. At least from the book, it is not clear why these models should be described in detail compared with other tasks and the work of other researchers. The case study is also on a company associated with one of the authors. For a book focusing on such a wide topic, a biased focus could have been avoided.Still, overall, this book will be useful as a reference point for work currently done, and a starting point for further discussions on social media and NLP work.

Online forum discussions proceed differently from face-to-face conversations and any single thread on an online forum contains posts on different subtopics.This work aims to characterize the content of a forum thread as a conversation tree of topics.We present models that jointly perform two tasks: segment a thread into subparts, and assign a topic to each part.Our core idea is a definition of topic structure using probabilistic grammars.By leveraging the flexibility of two grammar formalisms, Context-Free Grammars and Linear Context-Free Rewriting Systems, our models create desirable structures for forum threads: our topic segmentation is hierarchical, links non-adjacent segments on the same topic, and jointly labels the topic during segmentation.We show that our models outperform a number of tree generation baselines.

Online discussion forums and community question-answering websites provide one of the primary avenues for online users to share information. In this paper, we propose text mining techniques which aid users navigate troubleshooting-oriented data such as questions asked on forums and their suggested solutions. We introduce Bayesian generative models of the troubleshooting data and apply them to two interrelated tasks: (a) predicting the complexity of the solutions (e.g., plugging a keyboard in the computer is easier compared to installing a special driver) and (b) presenting them in a ranked order from least to most complex. Experimental results show that our models are on par with human performance on these tasks, while outperforming baselines based on solution length or readability.

Length constraints impose implicit requirements on the type of content that can be included in a text.Here we propose the first model to computationally assess if a text deviates from these requirements.Specifically, our model predicts the appropriate length for texts based on content types present in a snippet of constant length.We consider a range of features to approximate content type, including syntactic phrasing, constituent compression probability, presence of named entities, sentence specificity and intersentence continuity.Weights for these features are learned using a corpus of summaries written by experts and on high quality journalistic writing.During test time, the difference between actual and predicted length allows us to quantify text verbosity.We use data from manual evaluation of summarization systems to assess the verbosity scores produced by our model.We show that the automatic verbosity scores are significantly negatively correlated with manual content quality scores given to the summaries.

We present a French to English transla-tion system for Wikipedia biography ar-ticles. We use training data from out-of-domain corpora and adapt the system for biographies. We propose two forms of domain adaptation. The first biases the system towards words likely in biogra-phies and encourages repetition of words across the document. Since biographies in Wikipedia follow a regular structure, our second model exploits this structure as a sequence of topic segments, where each segment discusses a narrower subtopic of the biography domain. In this structured model, the system is encouraged to use words likely in the current segment’s topic rather than in biographies as a whole. We implement both systems using cache-based translation techniques. We show that a system trained on Europarl and news can be adapted for biographies with 0.5 BLEU score improvement using our mod-els. Further the structure-aware model out-performs the system which treats the entire document as a single segment. 1

The most widely adopted approaches for evaluation of summary content follow some protocol for comparing a summary with gold-standard human summaries, which are traditionally called model summaries. This evaluation paradigm falls short when human summaries are not available and becomes less accurate when only a single model is available. We propose three novel evaluation techniques. Two of them are model-free and do not rely on a gold standard for the assessment. The third technique improves standard automatic evaluations by expanding the set of available model summaries with chosen system summaries. We show that quantifying the similarity between the source text and its summary with appropriately chosen measures produces summary scores which replicate human assessments accurately. We also explore ways of increasing evaluation quality when only one human model summary is available as a gold standard. We introduce pseudomodels, which are system summaries deemed to contain good content according to automatic evaluation. Combining the pseudomodels with the single human model to form the gold-standard leads to higher correlations with human judgments compared to using only the one available model. Finally, we explore the feasibility of another measure—similarity between a system summary and the pool of all other system summaries for the same input. This method of comparison with the consensus of systems produces impressively accurate rankings of system summaries, achieving correlation with human rankings above 0.9.

Great writing is rare and highly admired. Readers seek out articles that are beautifully written, informative and entertaining. Yet information-access technologies lack capabilities for predicting article quality at this level. In this paper we present first experiments on article quality prediction in the science journalism domain. We introduce a corpus of great pieces of science journalism, along with typical articles from the genre. We implement features to capture aspects of great writing, including surprising, visual and emotional content, as well as general features related to discourse organization and sentence structure. We show that the distinction between great and typical articles can be detected fairly accurately, and that the entire spectrum of our features contribute to the distinction.