Data-to-text natural language generation (NLG) models may produce outputs that closely mirror the structure of their input. We introduce formulaicness as a measure of the output-to-input structural resemblance, proposing it as an enhancement for reference-less naturalness evaluation. Focusing on logic-to-text generation, we construct a dataset and train a regressor to predict formulaicness scores. We collect human judgments on naturalness and examine how incorporating formulaicness into existing metrics affects alignment with these judgments.

Hallucinations pose a persistent challenge in open-ended question answering (QA). Traditional annotation methods, such as span-labelling, suffer from inconsistency and limited coverage. In this paper, we propose a rewriting-based framework as a new perspective on hallucinations in open-ended QA. We report on an experiment in which annotators are instructed to rewrite LLM-generated answers directly to ensure factual accuracy, with edits automatically recorded. Using the Chinese portion of the Mu-SHROOM dataset, we conduct a controlled rewriting experiment, comparing fact-checking tools (Google vs. GPT-4o), and analysing how tool choice, annotator background, and question openness influence rewriting behaviour. We find that rewriting leads to more hallucinations being identified, with higher inter-annotator agreement, than span-labelling.

Recently, a human evaluation study of Referring Expression Generation (REG) models had an unexpected conclusion: on WEBNLG, Referring Expressions (REs) generated by the state-of-the-art neural models were not only indistinguishable from the REs in WEBNLG but also from the REs generated by a simple rulebased system.Here, we argue that this limitation could stem from the use of a purely ratings-based human evaluation (which is a common practice in Natural Language Generation).To investigate these issues, we propose an intrinsic task-based evaluation for REG models, in which, in addition to rating the quality of REs, participants were asked to accomplish two meta-level tasks.One of these tasks concerns the referential success of each RE; the other task asks participants to suggest a better alternative for each RE.The outcomes suggest that, in comparison to previous evaluations, the new evaluation protocol assesses the performance of each REG model more comprehensively and makes the participants' ratings more reliable and discriminable.

Theoretical linguists have suggested that some languages (e.g., Chinese and Japanese) are “cooler” than other languages based on the observation that the intended meaning of phrases in these languages depends more on their contexts. As a result, many expressions in these languages are shortened, and their meaning is inferred from the context. In this paper, we focus on the omission of the plurality and definiteness markers in Chinese noun phrases (NPs) to investigate the predictability of their intended meaning given the contexts. To this end, we built a corpus of Chinese NPs, each of which is accompanied by its corresponding context, and by labels indicating its singularity/plurality and definiteness/indefiniteness. We carried out corpus assessments and analyses. The results suggest that Chinese speakers indeed drop plurality and definiteness markers very frequently. Building on the corpus, we train a bank of computational models using both classic machine learning models and state-of-the-art pre-trained language models to predict the plurality and definiteness of each NP. We report on the performance of these models and analyse their behaviours.

Abstract Despite impressive advances in Natural Language Generation (NLG) and Large Language Models (LLMs), researchers are still unclear about important aspects of NLG evaluation. To substantiate this claim, I examine current classifications of hallucination and omission in data-text NLG, and I propose a logic-based synthesis of these classfications. I conclude by highlighting some remaining limitations of all current thinking about hallucination and by discussing implications for LLMs.

Current captioning datasets focus on object-centric captions, describing the visible objects in the image, often ending up stating the obvious (for humans), e.g. "people eating food in a park". Although these datasets are useful to evaluate the ability of Vision & Language models to recognize and describe visual content, they do not support controlled experiments involving model testing or fine-tuning, with more high-level captions, which humans find easy and natural to produce. For example, people often describe images based on the type of scene they depict ("people at a holiday resort") and the actions they perform ("people having a picnic"). Such concepts are based on personal experience and contribute to forming common sense assumptions. We present the High-Level Dataset, a dataset extending 14997 images from the COCO dataset, aligned with a new set of 134,973 human-annotated (high-level) captions collected along three axes: scenes, actions and rationales. We further extend this dataset with confidence scores collected from an independent set of readers, as well as a set of narrative captions generated synthetically, by combining each of the three axes. We describe this dataset and analyse it extensively. We also present baseline results for the High-Level Captioning task.

In recent years, many NLP studies have focused solely on performance improvement.In this work, we focus on the linguistic and scientific aspects of NLP.We use the task of generating referring expressions in context (REG-incontext) as a case study and start our analysis from GREC, a comprehensive set of shared tasks in English that addressed this topic over a decade ago.We ask what the performance of models would be if we assessed them (1) on more realistic datasets, and (2) using more advanced methods.We test the models using different evaluation metrics and feature selection experiments.We conclude that GREC can no longer be regarded as offering a reliable assessment of models' ability to mimic human reference production, because the results are highly impacted by the choice of corpus and evaluation metrics.Our results also suggest that pre-trained language models are less dependent on the choice of corpus than classic Machine Learning models, and therefore make more robust class predictions.

Abstract Performance on a dataset is often regarded as the key criterion for assessing NLP models. I argue for a broader perspective, which emphasizes scientific explanation. I draw on a long tradition in the philosophy of science, and on the Bayesian approach to assessing scientific theories, to argue for a plurality of criteria for assessing NLP models. To illustrate these ideas, I compare some recent models of language production with each other. I conclude by asking what it would mean for institutional policies if the NLP community took these ideas onboard.

In recent years, a range of Neural Referring Expression Generation (REG) systems have been built and they have often achieved encouraging results. However, these models are often thought to lack transparency and generality. Firstly, it is hard to understand what these neural REG models can learn and to compare their performance with existing linguistic theories. Secondly, it is unclear whether they can generalise to data in different text genres and different languages. To answer these questions, we propose to focus on a sub-task of REG: Referential Form Selection (RFS). We introduce the task of RFS and a series of neural RFS models built on state-of-the-art neural REG models. To address the issue of interpretability, we probe these RFS models using probing classifiers that consider information known to impact the human choice of Referential Forms. To address the issue of generalisability, we assess the performance of RFS models on multiple datasets in multiple genres and two different languages, namely, English and Chinese.

A prominent strand of work in formal semantics investigates the ways in which human languages quantify the elements of a set, as when we say All A are B, Few A are B, and so on. Building on a growing body of empirical studies that shed light on the meaning and the use of quantifiers, we extend this line of work by computationally modelling how human speakers textually describe complex scenes in which quantitative relations play an important role. To this end, we conduct a series of elicitation experiments in which human speakers were asked to perform a linguistic task that invites the use of quantified expressions. The experiments result in a corpus, called QTUNA, made up of short texts that contain a large variety of quantified expressions. We analyse QTUNA, summarise our findings, and explain how we design computational models of human quantifier use accordingly. Finally, we evaluate these models in accordance with QTUNA.

In recent years, neural models have often outperformed rule-based and classic Machine Learning approaches in NLG.These classic approaches are now often disregarded, for example when new neural models are evaluated.We argue that they should not be overlooked, since for some tasks, well-designed non-neural approaches achieve better performance than neural ones.In this paper, the task of generating referring expressions in linguistic context is used as an example.We examined two very different English datasets (WEBNLG and WSJ), and evaluated each algorithm using both automatic and human evaluations.Overall, the results of these evaluations suggest that rule-based systems with simple rule sets achieve on-par or better performance on both datasets compared to state-of-the-art neural REG systems.In the case of the more realistic dataset, WSJ, a machine learning-based system with well-designed linguistic features performed best.We hope that our work can encourage researchers to consider non-neural models in future.* Equal contribution.Order determined by swapping the order in Chen et al. (2021).

Despite achieving encouraging results, neural Referring Expression Generation models are often thought to lack transparency.We probed neural Referential Form Selection (RFS) models to find out to what extent the linguistic features influencing the RE form are learnt and captured by state-of-the-art RFS models.The results of 8 probing tasks show that all the defined features were learnt to some extent.The probing tasks pertaining to referential status and syntactic position exhibited the highest performance.The lowest performance was achieved by the probing models designed to predict discourse structure properties beyond the sentence level.

Choosing the most suitable classifier in a linguistic context is a well-known problem in the production of Mandarin and many other languages.The present paper proposes a solution based on BERT, compares this solution to previous neural and rule-based models, and argues that the BERT model performs particularly well on those difficult cases where the classifier adds information to the text.

This paper reports on a structured evaluation of feature-based Machine Learning algorithms for selecting the form of a referring expression in discourse context.Based on this evaluation, we selected seven feature sets from the literature, amounting to 65 distinct linguistic features.The features were then grouped into 9 broad classes.After building Random Forest models, we used Feature Importance Ranking and Sequential Forward Search methods to assess the "importance" of the features.Combining the results of the two methods, we propose a consensus feature set.The 6 features in our consensus set come from 4 different classes, namely grammatical role, inherent features of the referent, antecedent form and recency.

Referring expression generation (REG) algorithms offer computational models of the production of referring expressions.In earlier work, a corpus of referring expressions (REs) in Mandarin was introduced.In the present paper, we annotate this corpus, evaluate classic REG algorithms on it, and compare the results with earlier results on the evaluation of REG for English referring expressions.Next, we offer an in-depth analysis of the corpus, focusing on issues that arise from the grammar of Mandarin.We discuss shortcomings of previous REG evaluations that came to light during our investigation and we highlight some surprising results.Perhaps most strikingly, we found a much higher proportion of under-specified expressions than previous studies had suggested, not just in Mandarin but in English as well.

Earlier research has shown that evaluation metrics based on textual similarity (e.g., BLEU, CIDEr, Meteor) do not correlate well with human evaluation scores for automatically generated text.We carried out an experiment with Chinese speakers, where we systematically manipulated image descriptions to contain different kinds of errors.Because our manipulated descriptions form minimal pairs with the reference descriptions, we are able to assess the impact of different kinds of errors on the perceived quality of the descriptions.Our results show that different kinds of errors elicit significantly different evaluation scores, even though all erroneous descriptions differ in only one character from the reference descriptions.Evaluation metrics based solely on textual similarity are unable to capture these differences, which (at least partially) explains their poor correlation with human judgments.Our work provides the foundations for future work, where we aim to understand why different errors are seen as more or less severe.

Quantified expressions have always taken up a central position in formal theories of meaning and language use.Yet quantified expressions have so far attracted far less attention from the Natural Language Generation community than, for example, referring expressions.In an attempt to start redressing the balance, we investigate a recently developed corpus in which quantified expressions play a crucial role; the corpus is the result of a carefully controlled elicitation experiment, in which human participants were asked to describe visually presented scenes.Informed by an analysis of this corpus, we propose algorithms that produce computer-generated descriptions of a wider class of visual scenes, and we evaluate the descriptions generated by these algorithms in terms of their correctness, completeness, and human-likeness.We discuss what this exercise can teach us about the nature of quantification and about the challenges posed by the generation of quantified expressions.

A prominent strand of work in formal semantics investigates the ways in which human languages quantify over the elements of a set, as when we say "All A are B", "All except two A are B", "Only a few of the A are B" and so on.Our aim is to build Natural Language Generation algorithms that mimic humans' use of quantified expressions.To inform these algorithms, we conducted on a series of elicitation experiments in which human speakers were asked to perform a linguistic task that invites the use of quantified expressions.We discuss how these experiments were conducted and what corpora they gave rise to.We conduct an informal analysis of the corpora, and offer an initial assessment of the challenges that these corpora pose for Natural Language Generation.The dataset is available at: https: //github.com/a-quei/qtuna.

Between 80% and 90% of all Chinese words have long and short form such as 老虎/虎 (laohu/hu , tiger) (Duanmu, 2013).Consequently, the choice between long and short forms is a key problem for lexical choice across NLP and NLG in Chinese.Following on from earlier work on abbreviations in English (Mahowald et al., 2013), we bring a probabilistic perspective to word length choice, using both a behavioural and a corpus-based approach.Thus, we hypothesise that, in Chinese, short forms are likelier in supportive than in neutral contexts.Our corpus and behavioral study supported this hypothesis, but a closer analysis revealed striking differences between different types of Chinese words.

We introduce SimpleNLG-ZH, a realisation engine for Mandarin that follows the software design paradigm of SimpleNLG (Gatt and Reiter, 2009).We explain the core grammar (morphology and syntax) and the lexicon of SimpleNLG-ZH, which is very different from English and other languages for which SimpleNLG engines have been built.The system was evaluated by regenerating expressions from a body of test sentences and a corpus of humanauthored expressions.Human evaluation was conducted to estimate the quality of regenerated sentences.

We extend the classic Referring Expressions Generation task by considering zero pronouns in "pro-drop" languages such as Chinese, modelling their use by means of the Bayesian Rational Speech Acts model (Frank and Goodman, 2012).By assuming that highly salient referents are most likely to be referred to by zero pronouns (i.e., pro-drop is more likely for salient referents than the less salient ones), the model offers an attractive explanation of a phenomenon not previously addressed probabilistically.

We explored the task of creating a textual summary describing a large set of objects characterised by a small number of features using an e-commerce dataset.When a set of consumer products is large and varied, it can be difficult for a consumer to understand how the products in the set differ; consequently, it can be challenging to choose the most suitable product from the set.To assist consumers, we generated high-level summaries of product sets.Two generation algorithms are presented, discussed, and evaluated with human users.Our evaluation results suggest a positive contribution to consumers' understanding of the domain.

This paper argues that a new generic approach to statistical NLG can be made to perform Referring Expression Generation (REG) successfully.The model does not only select attributes and values for referring to a target referent, but also performs Linguistic Realisation, generating an actual Noun Phrase.Our evaluations suggest that the attribute selection aspect of the algorithm exceeds classic REG algorithms, while the Noun Phrases generated are as similar to those in a previously developed corpus as were Noun Phrases produced by a new set of human speakers.W 10-12 ⇒ [direction] [speed] WS 22-24 ⇒ [direction] [speed]

We present a data resource which can be useful for research purposes on language grounding tasks in the context of geographical referring expression generation.The resource is composed of two data sets that encompass 25 different geographical descriptors and a set of associated graphical representations, drawn as polygons on a map by two groups of human subjects: teenage students and expert meteorologists.

This paper presents an empirical research. It focuses on testing empirically the benefits of providing users, in a specific domain, with textual interpretation of the fuzzy inferences carried out by a fuzzy classifier for a given selection of samples. The hypothesis to test is as follows: “Users understand easier the decision made by a fuzzy system when they are provided with a textual interpretation of the fuzzy inference mechanism that the system carried out”. This hypothesis was successfully tested in a web survey. The application domain was leaf classification. The fuzzy classifiers were built with the GUAJE fuzzy modeling open source software which is aimed at generating interpretable fuzzy systems. The textual interpretation was handmade by an expert who followed the guidelines of the Natural Language Generation approach proposed by Reiter and Dale. Reported results encourage us to go on with a series of additional experiments devoted to deeply explore how Natural Language Generation techniques can contribute to facilitate the understanding of fuzzy systems.

We present a novel heuristic approach that defines fuzzy geographical descriptors using data gathered from a survey with human subjects. The participants were asked to provide graphical interpretations of the descriptors `north' and `south' for the Galician region (Spain). Based on these interpretations, our approach builds fuzzy descriptors that are able to compute membership degrees for geographical locations. We evaluated our approach in terms of efficiency and precision. The fuzzy descriptors are meant to be used as the cornerstones of a geographical referring expression generation algorithm that is able to linguistically characterize geographical locations and regions. This work is also part of a general research effort that intends to establish a methodology which reunites the empirical studies traditionally practiced in data-to-text and the use of fuzzy sets to model imprecision and vagueness in words and expressions for text generation purposes.

East Asian languages are thought to handle reference differently from English, particularly in terms of the marking of definiteness and number.We present the first Data-Text corpus for Referring Expressions in Mandarin, and we use this corpus to test some initial hypotheses inspired by the theoretical linguistics literature.Our findings suggest that function words deserve more attention in Referring Expression Generation than they have so far received, and they have a bearing on the debate about whether different languages make different trade-offs between clarity and brevity.

We explore how the problem of uncertainty and imprecision in natural language generation (NLG) could be addressed through the use of fuzzy sets. We propose bringing together standard empirical procedures for knowledge acquisition in NLG and computing with words/perceptions related techniques (with a special focus on linguistic description of data) to address an open challenge in NLG: the generation of geographical referring expressions. Following this methodology, we present an exploratory experiment which provides some insights about how human subjects refer to geographical expressions and discuss how the obtained results might relate to the use of fuzzy sets.

Standard algorithms for attribute choice in the generation of referring expressions have little to say about the role of Proper Names in referring expressions.We discuss the implications of letting these algorithms produce Proper Names and expressions that have Proper Names as parts.

We discuss a fully statistical approach to the expression of quantitative information in English.We outline the approach, focussing on the problem of Lexical Choice.An initial evaluation experiment suggests that it is worth investigating the method further.

Presenting entailments of axioms in a formal ontology is a non-trivial task. This position paper argues that the problem of selecting entailments for presentation to users is not adequately acknowledged or addressed in the literature or in implemented systems. We analyse the problem and consider some di↵erent approaches that can help to address the problem.

This article offers a survey of computational research on referring expression generation (REG). It introduces the REG problem and describes early work in this area, discussing what basic assumptions lie behind it, and showing how its remit has widened in recent years. We discuss computational frameworks underlying REG, and demonstrate a recent trend that seeks to link REG algorithms with well-established Knowledge Representation techniques. Considerable attention is given to recent efforts at evaluating REG algorithms and the lessons that they allow us to learn. The article concludes with a discussion of the way forward in REG, focusing on references in larger and more realistic settings.

It is often desirable that referring expressions be chosen in such a way that their referents are easy to identify. This article focuses on referring expressions in hierarchically structured domains, exploring the hypothesis that referring expressions can be improved by including logically redundant information in them if this leads to a significant reduction in the amount of search that is needed to identify the referent. Generation algorithms are presented that implement this idea by including logically redundant information into the generated expression, in certain well-circumscribed situations. To test our hypotheses, and to assess the performance of our algorithms, two controlled experiments with human subjects were conducted. The first experiment confirms that human judges have a preference for logically redundant expressions in the cases where our model predicts this to be the case. The second experiment suggests that readers benefit from the kind of logical redundancy that our algorithms produce, as measured in terms of the effort needed to identify the referent of the expression.

This article examines the role of gradable properties in referring expressions from the perspective of natural language generation. First, we propose a simple semantic analysis of vague descriptions (i.e., referring expressions that contain gradable adjectives) that reflects the context-dependent meaning of the adjectives in them. Second, we show how this type of analysis can inform algorithms for the generation of vague descriptions from numerical data. Third, we ask when such descriptions should be used. The article concludes with a discussion of salience and pointing, which are analyzed as if they were gradable adjectives.

This article challenges the received wisdom that template-based approaches to the generation of language are necessarily inferior to other approaches as regards their maintainability, linguistic well-foundedness, and quality of output. Some recent NLG systems that call themselves “template-based” will illustrate our claims.

This paper starts by introducing a class of future document authoring systems that will allow authors to specify the content and form of a text+pictures document at a high level of abstraction, while leaving responsibility for linguistic and graphical details to the system. Next, we describe two working prototypes that implement parts of this functionality, based on semantic modeling of the pictures and the text of the document; one of these two, the ILLUSTRATE prototype, is a multimedia extension of previous text authoring systems in the What You See Is What You Meant (WYSIWYM) tradition. The paper concludes with an exploration of the ways in which Multimedia WYSIWYM can be further enhanced, allowing it to approximate the ‘ideal’ systems that were sketched earlier in the paper. Applications of Multimedia WYSIWYM to general-purpose picture retrieval (in the context of the Semantic Web, for example) are also discussed.

This paper brings a logical perspective to the generation of referring expressions, addressing the incompleteness of existing algorithms in this area. After studying references to individual objects, we discuss references to sets, including Boolean descriptions that make use of negated and disjoined properties. To guarantee that a distinguishing description is generated whenever such descriptions exist, the paper proposes generalizations and extensions of the Incremental Algorithm of Dale and Reiter (1995).

The RAGS proposals for generic specification of NLG systems includes a detailed account of data representation, but only an outline view of processing aspects. In this paper we introduce a modular processing architecture with a concrete implementation which aims to meet the RAGS goals of transparency and reusability. We illustrate the model with the RICHES system -- a generation system built from simple linguistically-motivated modules.

This paper deals with the generation of definite (i.e., uniquely referring) descriptions containing semantically vague expressions ('large', 'small', etc.). Firstly, the paper proposes a semantic analysis of vague descriptions that does justice to the context-dependent meaning of the vague expressions in them. Secondly, the paper shows how this semantic analysis can be implemented using a modification of the Dale and Reiter (1995) algorithm for the generation of referring expressions. A notable feature of the new algorithm is that, unlike Dale and Reiter (1995), it covers plural as well as singular NPs. This algorithm has been implemented in an experimental NLG program using ProFIT. The paper concludes by formulating some pragmatic constraints that could allow a generator to choose between different semantically correct descriptions.

In this paper, it is argued that “coreference” annotations, as performed in the MUC community for example, go well beyond annotation of the relation of coreference proper. As a result, it is not always clear what semantic relation these annotations are encoding. The paper discusses a number of problems with these annotations and concludes that rethinking of the coreference task is needed before the task is expanded. In particular, it suggests a division of labor whereby annotation of the coreference relation proper is separated from other tasks such as annotation of bound anaphora and of the relation between a subject and a predicative NP.