A relatively under-explored area in research on neural natural language generation is the impact of the data representation on text quality.Here we report experiments on two leading input representations for data-to-text generation: attribute-value pairs and Resource Description Framework (RDF) triples.Evaluating the performance of encoder-decoder seq2seq models as well as recent large language models (LLMs) with both automated metrics and human evaluation, we find that the input representation does not seem to have a large impact on the performance of either purpose-built seq2seq models or LLMs.Finally, we present an error analysis of the texts generated by the LLMs and provide some insights into where these models fail.

Patricia Schmidtova, Saad Mahamood, Simone Balloccu, Ondrej Dusek, Albert Gatt, Dimitra Gkatzia, David M. Howcroft, Ondrej Platek, Adarsa Sivaprasad. Proceedings of the 17th International Natural Language Generation Conference. 2024.

Gàidhlig (Scottish Gaelic; gd) is spoken by about 57k people in Scotland, 1 but remains an under-resourced language with respect to natural language processing in general and natural language generation (NLG) in particular.To address this gap, we developed the first datasets for Scottish Gaelic NLG, collecting both conversational and summarisation data in a single setting.Our task setup involves dialogues between a pair of proficient speakers discussing museum exhibits, grounding the conversation in images and texts.Then, each interlocutor summarises the dialogue resulting in a secondary dialogue summarisation dataset.This paper presents the dialogue and summarisation corpora, as well as the software used for data collection.The dialogue dataset consists of 43 conversations (13.7k words) and 61 summaries (2.0k words). 2

Karin Sevegnani, David M. Howcroft, Ioannis Konstas, Verena Rieser. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Previous work has shown that human evaluations in NLP are notoriously under-powered.Here, we argue that there are two common factors which make this problem even worse: NLP studies usually (a) treat ordinal data as interval data and (b) operate under high variance settings while the differences they are hoping to detect are often subtle.We demonstrate through simulation that ordinal mixed effects models are better able to detect small differences between models, especially in high variance settings common in evaluations of generated texts.We release tools for researchers to conduct their own power analysis and test their assumptions.We also make recommendations for improving statistical power.

Current standards for designing and reporting human evaluations in NLP mean it is generally unclear which evaluations are comparable and can be expected to yield similar results when applied to the same system outputs.This has serious implications for reproducibility testing and meta-evaluation, in particular given that human evaluation is considered the gold standard against which the trustworthiness of automatic metrics is gauged.Using examples from NLG, we propose a classification system for evaluations based on disentangling (i) what is being evaluated (which aspect of quality), and (ii) how it is evaluated in specific (a) evaluation modes and (b) experimental designs.We show that this approach provides a basis for determining comparability, hence for comparison of evaluations across papers, meta-evaluation experiments, reproducibility testing.

David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, Verena Rieser. Proceedings of the 13th International Conference on Natural Language Generation. 2020.

Neural natural language generation (NNLG) systems are known for their pathological outputs, i.e. generating text which is unrelated to the input specification.In this paper, we show the impact of semantic noise on state-of-theart NNLG models which implement different semantic control mechanisms.We find that cleaned data can improve semantic correctness by up to 97%, while maintaining fluency.We also find that the most common error is omitting information, rather than hallucination.

Developing conventional natural language generation systems requires extensive attention from human experts in order to craft complex sets of sentence planning rules.We propose a Bayesian nonparametric approach to learn sentence planning rules by inducing synchronous tree substitution grammars for pairs of text plans and morphosyntactically-specified dependency trees.Our system is able to learn rules which can be used to generate novel texts after training on small datasets.

While previous research on readability has typically focused on document-level measures, recent work in areas such as natural language generation has pointed out the need of sentence-level readability measures.Much of psycholinguistics has focused for many years on processing measures that provide difficulty estimates on a word-by-word basis.However, these psycholinguistic measures have not yet been tested on sentence readability ranking tasks.In this paper, we use four psycholinguistic measures: idea density, surprisal, integration cost, and embedding depth to test whether these features are predictive of readability levels.We find that psycholinguistic features significantly improve performance by up to 3 percentage points over a standard document-level readability metric baseline.

Corpora of referring expressions elicited from human participants in a controlled environment are an important resource for research on automatic referring expression generation.We here present G-TUNA, a new corpus of referring expressions for German.Using images of furniture as stimuli similarly to the TUNA and D-TUNA corpora, our corpus extends on these corpora by providing data collected in a simulated driving dual-task setting, and additionally provides exact duration annotations for the spoken referring expressions.This corpus will hence allow researchers to analyze the interaction between referring expression length and speech rate, under conditions where the listener is under high vs. low cognitive load.

The search space in grammar-based natural language generation tasks can get very large, which is particularly problematic when generating long utterances or paragraphs. Using surface realization with OpenCCG as an example, we show that we can effectively detect partial solutions (edges) which cannot ultimately be part of a complete sentence because of their syntactic category. Formulating the completion of an edge into a sentence as finding a solution path in a large state-transition system, we demonstrate a connection to AI Planning which is concerned with this kind of problem. We design a compilation from OpenCCG into AI Planning allowing the detection of infeasible edges via AI Planning dead-end detection methods (proving the absence of a solution to the compilation). Our experiments show that this can filter out large fractions of infeasible edges in, and thus benefit the performance of, complex realization processes.