Parker Riley, Daniel Deutsch, Mara Finkelstein, Colten DiIanni, Juraj Juraska, Markus Freitag. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

As LLMs continue to become more powerful and versatile, human evaluation has become intractable at scale and reliance on automatic metrics has become the norm. Recently, it has been shown that LLMs are themselves state-of-the-art evaluators for many tasks. These *Autoraters* are typically designed so that they generalize to new systems *and* test sets. In practice, however, evaluation is performed on a small set of fixed, canonical test sets, which are carefully curated to measure the capabilities of interest and are not changed frequently. In this work, we design a method which specializes a prompted Autorater to a given test set, by leveraging historical ratings on the test set to construct in-context learning (ICL) examples. We evaluate our *Specialist* method on the task of fine-grained machine translation evaluation, and show that it dramatically outperforms the state-of-the-art XCOMET metric by 54% and 119% on the WMT'23 and WMT'24 test sets, respectively. We perform extensive analyses to understand the representations learned by our Specialist metrics, and how variability in rater behavior affects their performance. We also verify the generalizability and robustness of our Specialist method across different numbers of ICL examples, LLM backbones, systems to evaluate, and evaluation tasks.

Dialogue systems need to produce responses that realize multiple types of dialogue acts (DAs) with high semantic fidelity.In the past, natural language generators (NLGs) for dialogue were trained on large parallel corpora that map from a domain-specific DA and its semantic attributes to an output utterance.Recent work shows that pretrained language models (LLMs) offer new possibilities for controllable NLG using prompt-based learning.Here we develop a novel few-shot overgenerate-andrank approach that achieves the controlled generation of DAs.We compare eight few-shot prompt styles that include a novel method of generating from textual pseudo-references using a textual style transfer approach.We develop six automatic ranking functions that identify outputs with both the correct DA and high semantic accuracy at generation time.We test our approach on three domains and four LLMs.To our knowledge, this is the first work on NLG for dialogue that automatically ranks outputs using both DA and attribute accuracy.For completeness, we compare our results to fine-tuned few-shot models trained with 5 to 100 instances per DA.Our results show that several prompt settings achieve perfect DA accuracy, and near perfect semantic accuracy (99.81%) and perform better than few-shot fine-tuning.

Ever since neural models were adopted in datato-text language generation, they have invariably been reliant on extrinsic components to improve their semantic accuracy, because the models normally do not exhibit the ability to generate text that reliably mentions all of the information provided in the input.In this paper, we propose a novel decoding method that extracts interpretable information from encoder-decoder models' cross-attention, and uses it to infer which attributes are mentioned in the generated text, which is subsequently used to rescore beam hypotheses.Using this decoding method with T5 and BART, we show on three datasets its ability to dramatically reduce semantic errors in the generated outputs, while maintaining their state-of-the-art quality.

The uptake of deep learning in natural language generation (NLG) led to the release of both small and relatively large parallel corpora for training neural models.The existing datato-text datasets are, however, aimed at taskoriented dialogue systems, and often thus limited in diversity and versatility.They are typically crowdsourced, with much of the noise left in them.Moreover, current neural NLG models do not take full advantage of large training data, and due to their strong generalizing properties produce sentences that look template-like regardless.We therefore present a new corpus of 7K samples, which (1) is clean despite being crowdsourced, (2) has utterances of 9 generalizable and conversational dialogue act types, making it more suitable for opendomain dialogue systems, and (3) explores the domain of video games, which is new to dialogue systems despite having excellent potential for supporting rich conversations.

One of the biggest challenges of endto-end language generation from meaning representations in dialogue systems is making the outputs more natural and varied.Here we take a large corpus of 50K crowd-sourced utterances in the restaurant domain and develop text analysis methods that systematically characterize types of sentences in the training data.We then automatically label the training data to allow us to conduct two kinds of experiments with a neural generator.First, we test the effect of training the system with different stylistic partitions and quantify the effect of smaller, but more stylistically controlled training data.Second, we propose a method of labeling the style variants during training, and show that we can modify the style of the generated utterances using our stylistic labels.We contrast and compare these methods that can be used with any existing large corpus, showing how they vary in terms of semantic quality and stylistic control.

Juraj Juraska, Panagiotis Karagiannis, Kevin Bowden, Marilyn Walker. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.