Charles Lovering, Michael Krumdick, Viet Dac Lai, Varshini Reddy, Seth Ebner, Nilesh Kumar, Rik Koncel-Kedziorski, Chris Tanner. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

As the adoption of language models advances, so does the need to better represent individual users to the model.Are there aspects of an individual's belief system that a language model can utilize for improved alignment?Following prior research, we investigate this question in the domain of opinion prediction by developing PRIMEX, a dataset of public opinion survey data from 858 US residents with two additional sources of belief information: written explanations from the respondents for why they hold specific opinions, and the Primal World Belief survey for assessing respondent worldview.We provide an extensive initial analysis of our data and show the value of belief explanations and worldview for personalizing language models.Our results demonstrate how the additional belief information in PRIMEX can benefit both the NLP and psychological research communities, opening up avenues for further study.Public Opinion Surveys on multiple topics 858 Participants of various age, gender, education, race, & US region 1 2 3 Free-response explanations of 3 questions per survey Good vs. Bad Safe vs.

Michael Krumdick, Rik Koncel-Kedziorski, Viet Dac Lai, Varshini Reddy, Charles Lovering, Chris Tanner. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Recent studies show that sentence-level extractive QA, i.e., based on Answer Sentence Selection (AS2), is outperformed by Generationbased QA (GenQA) models, which generate answers using the top-k answer sentences ranked by AS2 models (a la retrieval-augmented generation style).In this paper, we propose a novel training paradigm for GenQA using supervision from automatic QA evaluation models (GAVA).Specifically, we propose three strategies to transfer knowledge from these QA evaluation models to a GenQA model: (i) augmenting training data with answers generated by the GenQA model and labelled by GAVA (either statically, before training, or (ii) dynamically, at every training epoch); and (iii) using the GAVA score for weighting the generator loss during the learning of the GenQA model.We evaluate our proposed methods on two academic and one industrial dataset, obtaining a significant improvement in answering accuracy over the previous state of the art.

Modern neural language models can produce remarkably fluent and grammatical text.So much, in fact, that recent work by Clark et al. (2021) has reported that conventional crowdsourcing can no longer reliably distinguish between machine-authored (GPT-3) and humanauthored writing.As errors in machine generations become ever subtler and harder to spot, it poses a new challenge to the research community for robust machine text evaluation.We propose a new framework called SCARE-CROW for scrutinizing machine text via crowd annotation.To support the broad range of real machine errors that can be identified by laypeople, the ten error categories of SCARECROWsuch as redundancy , commonsense errors , and incoherence -are identified through several rounds of crowd annotation experiments without a predefined ontology.We then use SCARECROW to collect over 41k error spans in human-written and machinegenerated paragraphs of English language news text.We isolate factors for detailed analysis, including parameter count, training data, and various decoding-time configurations.Our approach successfully quantifies measurable gaps between human authored text and generations from models of several sizes, including fourteen configurations of GPT-3.In addition, our analysis unveils new insights, with detailed rationales provided by laypeople, e.g., that the commonsense capabilities have been improving with larger models while math capabilities have not, and that the choices of simple decoding hyperparameters can make remarkable differences on the perceived quality of machine text.We release our training material, annotation toolkit and dataset at

Recent studies show that Question Answering (QA) based on Answer Sentence Selection (AS2) can be improved by generating an improved answer from the top-k ranked answer sentences (termed GenQA).This allows for synthesizing the information from multiple candidates into a concise, natural-sounding answer.However, creating large-scale supervised training data for GenQA models is very challenging.In this paper, we propose to train a GenQA model by transferring knowledge from a trained AS2 model, to overcome the aforementioned issue.First, we use an AS2 model to produce a ranking over answer candidates for a set of questions.Then, we use the top ranked candidate as the generation target, and the next k top ranked candidates as context for training a GenQA model.We also propose to use the AS2 model prediction scores for loss weighting and score-conditioned input/output shaping, to aid the knowledge transfer.Our evaluation on three public and one large industrial datasets demonstrates the superiority of our approach over the AS2 baseline, and GenQA trained using supervised data.

Current end-to-end neural conversation models inherently lack the flexibility to impose semantic control in the response generation process, often resulting in uninteresting responses. Attempts to boost informativeness alone come at the expense of factual accuracy, as attested by pretrained language models' propensity to "hallucinate" facts. While this may be mitigated by access to background knowledge, there is scant guarantee of relevance and informativeness in generated responses. We propose a framework that we call controllable grounded response generation (CGRG), in which lexical control phrases are either provided by a user or automatically extracted by a control phrase predictor from dialogue context and grounding knowledge. Quantitative and qualitative results show that, using this framework, a transformer based model with a novel inductive attention mechanism, trained on a conversation-like Reddit dataset, outperforms strong generation baselines.

Kelvin Luu, Xinyi Wu, Rik Koncel-Kedziorski, Kyle Lo, Isabel Cachola, Noah A. Smith. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

LSTMs are powerful tools for modeling contextual information, as evidenced by their success at the task of language modeling.However, modeling contexts in very high dimensional space can lead to poor generalizability.We introduce the Pyramidal Recurrent Unit (PRU), which enables learning representations in high dimensional space with more generalization power and fewer parameters.PRUs replace the linear transformation in LSTMs with more sophisticated interactions including pyramidal and grouped linear transformations.This architecture gives strong results on wordlevel language modeling while reducing the number of parameters significantly.In particular, PRU improves the perplexity of a recent state-of-the-art language model Merity et al. (2018) by up to 1.3 points while learning 15-20% fewer parameters.For similar number of model parameters, PRU outperforms all previous RNN models that exploit different gating mechanisms and transformations.We provide a detailed examination of the PRU and its behavior on the language modeling tasks.

Texts present coherent stories that have a particular theme or overall setting, for example science fiction or western.In this paper, we present a text generation method called rewriting that edits existing human-authored narratives to change their theme without changing the underlying story.We apply the approach to math word problems, where it might help students stay more engaged by quickly transforming all of their homework assignments to the theme of their favorite movie without changing the math concepts that are being taught.Our rewriting method uses a twostage decoding process, which proposes new words from the target theme and scores the resulting stories according to a number of factors defining aspects of syntactic, semantic, and thematic coherence.Experiments demonstrate that the final stories typically represent the new theme well while still testing the original math concepts, outperforming a number of baselines.We also release a new dataset of human-authored rewrites of math word problems in several themes.

Aaron Jaech, Rik Koncel-Kedziorski, Mari Ostendorf. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016.

Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, Hannaneh Hajishirzi. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016.

This paper formalizes the problem of solving multi-sentence algebraic word problems as that of generating and scoring equation trees. We use integer linear programming to generate equation trees and score their likelihood by learning local and global discriminative models. These models are trained on a small set of word problems and their answers, without any manual annotation, in order to choose the equation that best matches the problem text. We refer to the overall system as Alges. We compare Alges with previous work and show that it covers the full gamut of arithmetic operations whereas Hosseini et al. (2014) only handle addition and subtraction. In addition, Alges overcomes the brittleness of the Kushman et al. (2014) approach on single-equation problems, yielding a 15% to 50% reduction in error.

Language is given meaning through its correspondence with a world representation.This correspondence can be at multiple levels of granularity or resolutions.In this paper, we introduce an approach to multi-resolution language grounding in the extremely challenging domain of professional soccer commentaries.We define and optimize a factored objective function that allows us to leverage discourse structure and the compositional nature of both language and game events.We show that finer resolution grounding helps coarser resolution grounding, and vice versa.Our method results in an F1 improvement of more than 48% versus the previous state of the art for fine-resolution grounding 1 .