Kushal Chawla, Hannah Rashkin, Gaurav Singh Tomar, David Reitter. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Abstract Large neural models have brought a new challenge to natural language generation (NLG): It has become imperative to ensure the safety and reliability of the output of models that generate freely. To this end, we present an evaluation framework, Attributable to Identified Sources (AIS), stipulating that NLG output pertaining to the external world is to be verified against an independent, provided source. We define AIS and a two-stage annotation pipeline for allowing annotators to evaluate model output according to annotation guidelines. We successfully validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset). We provide full annotation guidelines in the appendices and publicly release the annotated data at https://github.com/google-research-datasets/AIS.

AI researchers have posited Dungeons and Dragons (D&D) as a challenge problem to test systems on various language-related capabilities.In this paper, we frame D&D specifically as a dialogue system challenge, where the tasks are to both generate the next conversational turn in the game and predict the state of the game given the dialogue history.We create a gameplay dataset consisting of nearly 900 games, with a total of 7,000 players, 800,000 dialogue turns, 500,000 dice rolls, and 58 million words.We automatically annotate the data with partial state information about the game play.We train a large language model (LM) to generate the next game turn, conditioning it on different information.The LM can respond as a particular character or as the player who runs the game-i.e., the Dungeon Master (DM).It is trained to produce dialogue that is either in-character (roleplaying in the fictional world) or out-of-character (discussing rules or strategy).We perform a human evaluation to determine what factors make the generated output plausible and interesting.We further perform an automatic evaluation to determine how well the model can predict the game state given the history and examine how well tracking the game state improves its ability to produce plausible conversational output.

Compared to standard retrieval tasks, passage retrieval for conversational question answering (CQA) poses new challenges in understanding the current user question, as each question needs to be interpreted within the dialogue context.Moreover, it can be expensive to retrain well-established retrievers such as search engines that are originally developed for nonconversational queries.To facilitate their use, we develop a query rewriting model CONQRR that rewrites a conversational question in the context into a standalone question.It is trained with a novel reward function to directly optimize towards retrieval using reinforcement learning and can be adapted to any off-theshelf retriever.CONQRR achieves state-ofthe-art results on a recent open-domain CQA dataset containing conversations from three different sources, and is effective for two different off-the-shelf retrievers.Our extensive analysis also shows the robustness of CON-QRR to out-of-domain dialogues as well as to zero query rewriting supervision.

Abstract Knowledge-grounded dialogue systems powered by large language models often generate responses that, while fluent, are not attributable to a relevant source of information. Progress towards models that do not exhibit this issue requires evaluation metrics that can quantify its prevalence. To this end, we introduce the Benchmark for Evaluation of Grounded INteraction (Begin), comprising 12k dialogue turns generated by neural dialogue systems trained on three knowledge-grounded dialogue corpora. We collect human annotations assessing the extent to which the models’ responses can be attributed to the given background information. We then use Begin to analyze eight evaluation metrics. We find that these metrics rely on spurious correlations, do not reliably distinguish attributable abstractive responses from unattributable ones, and perform substantially worse when the knowledge source is longer. Our findings underscore the need for more sophisticated and robust evaluation metrics for knowledge-grounded dialogue. We make Begin publicly available at https://github.com/google/BEGIN-dataset.

Hannah Rashkin, David Reitter, Gaurav Singh Tomar, Dipanjan Das. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Why do bilinguals switch languages within a sentence? The present observational study asks whether word surprisal and word entropy predict code-switching in bilingual written conversation. We describe and model a new dataset of Chinese-English text with 1476 clean code-switched sentences, translated back into Chinese. The model includes known control variables together with word surprisal and word entropy. We found that word surprisal, but not entropy, is a significant predictor that explains code-switching above and beyond other well-known predictors. We also found sentence length to be a significant predictor, which has been related to sentence complexity. We propose high cognitive effort as a reason for code-switching, as it leaves fewer resources for inhibition of the alternative language. We also corroborate previous findings, but this time using a computational model of surprisal, a new language pair, and doing so for written language.

We face delays in a variety of situations. They are either inevitable, e.g., due to system limits, or are intentionally added, e.g., advertisements. In many situations, a visual feedback is provided during the delay to manage expectations. This feedback is usually provided through progress bars, percentages, or countdowns, depending on design limitations such as screen size. In this article, we use 15-second delays and examine (a) how delays affect users’ decision-making and task satisfaction, and (b) how to manipulate time perception to reduce the negative consequences of delays. Experiment 1 ( N =421) shows that faster countdowns increase task satisfaction and lead to more rational decisions in the subsequent task. In Experiment 2, we investigate the effect of countdown speed on delay perception and recall ( N =531). We show that faster countdowns lead to shorter perceived delays, while the delay will be recalled as longer after the task. The opposite is obtained for slower countdowns. We also increased the countdown rate and found a limit for the effect of increased speed. Thus, designers have to trade-off between how delays are perceived at the moment of experience and how they are recalled. We discuss the implications of these findings for user interface design.

We examine the benefits of visual context in training neural language models to perform next-word prediction.A multi-modal neural architecture is introduced that outperform its equivalent trained on language alone with a 2% decrease in perplexity, even when no visual context is available at test.Fine-tuning the embeddings of a pre-trained state-of-theart bidirectional language model (BERT) in the language modeling framework yields a 3.5% improvement.The advantage for training with visual context when testing without is robust across different languages (English, German and Spanish) and different models (GRU, LSTM, ∆-RNN, as well as those that use BERT embeddings).Thus, language models perform better when they learn like a baby, i.e, in a multi-modal environment.This finding is compatible with the theory of situated cognition: language is inseparable from its physical context.

Chris Alberti, Jeffrey Ling, Michael Collins, David Reitter. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

In this paper, we examine the origination and dispersion of neologisms from the perspective of both communities and cognitive modeling. We use the Reddit corpus to identify words that were first used by Reddit communities from 2013 to 2014. We induce a hierarchy on Reddit based on the specificity of the topic. Generally, less specific communities have more users, while more specific communities likely feature closer social ties. We ask whether larger numbers of people or closer social ties are better environments to faster the adoption of new words. We found that the majority of new words are first adopted/created in more general communities; though this account is relativized by the size of the communities. We also examined the pace of dispersion of such words in new communities and discuss how this relates to models of memory by using an ACT-R cognitive model. We discuss some parameterizations of such a model of memory and its implications in the word adoption paradigm. Finally, we show that there is an increasing trend in the number of new words being adopted/created in Reddit communities, even during a relatively short time period.

Linguistic alignment between dialogue partners has been claimed to be affected by their relative social power.A common finding has been that interlocutors of higher power tend to receive more alignment than those of lower power.However, these studies overlook some low-level linguistic features that can also affect alignment, which casts doubts on these findings.This work characterizes the effect of power on alignment with logistic regression models in two datasets, finding that the effect vanishes or is reversed after controlling for low-level features such as utterance length.Thus, linguistic alignment is explained better by low-level features than by social power.We argue that a wider range of factors, especially cognitive factors, need to be taken into account for future studies on observational data when social factors of language use are in question.

Jeremy Cole, David Reitter. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

We propose a perspective on dialogue that focuses on relative information contributions of conversation partners as a key to successful communication.We predict the success of collaborative task in English and Danish corpora of task-oriented dialogue.Two features are extracted from the frequency domain representations of the lexical entropy series of each interlocutor, power spectrum overlap (PSO) and relative phase (RP).We find that PSO is a negative predictor of task success, while RP is a positive one.An SVM with these features significantly improved on previous task success prediction models.Our findings suggest that the strategic distribution of information density between interlocutors is relevant to task success.

This paper improves on several aspects of a sieve-based event ordering architecture, CAEVO (Chambers et al., 2014), which creates globally consistent temporal relations between events and time expressions. First, we examine the usage of word embeddings and semantic role features. With the incorporation of these new features, we demonstrate a 5% relative F1 gain over our replicated version of CAEVO. Second, we reformulate the architecture’s sieve-based inference algorithm as a prediction reranking method that approximately optimizes a scoring function computed using classifier precisions. Within this prediction reranking framework, we propose an alternative scoring function, showing an 8.8% relative gain over the original CAEVO. We further include an in-depth analysis of one of the main datasets that is used to evaluate temporal classifiers, and we show how despite using the densest corpus, there is still a danger of overfitting. While this paper focuses on temporal ordering, its results are applicable to other areas that use sieve-based architectures.

Learning useful information across long time lags is a critical and difficult problem for temporal neural models in tasks such as language modeling. Existing architectures that address the issue are often complex and costly to train. The differential state framework (DSF) is a simple and high-performing design that unifies previously introduced gated neural models. DSF models maintain longer-term memory by learning to interpolate between a fast-changing data-driven representation and a slowly changing, implicitly stable state. Within the DSF framework, a new architecture is presented, the delta-RNN. This model requires hardly any more parameters than a classical, simple recurrent network. In language modeling at the word and character levels, the delta-RNN outperforms popular complex architectures, such as the long short-term memory (LSTM) and the gated recurrent unit (GRU), and, when regularized, performs comparably to several state-of-the-art baselines. At the subword level, the delta-RNN's performance is comparable to that of complex gated architectures.

Countdowns and progress bars provide computer users with estimates of remaining wait times. These types of feedback are intended to manage their expectations and allow users to direct attention elsewhere. We suggest that they also moderate user's impatience, which affects decision-making in the subsequent task. In an experiment with 421 participants, impatience in a timing decision task was effectively and systematically manipulated through a countdown, as it affected timing and performance of the user's actions in the task. The effect persisted even after users gained task experience. More rapid countdowns reduced impatience. Post-hoc analysis also showed increased task satisfaction with rising countdown speed and suggested greater task satisfaction with a rapid countdown than with no waiting period at all.

The applicability of entropy rate constancy to dialogue is examined on two spoken dialogue corpora.The principle is found to hold; however, new entropy change patterns within the topic episodes of dialogue are described, which are different from written text.Speaker's dynamic roles as topic initiators and topic responders are associated with decreasing and increasing entropy, respectively, which results in local convergence between these speakers in each topic episode.This implies that the sentence entropy in dialogue is conditioned on different contexts determined by the speaker's roles.Explanations from the perspectives of grounding theory and interactive alignment are discussed, resulting in a novel, unified informationtheoretic approach of dialogue.

We present a novel fine-tuning algorithm in a deep hybrid architecture for semisupervised text classification.During each increment of the online learning process, the fine-tuning algorithm serves as a top-down mechanism for pseudo-jointly modifying model parameters following a bottom-up generative learning pass.The resulting model, trained under what we call the Bottom-Up-Top-Down learning algorithm, is shown to outperform a variety of competitive models and baselines trained across a wide range of splits between supervised and unsupervised training data.

We report a way to build a series of GOMS-like cognitive user models representing a range of performance at different stages of learning. We use a spreadsheet task across multiple sessions as an example task; it takes about 20--30 min. to perform. The models were created in ACT-R using a compiler. The novice model has 29 rules and 1,152 declarative memory task elements (chunks)—it learns to create procedural knowledge to perform the task. The expert model has 617 rules and 614 task chunks (that it does not use) and 538 command string chunks—it gets slightly faster through limited declarative learning of the command strings and some further production compilation; there are a range of intermediate models. These models were tested against aggregate and individual human learning data, confirming the models’ predictions. This work suggests that user models can be created that learn like users while doing the task.

This paper addresses security and safety choices that involve a decision on the timing of an action. Examples of such decisions include when to check log files for intruders and when to monitor financial accounts for fraud or errors. To better understand how performance in timing-related security situations is shaped by individuals' cognitive predispositions, we effectively combine survey measures with economic experiments. Two behavioral experiments are presented in which the timing of online security actions is the critical decision-making factor. The feedback modality in the decision-environment is varied between visual feedback with history (Experiment 1), and temporal feedback without history (Experiment 2). Using psychometric scales, we study the role of individual difference variables, specifically risk propensity and need for cognition. The analysis is based on the data from over 450 participants. We find that risk propensity is not a hindrance in timing tasks. Participants of average risk propensity generally benefit from a reflective disposition (high need for cognition), particularly when visual feedback is given. Overall, participants benefit from need for cognition, however, in the more difficult, temporal-estimation task, this requires familiarity with the task.

Two information decay methods are examined that help multi-agent systems cope with dynamic environments. The agents in this simulation have human-like memory and a mechanism to moderate their communications: they forget internally stored information via temporal decay, and they forget distributed information by filtering it as it passes through a communication network. The agents play a foraging game, in which performance depends on communicating facts and requests and on storing facts in internal memory. Parameters of the game and agent models are tuned to human data. Agent groups with moderated communication in small-world networks achieve optimal performance for typical human memory decay values, while non-adaptive agents benefit from stronger memory decay. The decay and filtering strategies interact with the properties of the network graph in ways suggestive of an evolutionary co-optimization between the human cognitive system and an external social structure.

We discuss an experiment involving visual path planning for multiple, remote robots in a partially visible building, with a partial 2D map available. Participants in the experiment defined waypoints for each robot to circumnavigate obstacles and explore the building. A cognitively plausible model of visual planning is evaluated using a normalized metric of the fit between model and subject itineraries. We discuss variation in the data and model fit, indicating individual differences in strategies to cope with task demands.

In the context of the FASiL project, we have studied natural language interactions in a unimodal (speech only) and multimodal (speech and graphics) interface to a personal information management database. We collected multilingual corpora to investigate these interactions in Portuguese, English and Swedish. The corpora are used to train language models, to update acoustic models, to study semantic concepts, multimodal interactions, and dialogue management strategies. The corpora are annotated in a uniform way, with timings, transcriptions, and semantics. We report on the structure and design of the corpora which are now available via ELRA. 1.