Incremental models that process sentences one token at a time will sometimes encounter points where more than one interpretation is possible.Causal models are forced to output one interpretation and continue, whereas models that can revise may edit their previous output as the ambiguity is resolved.In this work, we look at how restart-incremental Transformers build and update internal states, in an effort to shed light on what processes cause revisions not viable in autoregressive models.We propose an interpretable way to analyse the incremental states, showing that their sequential structure encodes information on the garden path effect and its resolution.Our method brings insights on various bidirectional encoders for contextualised meaning representation and dependency parsing, contributing to show their advantage over causal models when it comes to revisions. 1

Active participation in a conversation is key to building common ground, since understanding is jointly tailored by producers and recipients.Overhearers are deprived of the privilege of performing grounding acts and can only conjecture about intended meanings.Still, data generation and annotation, modelling, training and evaluation of NLP dialogue models place reliance on the overhearing paradigm.How much of the underlying grounding processes are thereby forfeited?As we show, there is evidence pointing to the impossibility of properly modelling human meta-communicative acts with data-driven learning models.In this paper, we discuss this issue and provide a preliminary analysis on the variability of human decisions for requesting clarification.Most importantly, we wish to bring this topic back to the community's table, encouraging discussion on the consequences of having models designed to only "listen in".

In NLP, incremental processors produce output in instalments, based on incoming prefixes of the linguistic input.Some tokens trigger revisions, causing edits to the output hypothesis, but little is known about why models revise when they revise.A policy that detects the time steps where revisions should happen can improve efficiency.Still, retrieving a suitable signal to train a revision policy is an open problem, since it is not naturally available in datasets.In this work, we investigate the appropriateness of regressions and skips in human reading eye-tracking data as signals to inform revision policies in incremental sequence labelling.Using generalised mixed-effects models, we find that the probability of regressions and skips by humans can potentially serve as useful predictors for revisions in BiLSTMs and Transformer models, with consistent results for various languages.

In visual instruction-following dialogue games, players can engage in repair mechanisms in face of an ambiguous or underspecified instruction that cannot be fully mapped to actions in the world.In this work, we annotate Instruction Clarification Requests (iCRs) in CoDraw, an existing dataset of interactions in a multimodal collaborative dialogue game.We show that it contains lexically and semantically diverse iCRs being produced self-motivatedly by players deciding to clarify in order to solve the task successfully.With 8.8k iCRs found in 9.9k dialogues, CoDraw-iCR (v1) is a large spontaneous iCR corpus, making it a valuable resource for data-driven research on clarification in dialogue.We then formalise and provide baseline models for two tasks: Determining when to make an iCR and how to recognise them, in order to investigate to what extent these tasks are learnable from data.

Recent work has proposed a methodology for the systematic evaluation of "Situated Language Understanding Agents"-agents that operate in rich linguistic and non-linguistic contexts-through testing them in carefully constructed interactive settings.Other recent work has argued that Large Language Models (LLMs), if suitably set up, can be understood as (simulators of) such agents.A connection suggests itself, which this paper explores: Can LLMs be evaluated meaningfully by exposing them to constrained game-like settings that are built to challenge specific capabilities?As a proof of concept, this paper investigates five interaction settings, showing that current chatoptimised LLMs are, to an extent, capable of following game-play instructions.Both this capability and the quality of the game play, measured by how well the objectives of the different games are met, follows the development cycle, with newer models generally performing better.The metrics even for the comparatively simple example games are far from being saturated, suggesting that the proposed instrument will remain to have diagnostic value.Our general framework for implementing and evaluating games with LLMs is available at https://github.com/clembench.

Incremental dialogue model components produce a sequence of output prefixes based on incoming input.Mistakes can occur due to local ambiguities or to wrong hypotheses, making the ability to revise past outputs a desirable property that can be governed by a policy.In this work, we formalise and characterise edits and revisions in incremental sequence labelling and propose metrics to evaluate revision policies.We then apply our methodology to profile the incremental behaviour of three Transformerbased encoders in various tasks, paving the road for better revision policies.

Incremental processing allows interactive systems to respond based on partial inputs, which is a desirable property e.g. in dialogue agents.The currently popular Transformer architecture inherently processes sequences as a whole, abstracting away the notion of time.Recent work attempts to apply Transformers incrementally via restart-incrementality by repeatedly feeding, to an unchanged model, increasingly longer input prefixes to produce partial outputs.However, this approach is computationally costly and does not scale efficiently for long sequences.In parallel, we witness efforts to make Transformers more efficient, e.g. the Linear Transformer (LT) with a recurrence mechanism.In this work, we examine the feasibility of LT for incremental NLU in English.Our results show that the recurrent LT model has better incremental performance and faster inference speed compared to the standard Transformer and LT with restartincrementality, at the cost of part of the nonincremental (full sequence) quality.We show that the performance drop can be mitigated by training the model to wait for right context before committing to an output and that training with input prefixes is beneficial for delivering correct partial outputs.

While humans process language incrementally, the best language encoders currently used in NLP do not.Both bidirectional LSTMs and Transformers assume that the sequence that is to be encoded is available in full, to be processed either forwards and backwards (BiL-STMs) or as a whole (Transformers).We investigate how they behave under incremental interfaces, when partial output must be provided based on partial input seen up to a certain time step, which may happen in interactive systems.We test five models on various NLU datasets and compare their performance using three incremental evaluation metrics.The results support the possibility of using bidirectional encoders in incremental mode while retaining most of their non-incremental quality.The "omni-directional" BERT model, which achieves better non-incremental performance, is impacted more by the incremental access.This can be alleviated by adapting the training regime (truncated training), or the testing procedure, by delaying the output until some right context is available or by incorporating hypothetical right contexts generated by a language model like GPT-2.