Salient Span Masking (SSM) has shown itself to be an effective strategy to improve closedbook question answering performance.SSM extends general masked language model pretraining by creating additional unsupervised training sentences that mask a single entity or date span, thus oversampling factual information.Despite the success of this paradigm, the span types and sampling strategies are relatively arbitrary and not widely studied for other tasks.Thus, we investigate SSM from the perspective of temporal tasks, where learning a good representation of various temporal expressions is important.To that end, we introduce Temporal Span Masking (TSM) intermediate training.First, we find that SSM alone improves the downstream performance on three temporal tasks by an avg.+5.8 points.Further, we are able to achieve additional improvements (avg.+0.29 points) by adding the TSM task.These comprise the new best reported results on the targeted tasks.Our analysis suggests that the effectiveness of SSM stems from the sentences chosen in the training data rather than the mask choice: sentences with entities frequently also contain temporal expressions.Nonetheless, the additional targeted spans of TSM can still improve performance, especially in a zero-shot context.

Jeremy R. Cole, Palak Jain, Julian Martin Eisenschlos, Michael J.Q. Zhang, Eunsol Choi, Bhuwan Dhingra. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.

We introduce a new in-context learning paradigm to measure Large Language Models' (LLMs) ability to learn novel words during inference.In particular, we rewrite Winogradstyle co-reference resolution problems by replacing the key concept word with a synthetic but plausible word that the model must understand to complete the task.Solving this task requires the model to make use of the dictionary definition of the new word given in the prompt.This benchmark addresses word acquisition, one important aspect of the diachronic degradation known to afflict LLMs.As LLMs are frozen in time at the moment they are trained, they are normally unable to reflect the way language changes over time.We show that the accuracy of LLMs compared to the original Winograd tasks decreases radically in our benchmark, thus identifying a limitation of current models and providing a benchmark to measure future improvements in LLMs ability to do in-context learning.

Trustworthy language models should abstain from answering questions when they do not know the answer.However, the answer to a question can be unknown for a variety of reasons.Prior research has focused on the case in which the question is clear and the answer is unambiguous but possibly unknown.But the answer to a question can also be unclear due to uncertainty of the questioner's intent or context.We investigate question answering from this perspective, focusing on answering a subset of questions with a high degree of accuracy, from a set of questions in which many are inherently ambiguous.In this setting, we find that the most reliable approach to decide when to abstain involves quantifying repetition within sampled model outputs, rather than the model's likelihood or self-verification as used in prior work.We find this to be the case across different types of uncertainty and model scales, and with or without instruction tuning.Our results suggest that sampling-based confidence scores help calibrate answers to relatively unambiguous questions, with more dramatic improvements on ambiguous questions.

Neural document rerankers are extremely effective in terms of accuracy.However, the best models require dedicated hardware for serving, which is costly and often not feasible.To avoid this serving-time requirement, we present a method of capturing up to 86% of the gains of a Transformer cross-attention model with a lexicalized scoring function that only requires 10 -6 % of the Transformer's FLOPs per document and can be served using commodity CPUs.When combined with a BM25 retriever, this approach matches the quality of a state-ofthe art dual encoder retriever, that still requires an accelerator for query encoding.We introduce NAIL (Non-Autoregressive Indexing with Language models) as a model architecture that is compatible with recent encoder-decoder and decoder-only large language models, such as T5, GPT-3 and PaLM.This model architecture can leverage existing pre-trained checkpoints and can be fine-tuned for efficiently constructing document representations that do not require neural processing of queries.

Abstract Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. However, most language models (LMs) are trained on snapshots of data collected at a specific moment in time. This can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum—those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently “refreshed” as new data arrives, without the need for retraining from scratch.

Why do bilinguals switch languages within a sentence? The present observational study asks whether word surprisal and word entropy predict code-switching in bilingual written conversation. We describe and model a new dataset of Chinese-English text with 1476 clean code-switched sentences, translated back into Chinese. The model includes known control variables together with word surprisal and word entropy. We found that word surprisal, but not entropy, is a significant predictor that explains code-switching above and beyond other well-known predictors. We also found sentence length to be a significant predictor, which has been related to sentence complexity. We propose high cognitive effort as a reason for code-switching, as it leaves fewer resources for inhibition of the alternative language. We also corroborate previous findings, but this time using a computational model of surprisal, a new language pair, and doing so for written language.

In this paper, we examine the origination and dispersion of neologisms from the perspective of both communities and cognitive modeling. We use the Reddit corpus to identify words that were first used by Reddit communities from 2013 to 2014. We induce a hierarchy on Reddit based on the specificity of the topic. Generally, less specific communities have more users, while more specific communities likely feature closer social ties. We ask whether larger numbers of people or closer social ties are better environments to faster the adoption of new words. We found that the majority of new words are first adopted/created in more general communities; though this account is relativized by the size of the communities. We also examined the pace of dispersion of such words in new communities and discuss how this relates to models of memory by using an ACT-R cognitive model. We discuss some parameterizations of such a model of memory and its implications in the word adoption paradigm. Finally, we show that there is an increasing trend in the number of new words being adopted/created in Reddit communities, even during a relatively short time period.

Linguistic alignment between dialogue partners has been claimed to be affected by their relative social power.A common finding has been that interlocutors of higher power tend to receive more alignment than those of lower power.However, these studies overlook some low-level linguistic features that can also affect alignment, which casts doubts on these findings.This work characterizes the effect of power on alignment with logistic regression models in two datasets, finding that the effect vanishes or is reversed after controlling for low-level features such as utterance length.Thus, linguistic alignment is explained better by low-level features than by social power.We argue that a wider range of factors, especially cognitive factors, need to be taken into account for future studies on observational data when social factors of language use are in question.

Jeremy Cole, David Reitter. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.