Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, Hannaneh Hajishirzi. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, Kyle Lo. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew Peters, Arman Cohan. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

The integration of multi-document pre-training objectives into language models has resulted in remarkable improvements in multi-document downstream tasks.In this work, we propose extending this idea by pre-training a generic multi-document model from a novel crossdocument question answering pre-training objective.To that end, given a set (or cluster) of topically-related documents, we systematically generate semantically-oriented questions from a salient sentence in one document and challenge the model, during pre-training, to answer these questions while "peeking" into other topically-related documents.In a similar manner, the model is also challenged to recover the sentence from which the question was generated, again while leveraging cross-document information.This novel multidocument QA formulation directs the model to better recover cross-text informational relations, and introduces a natural augmentation that artificially increases the pre-training data.Further, unlike prior multi-document models that focus on either classification or summarization tasks, our pre-training objective formulation enables the model to perform tasks that involve both short text generation (e.g., QA) and long text generation (e.g., summarization).Following this scheme, we pre-train our model -termed QAMDEN -and evaluate its performance across several multi-document tasks, including multi-document QA, summarization, and query-focused summarization, yielding improvements of up to 7%, and significantly outperforms zero-shot GPT-3.5 and GPT-4. 1

Recent NLP models have shown the remarkable ability to effectively generalise 'zero-shot' to new tasks using only natural language instructions as guidance.However, many of these approaches suffer from high computational costs due to their reliance on concatenating lengthy instructions with every input example, resulting in costly reprocessing of the instruction.To avoid this, we introduce Hypernetworks for INstruction Tuning (HINT), which convert task instructions and examples into parameter-efficient modules inserted into an underlying model using a pretrained text encoder, eliminating the need to include instructions in the model input.The hypernetwork in HINT also produces an encoded instruction, which we concatenate with encoded inputs during decoding to further improve performance.HINT models outperform strong state-of-theart baselines by over 10% when controlling for compute (measured in FLOPs).By converting instructions into modules, HINT models can effectively disregard the length of instructions and few-shot example inputs in terms of compute usage.As a result, HINT can enhance its performance by up to 25% by incorporating additional few-shot data, while utilizing only up to 5% more compute.This combines the strengths of parameter-efficient fine-tuning and in-context learning.We release our code publicly 1 .

Large pre-trained models are capable of fewshot in-context learning (ICL), i.e., performing a new task by prepending a few demonstrations before the test input.However, the concatenated demonstrations are often excessively long and induce additional computation.Inspired by fusion-in-decoder (FiD) models which efficiently aggregate more passages and thus outperforms concatenation-based models in opendomain QA, we hypothesize that similar techniques can be applied to improve the efficiency and end-task performance of ICL.To verify this, we present a comprehensive study on applying three fusion methods-concatenationbased (early fusion), FiD (intermediate), and ensemble-based (late)-to ICL.We adopt a meta-learning setup where a model is first trained to perform ICL on a mixture of tasks using one selected fusion method, then evaluated on held-out tasks for ICL.Results on 11 heldout tasks show that FiD-ICL matches or outperforms the other two fusion methods.Additionally, we show that FiD-ICL (1) is 10x faster at inference time compared to concat-based and ensemble-based ICL, as we can easily precompute the representations of in-context examples and reuse them; (2) enables scaling up to meta-training 3B-sized models, which would fail for concat-based ICL.1

Controlled text perturbation is useful for evaluating and improving model generalizability.However, current techniques rely on training a model for every target perturbation, which is expensive and hard to generalize.We present Tailor, a semantically-controlled text generation system.Tailor builds on a pretrained seq2seq model and produces textual outputs conditioned on control codes derived from semantic representations.We craft a set of operations to modify the control codes, which in turn steer generation towards targeted attributes.These operations can be further composed into higher-level ones, allowing for flexible perturbation strategies.We demonstrate the effectiveness of these perturbations in multiple applications.First, we use Tailor to automatically create high-quality contrast sets for four distinct natural language processing (NLP) tasks.These contrast sets contain fewer spurious artifacts and are complementary to manually annotated ones in their lexical diversity.Second, we show that Tailor perturbations can improve model generalization through data augmentation.Perturbing just ∼2% of training data leads to a 5.8-point gain on an NLI challenge set measuring reliance on syntactic heuristics.

This work introduces a new multi-task, parameter-efficient language model (LM) tuning method that learns to transfer knowledge across different tasks via a mixture of soft prompts-small prefix embedding vectors pretrained for different tasks.Our method, called ATTEMPT (ATTEntional Mixtures of Prompt Tuning), obtains source prompts as encodings of large-scale source tasks into a small number of parameters and trains an attention module to interpolate the source prompts and a newly initialized target prompt for every instance in the target task.During training, only the target task prompt and the attention weights, which are shared between tasks in multi-task training, are updated, while the original LM and source prompts are intact.ATTEMPT is highly parameter-efficient (e.g., updates 2,300 times fewer parameters than full fine-tuning), while achieving high task performance using knowledge from high-resource tasks.Moreover, it is modular using pre-trained soft prompts and can flexibly add or remove source prompts for effective knowledge transfer.Our experimental results across 21 diverse NLP datasets show that ATTEMPT significantly outperforms prompt tuning and outperforms or matches fully finetuned or other parameter-efficient tuning approaches that use over ten times more parameters.Finally, ATTEMPT outperforms previous work in few-shot learning settings.

Rationalization is fundamental to human reasoning and learning.NLP models trained to produce rationales along with predictions, called self-rationalization models, have been investigated for their interpretability and utility to end-users.However, the extent to which training with human-written rationales facilitates learning remains an under-explored question.We ask whether training models to selfrationalize can aid in their learning to solve tasks for the right reasons.Specifically, we evaluate how training self-rationalization models with free-text rationales affects robustness to spurious correlations in fine-tuned encoderdecoder and decoder-only models of six different sizes.We evaluate robustness to spurious correlations by measuring performance on 1) manually annotated challenge datasets and 2) subsets of original test sets where reliance on spurious correlations would fail to produce correct answers.We find that while self-rationalization can improve robustness to spurious correlations in low-resource settings, it tends to hurt robustness in higher-resource settings.Furthermore, these effects depend on model family and size, as well as on rationale content.Together, our results suggest that explainability can come at the cost of robustness; thus, appropriate care should be taken when training self-rationalizing models with the goal of creating more trustworthy models.

The current standard approach to scaling transformer language models trains each model size from a different random initialization. As an alternative, we consider a staged training setup that begins with a small model and incrementally increases the amount of compute used for training by applying a "growth operator" to increase the model depth and width. By initializing each stage with the output of the previous one, the training process effectively re-uses the compute from prior stages and becomes more efficient. Our growth operators each take as input the entire training state (including model parameters, optimizer state, learning rate schedule, etc.) and output a new training state from which training continues. We identify two important properties of these growth operators, namely that they preserve both the loss and the “training dynamics” after applying the operator. While the loss-preserving property has been discussed previously, to the best of our knowledge this work is the first to identify the importance of preserving the training dynamics (the rate of decrease of the loss during training). To find the optimal schedule for stages, we use the scaling laws from (Kaplan et al., 2020) to find a precise schedule that gives the most compute saving by starting a new stage when training efficiency starts decreasing. We empirically validate our growth operators and staged training for autoregressive language models, showing up to 22% compute savings compared to a strong baseline trained from scratch. Our code is available at https://github.com/allenai/staged-training.

Alexandra Chronopoulou, Matthew Peters, Jesse Dodge. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Rowan Zellers, Ari Holtzman, Matthew Peters, Roozbeh Mottaghi, Aniruddha Kembhavi, Ali Farhadi, Yejin Choi. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Much recent work in NLP has documented dataset artifacts, bias, and spurious correlations between input features and output labels.However, how to tell which features have "spurious" instead of legitimate correlations is typically left unspecified.In this work we argue that for complex language understanding tasks, all simple feature correlations are spurious, and we formalize this notion into a class of problems which we call competency problems.For example, the word "amazing" on its own should not give information about a sentiment label independent of the context in which it appears, which could include negation, metaphor, sarcasm, etc.We theoretically analyze the difficulty of creating data for competency problems when human bias is taken into account, showing that realistic datasets will increasingly deviate from competency problems as dataset size increases.This analysis gives us a simple statistical test for dataset artifacts, which we use to show more subtle biases than were described in prior work, including demonstrating that models are inappropriately affected by these less extreme biases.Our theoretical treatment of this problem also allows us to analyze proposed solutions, such as making local edits to dataset instances, and to give recommendations for future data collection and model design efforts that target competency problems.

Typically, machine learning systems solve new tasks by training on thousands of examples.In contrast, humans can solve new tasks by reading some instructions, with perhaps an example or two.To take a step toward closing this gap, we introduce a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area.We instantiate this framework with a new English language dataset, ZEST, structured for task-oriented evaluation on unseen tasks.Formulating task descriptions as questions, we ensure each is general enough to apply to many possible inputs, thus comprehensively evaluating a model's ability to solve each task.Moreover, the dataset's structure tests specific types of systematic generalization.We find that the state-of-the-art T5 model achieves a score of 12% on ZEST, leaving a significant challenge for NLP researchers. 1

Large neural models have demonstrated human-level performance on language and vision benchmarks, while their performance degrades considerably on adversarial or out-of-distribution samples. This raises the question of whether these models have learned to solve a dataset rather than the underlying task by overfitting to spurious dataset biases. We investigate one recently proposed approach, AFLITE, which adversarially filters such dataset biases, as a means to mitigate the prevalent overestimation of machine performance. We provide a theoretical understanding for AFLITE, by situating it in the generalized framework for optimum bias reduction. We present extensive supporting evidence that AFLITE is broadly applicable for reduction of measurable dataset biases, and that models trained on the filtered datasets yield better generalization to out-of-distribution tasks. Finally, filtering results in a large drop in model performance (e.g., from 92% to 62% for SNLI), while human performance still remains high. Our work thus shows that such filtered datasets can pose new research challenges for robust generalization by serving as upgraded benchmarks.

Modeling human language requires the ability to not only generate fluent text but also encode factual knowledge.However, traditional language models are only capable of remembering facts seen at training time, and often have difficulty recalling them.To address this, we introduce the knowledge graph language model (KGLM), a neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context.These mechanisms enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens.We also introduce the Linked WikiText-2 dataset, 1 a corpus of annotated text aligned to the Wikidata knowledge graph whose contents (roughly) match the popular WikiText-2 benchmark (Merity et al., 2017).In experiments, we demonstrate that the KGLM achieves significantly better performance than a strong baseline language model.We additionally compare different language models' ability to complete sentences requiring factual knowledge, and show that the KGLM outperforms even very large language models in generating facts.

Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

We revisit domain adaptation for parsers in the neural era.First we show that recent advances in word representations greatly diminish the need for domain adaptation when the target domain is syntactically similar to the source domain.As evidence, we train a parser on the Wall Street Journal alone that achieves over 90% F 1 on the Brown corpus.For more syntactically distant domains, we provide a simple way to adapt a parser using only dozens of partial annotations.For instance, we increase the percentage of error-free geometry-domain parses in a held-out set from 45% to 73% using approximately five dozen training examples.In the process, we demonstrate a new state-of-the-art single model result on the Wall Street Journal test set of 94.3%.This is an absolute increase of 1.7% over the previous state-of-the-art of 92.6%.

Contextual word representations derived from pre-trained bidirectional language models (biLMs) have recently been shown to provide significant improvements to the state of the art for a wide range of NLP tasks.However, many questions remain as to how and why these models are so effective.In this paper, we present a detailed empirical study of how the choice of neural architecture (e.g.LSTM, CNN, or self attention) influences both end task accuracy and qualitative properties of the representations that are learned.We show there is a tradeoff between speed and accuracy, but all architectures learn high quality contextual representations that outperform word embeddings for four challenging NLP tasks.Additionally, all architectures learn representations that vary with network depth, from exclusively morphological based at the word embedding layer through local syntax based in the lower contextual layers to longer range semantics such coreference at the upper layers.Together, these results suggest that unsupervised biLMs, independent of architecture, are learning much more about the structure of language than previously appreciated.

Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

Pre-trained word embeddings learned from unlabeled text have become a standard component of neural network architectures for NLP tasks.However, in most cases, the recurrent network that operates on word-level representations to produce context sensitive representations is trained on relatively little labeled data.In this paper, we demonstrate a general semi-supervised approach for adding pretrained context embeddings from bidirectional language models to NLP systems and apply it to sequence labeling tasks.We evaluate our model on two standard datasets for named entity recognition (NER) and chunking, and in both cases achieve state of the art results, surpassing previous systems that use other forms of transfer or joint learning with additional labeled data and task specific gazetteers.

Bottom crevasses at the base of an iceberg or ice shelf are identified in radar sounding observations from their long echo tails. In November 2001, a radar sounding survey was conducted over iceberg B15A, which calved off from the Ross Ice Shelf, Antarctica, in March 2000. Pervasive basal cracking was observed, and the distribution of bottom crevasses along the flight lines is presented. The echo tails were quantitatively analyzed using a physically based model for backscattering from bottom crevasses. The identified crevasses are classified as either major water-filled crevasses or incipient/freezing crevasses, and estimates for crevasse heights are given

This paper presents focused synthetic aperture radar (SAR) processing of airborne radar sounding data acquired with the High-Capability Radar Sounder system at 60 MHz. The motivation is to improve basal reflection analysis for water detection and to improve layer detection and tracking. The processing and reflection analyses are applied to data from Kamb Ice Stream, West Antarctica. The SAR processor correlates the radar data with reference echoes from subsurface point targets. The references are 1-D responses limited by the pulse nadir footprint or 2-D responses that include echo tails. Unfocused SAR and incoherent integration are included for comparison. Echoes are accurately preserved from along-track slopes up to about 0.5deg for unfocused SAR, 3deg for 1-D correlations, and 10deg for 2-D correlations. The noise/clutter levels increase from unfocused SAR to 1-D and 2-D correlations, but additional gain compensates at the basal interface. The basal echo signal-to-noise ratio improvement is typically about 5 dB, and up to 10 dB for 2-D correlations in rough regions. The increased noise degrades the clarity of internal layers in the 2-D correlations, but detection of layers with slopes greater than 3deg is improved. Reflection coefficients are computed for basal water detection, and the results are compared for the different processing methods. There is a significant increase in the detected water from unfocused SAR to 1-D correlations, indicating that substantial basal water exists on moderately sloped interfaces. Very little additional water is detected from the 2-D correlations. The results from incoherent integration are close to the focused SAR results, but the noise/clutter levels are much greater.