Nishant Balepur, Malachi Hamada, Varsha Kishore, Sergey Feldman, Amanpreet Singh, Pao Siangliulue, Joseph Chee Chang, Eunsol Choi, Jordan Lee Boyd-Graber, Aakanksha Naik. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

All text-based language problems can be reduced to either generation or embedding. Current models only perform well at one or the other. We introduce generative representational instruction tuning (GRIT) whereby a large language model is trained to handle both generative and embedding tasks by distinguishing between them through instructions. Compared to other open models, our resulting GritLM-7B is among the top models on the Massive Text Embedding Benchmark (MTEB) and outperforms various models up to its size on a range of generative tasks. By scaling up further, GritLM-8x7B achieves even stronger generative performance while still being among the best embedding models. Notably, we find that GRIT matches training on only generative or embedding data, thus we can unify both at no performance loss. Among other benefits, the unification via GRIT speeds up Retrieval-Augmented Generation (RAG) by > 60% for long documents, by no longer requiring separate retrieval and generation models. Models, code, etc. are freely available at https://github.com/ContextualAI/gritlm.

Abstract Large Language Models (LLMs) are often aligned using contrastive alignment objectives and preference pair datasets. The interaction between model, paired data, and objective makes alignment a complicated procedure, sometimes producing subpar results. We study this and find that (i) preference data gives a better learning signal when the underlying responses are contrastive, and (ii) alignment objectives lead to better performance when they specify more control over the model during training. Based on these insights, we introduce Contrastive Learning from AI Revisions (CLAIR), a data-creation method which leads to more contrastive preference pairs, and Anchored Preference Optimization (APO), a controllable and more stable alignment objective. We align Llama-3-8B-Instruct using various comparable datasets and alignment objectives and measure MixEval-Hard scores, which correlate highly with human judgments. The CLAIR preferences lead to the strongest performance out of all datasets, and APO consistently outperforms less controllable objectives. Our best model, trained on 32K CLAIR preferences with APO, improves Llama-3-8B-Instruct by 7.65%, closing the gap with GPT4-turbo by 45%. Our code and datasets are available.

Learned representations of scientific documents can serve as valuable input features for downstream tasks without further fine-tuning.However, existing benchmarks for evaluating these representations fail to capture the diversity of relevant tasks.In response, we introduce SciRepEval, the first comprehensive benchmark for training and evaluating scientific document representations.It includes 24 challenging and realistic tasks, 8 of which are new, across four formats: classification, regression, ranking and search.We then use this benchmark to study and improve the generalization ability of scientific document representation models.We show how state-of-the-art models like SPECTER and SciNCL struggle to generalize across the task formats, and that simple multi-task training fails to improve them.However, a new approach that learns multiple embeddings per document, each tailored to a different format, can improve performance.We experiment with task-format-specific control codes and adapters and find they outperform the existing single-embedding state-of-the-art by over 2 points absolute.We release the resulting family of multi-format models, called SPECTER2, for the community to use and build on.

Large multimodal models trained on natural documents, which interleave images and text, outperform models trained on image-text pairs on various multimodal benchmarks. However, the datasets used to train these models have not been released, and the collection process has not been fully specified. We introduce the OBELICS dataset, an open web-scale filtered dataset of interleaved image-text documents comprising 141 million web pages extracted from Common Crawl, 353 million associated images, and 115 billion text tokens. We describe the dataset creation process, present comprehensive filtering rules, and provide an analysis of the dataset's content. To show the viability of OBELICS, we train on the dataset vision and language models of 9 and 80 billion parameters, IDEFICS-9B and IDEFICS, and obtain competitive performance on different multimodal benchmarks. We release our dataset, models and code.

State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks. Generally, such models are often either cross-modal (contrastive) or multi-modal (with earlier fusion) but not both; and they often only target specific modalities or tasks. A promising direction would be to use a single holistic universal model, as a “foundation”, that targets all modalities at once-a true vision and language foundation model should be good at vision tasks, language tasks, and cross- and multi-modal vision and language tasks. We introduce FLAVA as such a model and demonstrate impressive performance on a wide range of 35 tasks spanning these target modalities.

We present a novel task and dataset for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning, which we call Winoground. Given two images and two captions, the goal is to match them correctly-but crucially, both captions contain a completely identical set of words, only in a different order. The dataset was carefully hand-curated by expert annotators and is labeled with a rich set offine-grained tags to assist in analyzing model performance. We probe a diverse range of state-of-the-art vision and language models and find that, surprisingly, none of them do much better than chance. Evidently, these models are not as skilled at visio-linguistic compositional reasoning as we might have hoped. We perform an extensive analysis to obtain insights into how future work might try to mitigate these models' shortcomings. We aim for Winoground to serve as a useful evaluation set for advancing the state of the art and driving further progress in the field. The dataset is available at https://huggingface.co/datasets/facebook/winoground.

Vision-and-Language (V+L) pre-training models have achieved tremendous success in recent years on various multi-modal benchmarks. However, the majority of existing models require pre-training on a large set of parallel imagetext data, which is costly to collect, compared to image-only or text-only data. In this paper, we explore unsupervised Vision-and-Language pre-training (UVLP) to learn the cross-modal representation from non-parallel image and text datasets. We found two key factors that lead to good unsupervised V + L pre-training without parallel data: (i) joint image-and-text input (ii) overall imagetext alignment (even for non-parallel data). Accordingly, we propose a novel unsupervised V + L pre-training curriculum for non-parallel texts and images. We first construct a weakly aligned imagetext corpus via a retrieval-based approach, then apply a set of multi-granular alignment pre-training tasks, including region-to-tag, region-to-phrase, and image-to-sentence alignment, to bridge the gap between the two modalities. A comprehensive ablation study shows each granularity is helpful to learn a stronger pre-trained model. We adapt our pre-trained model to a set of V+L downstream tasks, including VQA, NLVR2, Visual Entailment, and Ref-COCO+. Our model achieves the state-of-art performance in all these tasks under the unsupervised setting.

Can NLP assist in building formal models for verifying complex systems? We study this challenge in the context of parsing Network File System (NFS) specifications. We define a semantic-dependency problem over SpecIR, a representation language we introduce to model sentences appearing in NFS specification documents (RFCs) as IF-THEN statements, and present an annotated dataset of 1,198 sentences. We develop and evaluate semantic-dependency parsing systems for this problem. Evaluations show that even when using a state-of-the-art language model, there is significant room for improvement, with the best models achieving an F1 score of only 60.5 and 33.3 in the named-entity-recognition and dependency-link-prediction sub-tasks, respectively. We also release additional unlabeled data and other domain-related texts. Experiments show that these additional resources increase the F1 measure when used for simple domain-adaption and transfer-learning-based approaches, suggesting fruitful directions for further research

The vast scale and open-ended nature of knowledge graphs (KGs) make exploratory search over them cognitively demanding for users. We introduce a new technique, polymorphic lenses, that improves exploratory search over a KG by obtaining new leverage from the existing preference models that KG-based systems maintain for recommending content. The approach is based on a simple but powerful observation: in a KG, preference models can be re-targeted to recommend not only entities of a single base entity type (e.g., papers in the scientific literature KG, products in an e-commerce KG), but also all other types (e.g., authors, conferences, institutions; sellers, buyers). We implement our technique in a novel system, FeedLens, which is built over Semantic Scholar, a production system for navigating the scientific literature KG. FeedLens reuses the existing preference models on Semantic Scholar—people’s curated research feeds—as lenses for exploratory search. Semantic Scholar users can curate multiple feeds/lenses for different topics of interest, e.g., one for human-centered AI and another for document embeddings. Although these lenses are defined in terms of papers, FeedLens re-purposes them to also guide search over authors, institutions, venues, etc. Our system design is based on feedback from intended users via two pilot surveys (n = 17 and n = 13, respectively). We compare FeedLens and Semantic Scholar via a third (within-subjects) user study (n = 15) and find that FeedLens increases user engagement while reducing the cognitive effort required to complete a short literature review task. Our qualitative results also highlight people’s preference for this more effective exploratory search experience enabled by FeedLens.

A crucial component for the scene text based reasoning required for TextVQA and TextCaps datasets involve detecting and recognizing text present in the images using an optical character recognition (OCR) system. The current systems are crippled by the unavailability of ground truth text annotations for these datasets as well as lack of scene text detection and recognition datasets on real images disallowing the progress in the field of OCR and evaluation of scene text based reasoning in isolation from OCR systems. In this work, we propose TextOCR, an arbitrary-shaped scene text detection and recognition with 900k annotated words collected on real images from TextVQA dataset. We show that current state-of-the-art text-recognition (OCR) models fail to perform well on TextOCR and that training on TextOCR helps achieve state-of-the-art performance on multiple other OCR datasets as well. We use a TextOCR trained OCR model to create PixelM4C model which can do scene text based reasoning on an image in an end-to-end fashion, allowing us to revisit several design choices to achieve new state-of-the-art performance on TextVQA dataset.

We propose UniT, a Unified Transformer model to simultaneously learn the most prominent tasks across different domains, ranging from object detection to natural language understanding and multimodal reasoning. Based on the transformer encoder-decoder architecture, our UniT model encodes each input modality with an encoder and makes predictions on each task with a shared decoder over the encoded input representations, followed by task-specific output heads. The entire model is jointly trained end-to-end with losses from each task. Compared to previous efforts on multi-task learning with transformers, we share the same model parameters across all tasks instead of separately fine-tuning task-specific models and handle a much higher variety of tasks across different domains. In our experiments, we learn 7 tasks jointly over 8 datasets, achieving strong performance on each task with significantly fewer parameters. Our code is available in MMF at https://mmf.sh.

Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Performance on the most commonly used Visual Question Answering dataset (VQA v2) is starting to approach human accuracy. However, in interacting with state-of-the-art VQA models, it is clear that the problem is far from being solved. In order to stress test VQA models, we benchmark them against human-adversarial examples. Human subjects interact with a state-of-the-art VQA model, and for each image in the dataset, attempt to find a question where the model’s predicted answer is incorrect. We find that a wide range of state-of-the-art models perform poorly when evaluated on these examples. We conduct an extensive analysis of the collected adversarial examples and provide guidance on future research directions. We hope that this Adversarial VQA (AdVQA) benchmark can help drive progress in the field and advance the state of the art.

Many visual scenes contain text that carries crucial information, and it is thus essential to understand text in images for downstream reasoning tasks. For example, a deep water label on a warning sign warns people about the danger in the scene. Recent work has explored the TextVQA task that requires reading and understanding text in images to answer a question. However, existing approaches for TextVQA are mostly based on custom pairwise fusion mechanisms between a pair of two modalities and are restricted to a single prediction step by casting TextVQA as a classification task. In this work, we propose a novel model for the TextVQA task based on a multimodal transformer architecture accompanied by a rich representation for text in images. Our model naturally fuses different modalities homogeneously by embedding them into a common semantic space where self-attention is applied to model inter- and intra- modality context. Furthermore, it enables iterative answer decoding with a dynamic pointer network, allowing the model to form an answer through multi-step prediction instead of one-step classification. Our model outperforms existing approaches on three benchmark datasets for the TextVQA task by a large margin.

In this tutorial we show how to build deep learning recommendation systems and resolve the associated interpretability, integrity and privacy challenges. We start with an overview of the PyTorch framework, features that it offers and a brief review of the evolution of recommendation models. We delineate their typical components and build a proxy deep learning recommendation model (DLRM) in PyTorch. Then, we discuss how to interpret recommendation system results as well as how to address the corresponding integrity and quality challenges.

This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. It is constructed such that unimodal models struggle and only multimodal models can succeed: difficult examples (“benign confounders”) are added to the dataset to make it hard to rely on unimodal signals. The task requires subtle reasoning, yet is straightforward to evaluate as a binary classification problem. We provide baseline performance numbers for unimodal models, as well as for multimodal models with various degrees of sophistication. We find that state-of-the-art methods perform poorly compared to humans, illustrating the difficulty of the task and highlighting the challenge that this important problem poses to the community.

Studies have shown that a dominant class of questions asked by visually impaired users on images of their surroundings involves reading text in the image. But today’s VQA models can not read! Our paper takes a first step towards addressing this problem. First, we introduce a new “TextVQA” dataset to facilitate progress on this important problem. Existing datasets either have a small proportion of questions about text (e.g., the VQA dataset) or are too small (e.g., the VizWiz dataset). TextVQA contains 45,336 questions on 28,408 images that require reasoning about text to answer. Second, we introduce a novel model architecture that reads text in the image, reasons about it in the context of the image and the question, and predicts an answer which might be a deduction based on the text and the image or composed of the strings found in the image. Consequently, we call our approach Look, Read, Reason & Answer (LoRRA). We show that LoRRA outperforms existing state-of-the-art VQA models on our TextVQA dataset. We find that the gap between human performance and machine performance is significantly larger on TextVQA than on VQA 2.0, suggesting that TextVQA is well-suited to benchmark progress along directions complementary to VQA 2.0.

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at https://super.gluebenchmark.com.

This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al. (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.’s models and ours learn systematic generalizations like subject-verb-object order. However, all models we test perform far below human level on a wide range of grammatical constructions.

Conventional kinematic studies of serial manipulators involve the proper selection of coordinate frames of reference at appropriate positions. The standard practice, being used in the past, is the use of Denavit-Hartenberg (D-H) algorithm for assigning coordinate frames. However, it has been observed that when an open kinematic chain contains a spatial link with two consecutive joint axes at right angle to each other, the forward kinematics derived with D-H algorithm comes geometrically inconsistent. A typical spatial link involves more than one non-zero link/joint parameters, which are not being accounted for, in the corresponding D-H parameter table. Forward kinematic study of manipulators involving spatial links with two consecutive joint axes at right angles to each other leads to recognizable deficiency of the D-H algorithm, as one of its practical limitation. In the present work, the concept of dummy frames is proposed to eliminate this deficiency. The proposed concept is demonstrated successfully for the case study of a Manipulator for Medical Application (MMA), which is a seven degrees-of-freedom (DOF) manipulator containing spatial links. Both geometrical and physical validation is performed to ensure the efficacy of the proposed concept.

With the onset of multiple wireless technologies, the end-host devices of today are multi-homed. This has led to research in the simultaneous use of multiple paths between the multi-homed end devices. New multipath transport protocols such as Multipath TCP and Concurrent Multipath Transfer SCTP need to be fair to the existing transport protocols like TCP and SCTP in a congested network. One way of being fair is to couple the congestion control mechanisms of the protocol over its multiple utilized paths. This paper illustrates and compares the solutions that have been proposed to ascertain that Multipath TCP can yield an improved performance while being fair to standard TCP. In addition, this paper highlights improvements to one of the proposed solution - Dynamic Window Coupling. It is shown that the new solution performs best in most of the different scenarios of bottlenecks in a network with a higher preference given to being friendlier to TCP.

QoS/QoE becomes increasingly important due to the growing usage of multimedia applications. In this paper, a solution is presented where Multipath TCP which supports the mobility on the transport layer is combined with QoS-enabled OSPF for performance-aware route selection. For the latter protocol, a novel extension called QoSxOSPF-LSFM (Load Sensitive Flow Management) is introduced which increases the stability of OSPFxQoS routing by evaluating the link utilization and reducing the frequency of route changes. For comparison, results are shown for the combination of QoSxOSPF-LSFM both with MPTCP as a multipath-capable transport-layer based approach as well as with Mobile IP which is a network layer based mobility approach. Simulation results prove the increase of performance compared to the case without LSFM extensions both for MPTCP and Mobile IP; however, due to the multipath capabilities, MPTCP in all cases outperforms Mobile IP.

Multipath transport protocols such as Multipath TCP can concurrently use several subflows to transmit a TCP flow over potentially different paths. Since more than one subflow is used, an efficient multipath scheduling algorithm is needed at the sender. The objective of the scheduler is to identify the subflow over which the current data packet should be sent. This paper compares the most important types of schedulers for multipath transfers. We model their performance analytically and derive key metrics, most notably the resulting end-to-end delay over heterogeneous paths. Our results show that a scheduler minimizing the packet delivery delay yields the best overall performance, but it is complex to realize. An alternative scheduler based on the sender queue size is simpler and has sufficient performance for relatively small asymmetry between the multiple paths. Our model results are confirmed by measurements with a real multipath transport protocol.

As wireless IP-based networks become more popular and larger in size and coverage, simple mobility management protocols cannot deal with the demands. With the growing access to the Internet, mobility can be divided into macro/micro-mobility domains. Macro-mobility can be efficiently handled with Mobile IP and its derivatives but they do not scale that well for the micro-mobility case. An alternative for micro-mobility domains is to use a reactive route discovery mechanism. Ad-hoc routing protocols such as DSR and AODV are the well known protocols that can be used for this purpose. The disadvantage of the reactive approach is that the route discovery process becomes costly in terms of control overhead and latency of the connection setup when done frequently. In this paper, an optimization of the simple AODV route discovery is proposed, introducing a paging extension. The proposed mechanism is somewhat similar to other paging schemes like Cellular IP, but introduces modifications and enhancements for reducing the paging control overhead. Simulation results of AODV and the enhanced route discovery with paging are presented along with the performance evaluation of both schemes. The simulations were performed with the QualNet Network Simulator.

This paper addresses channel allocation for multi-user OFDM systems for real-time packet oriented data transmission. For high data rate wireless communication, optimum utilization of resources has become of utmost importance. The optimum problem we are facing is the minimization of the total transmitted power while meeting the quality of service (QoS) and rate requirements in a multi-user OFDM transmission. The idea is to utilize the knowledge of rate requirements not only as a constraint, but also in the channel allocation process. We propose to include the rate requirements in a two-fold manner: i) to perform pre-calculation of the maximum number of channels per user and, ii) to define the set of users competing for the available channels. Thereby obtaining a much balanced allocation policy, that results in reduced total power at low complexity

Software radio technology provides a means to bridge the incompatibility between radio system across different communication bands as well as between systems working on different standards. As the quest for the common global standards has so far remained elusive especially in the domain of personal communication, software radio, with its software reconfigurability features promise affordable interpretability. In today's wireless communication systems, the physical layer processing its typically implemented as static design, providing an abstract interface to the upper layer of the system as simply a bit transmission medium with some level of uncertainty at the destination. Fixed hardware implementations reinforce this view of the physical layer as immutable. The paper deals with computer modelling and analysis of random sub-sampling for software radio receiver.

An attempt has been made to design a digital FIR filter for channel separation in a software radio receiver. The study has an impact on the computer aided design of software definable radio (SDR) for wireless communications.

The world of electronics is rapidly going digital and digital signal processing is the key to growth of the digital world. It impacts various facets of telecommunication technology and as the need for sophisticated signal processing algorithms and hardware increases, the potential of signal processing in the communication revolution appears unbounded. The paper deals with the implementation of the continuous phase frequency shift keying (FSK) modulator (V.23) on a TMS320C50 DSP chip.