Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, Daphne Ippolito. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Large language models achieve high performance on many but not all downstream tasks.The interaction between pretraining data and task data is commonly assumed to determine this variance: a task with data that is more similar to a model's pretraining data is assumed to be easier for that model.We test whether distributional and example-specific similarity measures (embedding-, token-and model-based) correlate with language model performance through a large-scale comparison of the Pile and C4 pretraining datasets with downstream benchmarks.Similarity correlates with performance for multilingual datasets, but in other benchmarks, we surprisingly find that similarity metrics are not correlated with accuracy or even each other.This suggests that the relationship between pretraining data and downstream tasks is more complex than often assumed.

Much of the progress in contemporary NLP has come from learning representations, such as masked language model (MLM) contextual embeddings, that turn challenging problems into simple classification tasks.But how do we quantify and explain this effect?We adapt general tools from computational learning theory to fit the specific characteristics of text datasets and present a method to evaluate the compatibility between representations and tasks.Even though many tasks can be easily solved with simple bag-of-words (BOW) representations, BOW does poorly on hard natural language inference tasks.For one such task we find that BOW cannot distinguish between real and randomized labelings, while pre-trained MLM representations show 72x greater distinction between real and random labelings than BOW.This method provides a calibrated, quantitative measure of the difficulty of a classificationbased NLP task, enabling comparisons between representations without requiring empirical evaluations that may be sensitive to initializations and hyperparameters.The method provides a fresh perspective on the patterns in a dataset and the alignment of those patterns with specific labels.

Images can give us insights into the contextual meanings of words, but current imagetext grounding approaches require detailed annotations.Such granular annotation is rare, expensive, and unavailable in most domainspecific contexts.In contrast, unlabeled multiimage, multi-sentence documents are abundant.Can lexical grounding be learned from such documents, even though they have significant lexical and visual overlap?Working with a case study dataset of real estate listings, we demonstrate the challenge of distinguishing highly correlated grounded terms, such as "kitchen" and "bedroom", and introduce metrics to assess this document similarity.We present a simple unsupervised clusteringbased method that increases precision and recall beyond object detection and image tagging baselines when evaluated on labeled subsets of the dataset.The proposed method is particularly effective for local contextual meanings of a word, for example associating "granite" with countertops in the real estate dataset and with rocky landscapes in a Wikipedia dataset."kitchen" (18.4% labeled true) E: 72.9 AUC W: 52.7 AUC R: 21.1 AUC "outdoor" (16.9% labeled true) E: 68.5 AUC W: 20.0 AUC R: 13.2 AUC "washer" (1.6% labeled true)

Histopathology tissue samples are widely available in two states: paraffin-embedded unstained and non-paraffin-embedded stained whole slide RGB images (WSRI). Hematoxylin and eosin stain (H&E) is one of the principal stains in histology but suffers from several shortcomings related to tissue preparation, staining protocols, slowness and human error. We report two novel approaches for training machine learning models for the computational H&E staining and destaining of prostate core biopsy RGB images. The staining model uses a conditional generative adversarial network that learns hierarchical non-linear mappings between whole slide RGB image (WSRI) pairs of prostate core biopsy before and after H&E staining. The trained staining model can then generate computationally H&E-stained prostate core WSRIs using previously unseen non-stained biopsy images as input. The destaining model, by learning mappings between an H&E stained WSRI and a non-stained WSRI of the same biopsy, can computationally destain previously unseen H&E-stained images. Structural and anatomical details of prostate tissue and colors, shapes, geometries, locations of nuclei, stroma, vessels, glands and other cellular components were generated by both models with structural similarity indices of 0.68 (staining) and 0.84 (destaining). The proposed staining and destaining models can engender computational H&E staining and destaining of WSRI biopsies without additional equipment and devices.

Fluorescent biomarkers are important indicators of disease, but imaging them can require specialized and often-expensive devices. Periodontal and dental diseases resulting from microbial plaque biofilms, if diagnosed early with biomarker images and expert knowledge, can be treated to prevent occurrences of serious systemic illnesses. We report two convolutional neural network classifiers trained with dentist annotations of disease signatures and fluorescent porphyrin biomarker images to identify dental plaque in white light images as a per-pixel binary classification task. The classifiers were trained and tested with millions of image patches from two datasets collected from 27 consenting adults using handheld intraoral cameras. The areas under the receiver operating characteristic curves for the test sets were calculated to be 0.7694 and 0.8720. Once trained, the classifiers predict the location of plaque in white light images without requiring specialized biomarker imaging devices or expert intervention. This generalized approach can be useful in other domains where diagnostic biomarker predicting can augment expert knowledge using standard white light images.