The "LLM-as-a-judge" paradigm has become a standard method for evaluating open-ended generation.To address the quadratic scalability costs of pairwise comparisons, popular benchmarks like Arena-Hard and AlpacaEval compare all models against a single anchor.However, despite its widespread use, the impact of anchor selection on the reliability of the results remains largely unexplored.In this work, we systematically investigate the effect of anchor selection by evaluating 22 different anchors on the Arena-Hard-v2.0 dataset.We find that the choice of anchor is critical: a poor anchor can dramatically reduce correlation with human rankings.We identify that common anchor choices (best-performing and worst-performing models) make poor anchors.Because these extreme anchors are consistently better or worse than all other models, they are seldom indicative of the relative ranking of the models.We further quantify the effect size of anchor selection, showing it is comparable to the selection of a judge model.We conclude with actionable recommendations.First, we conduct a power analysis, and compute sufficient benchmark sizes for anchor-based evaluation, finding that standard benchmark sizes are insufficient for pairwise evaluation and fail to distinguish between competitive models reliably.Second, we provide guidelines for selecting informative anchors to ensure reliable and efficient evaluation practices.

We propose a new paradigm to continually evolve pretrained models, denoted ColD Fusion.It provides the benefits of multitask learning but leverages distributed computation with limited communication and eliminates the need for shared data.Consequentially, ColD Fusion can give rise to a synergistic loop, where finetuned models can be recycled to continually improve the pretrained model they are based upon.We show that ColD Fusion yields comparable benefits to multitask training by producing a model that (a) attains strong performance on all of the datasets it was trained on; and (b) is a better starting point for finetuning on unseen datasets.We show that ColD Fusion outperforms RoBERTa and even previous multitask models.Specifically, when training and testing on 35 diverse datasets, ColD Fusion-based model outperforms RoBERTa by 2.33 points on average without any changes to the architecture.1

Previous studies observed that finetuned models may be better base models than the vanilla pretrained model.Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset.Here, we perform a systematic analysis of this intertraining scheme, over a wide range of English classification tasks.Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed independently for the target dataset under consideration, and for a base model being considered as a starting point.Hence, a performant model is generally strong, even if its training data was not aligned with the target dataset.Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings.Last, we release an updating ranking of best models in the HuggingFace hub per architecture.

Generating images with a Text-to-Image model often requires multiple trials, where human users iteratively update their prompt based on feedback, namely the output image.Taking inspiration from cognitive work on reference games and dialogue alignment, this paper analyzes the dynamics of the user prompts along such iterations.We compile a dataset of iterative interactions of human users with Midjourney. 1 Our analysis then reveals that prompts predictably converge toward specific traits along these iterations.We further study whether this convergence is due to human users, realizing they missed important details, or due to adaptation to the model's "preferences", producing better images for a specific language style.We show initial evidence that both possibilities are at play.The possibility that users adapt to the model's preference raises concerns about reusing user data for further training.The prompts may be biased towards the preferences of a specific model, rather than align with human intentions and natural manner of expression.

We present the task of PreQuEL, Pre-(Quality-Estimation) Learning.A PreQuEL system predicts how well a given sentence will be translated, without recourse to the actual translation, thus eschewing unnecessary resource allocation when translation quality is bound to be low.PreQuEL can be defined relative to a given MT system (e.g., some industry service) or generally relative to the state-of-theart.From a theoretical perspective, PreQuEL places the focus on the source text, tracing properties, possibly linguistic features, that make a sentence harder to machine translate.We develop a baseline model for the task and analyze its performance.We also develop a data augmentation method (from parallel corpora), that improves results substantially.We show that this augmentation method can improve the performance of the Quality-Estimation task as well. 1 We investigate the properties of the input text that our model is sensitive to, by testing it on challenge sets and different languages.We conclude that it is aware of syntactic and semantic distinctions, and correlates and even over-emphasizes the importance of standard NLP features.cross-lingual representation learning at scale.