Abstract Current evaluations of large language models (LLMs) rely heavily on a growing collection of benchmarks and on aggregate benchmark scores, yet it remains unclear what this comparison actually captures, and what these scores reveal about models’ underlying capabilities. Here, we propose a new paradigm for LLM evaluation, by asking whether benchmark performance reflects many independent abilities, or rather, relies on a small number of shared dimensions. To answer this, we apply Factor Analysis (FA) to a massive performance matrix of LLMs versus benchmarks (60 × 44) revealing an intrinsically low-rank structure of that matrix. That is, a small number of latent factors captures most of the structure in the full task space. This low-rank geometry reveals substantial redundancy across existing tasks and explains why many benchmarks appear to be measuring overlapping abilities. We further show that these latent factors correspond to coherent, skill-like, dimensions of LLM behavior. Leveraging this latent skill-space, we deliver three practical tools for LLM evaluation and downstream users: (i) identifying redundant tasks, (ii) profiling new models using a small subset of tasks, and (iii) selecting models aligned with desired skill profiles. Our method provides a solid alternative to the de-facto standard of a single aggregate score, and establishes an interpretable and practical framework for understanding and benchmarking LLM core capabilities.

Coherence is an essential property of wellwritten texts, that refers to the way textual units relate to one another.In the era of generative AI, coherence assessment is essential for many NLP tasks such as summarization, long-form question-answering, and more.Current NLP approaches for modeling coherence often rely on a proxy task, specifically, sentence reordering.However, such an approach may not capture the full range of factors contributing to coherence.To remedy this, in this work we employ the formal linguistic definition by Reinhart of what makes a discourse coherent, consisting of three conditions, cohesion, consistency and relevance, and formalize these conditions as respective computational tasks, which are in turn jointly trained.We evaluate this modeling approach on two human-rated coherence benchmarks: one of automatically-generated stories and one of real-world texts.Our experiments show that jointly training on the proposed tasks leads to better performance on each task compared with task-specific models, and to better performance on assessing coherence overall.Our proposed computational framework thus paves the way for a more advanced, broad-coverage coherence assessment.

Improvements in language models' capabilities have pushed their applications towards longer contexts, making long-context evaluation and development an active research area.However, many disparate use cases are grouped together under the umbrella term of "long-context", defined simply by the total length of the model's input, including -for example -Needle-in-a-Haystack tasks, book summarization, and information aggregation.Given their varied difficulty, in this position paper we argue that conflating different tasks by their context length is unproductive.As a community, we require a more precise vocabulary to understand what makes long-context tasks similar or different.We propose to unpack the taxonomy of longcontext based on the properties that make them more difficult with longer contexts.We propose two orthogonal axes of difficulty: (I) Dispersion: How hard is it to find the necessary information in the context?(II) Scope: How much necessary information is there to find?We survey the literature on long context, provide justification for this taxonomy as an informative descriptor, and situate the literature with respect to it.We conclude that the most difficult and interesting settings, whose necessary information is very long and highly dispersed within the input, is severely under-explored.By using a descriptive vocabulary and discussing the relevant properties of difficulty in long context, we can implement more informed research in this area.We call for a careful design of tasks and benchmarks with distinctly long context, taking into account the characteristics that make it qualitatively different from shorter context.

Imagine observing someone scratching their arm; to understand why, additional context would be necessary. However, spotting a mosquito nearby would immediately offer a likely explanation for the person’s discomfort, thereby alleviating the need for further information. This example illustrates how subtle visual cues can challenge our cognitive skills and demonstrates the complexity of interpreting visual scenarios. To study these skills, we present Visual Riddles, a benchmark aimed to test vision and language models on visual riddles requiring commonsense and world knowledge. The benchmark comprises 400 visual riddles, each featuring a unique image created by a variety of text-to-image models, question, ground-truth answer, textual hint, and attribution. Human evaluation reveals that existing models lag significantly behind human performance, which is at 82% accuracy, with Gemini-Pro-1.5 leading with 40% accuracy. Our benchmark comes with automatic evaluation tasks to make assessment scalable. These findings underscore the potential of Visual Riddles as a valuable resource for enhancing vision and language models’ capabilities in interpreting complex visual scenarios. Data, code, and leaderboard are available at https://visual-riddles.github.io/.

Coherence is a linguistic term that refers to the relations between small textual units (sentences, propositions), which make the text logically consistent and meaningful to the reader.With the advances of generative foundational models in NLP, there is a pressing need to automatically assess the human-perceived coherence of automatically generated texts.Up until now, little work has been done on explicitly assessing the coherence of generated texts and analyzing the factors contributing to (in)coherence.Previous work on the topic used other tasks, e.g., sentence reordering, as proxies of coherence, rather than approaching coherence detection heads on.In this paper, we introduce COHESENTIA, a novel benchmark of humanperceived coherence of automatically generated texts.Our annotation protocol reflects two perspectives; one is global, assigning a single coherence score, and the other is incremental, scoring sentence by sentence.The incremental method produces an (in)coherence score for each text fragment and also pinpoints reasons for incoherence at that point.Our benchmark contains 500 automatically-generated and human-annotated paragraphs, each annotated in both methods, by multiple raters.Our analysis shows that the inter-annotator agreement in the incremental mode is higher than in the holistic alternative, and our experiments show that standard LMs fine-tuned for coherence detection show varied performance on the different factors contributing to (in)coherence.All in all, these models yield unsatisfactory performance, emphasizing the need for developing more reliable methods for coherence assessment.