Personalization of LLMs by sociodemographic subgroup often improves user experience, but can also introduce or amplify biases and unfair outcomes across groups.Prior work has employed so-called personas, sociodemographic user attributes conveyed to a model, to study bias in LLMs by relying on a single cue to prompt a persona, such as user names or explicit attribute mentions.This disregards LLM sensitivity to prompt variation and the rarity of some cues in real interactions (external validity).We compare six commonly used persona cues across seven open and proprietary LLMs on four writing and advice tasks.While cues are overall highly correlated, they produce substantial variance in responses across personas that can change findings on persona-induced differences and bias.We therefore caution against claims based on single persona cues, especially when they are overly explicit and have low external validity.

TikTok has emerged as a leading social media platform with increasing relevance for the consumption and distribution of news, especially for younger age groups. Despite its growing relevance, analyses of how traditional news outlets produce content for the platform and maintain journalistic news values are limited. Moreover, there are few large-scale datasets that are suitable for tracing larger journalistic trends and developing approaches for the automated annotation of multimodal content. This paper addresses these gaps and introduces “News on TikTok,” an annotated dataset of 8,623 TikTok videos published by 18 major German-speaking news outlets in 2023. Combining metadata with human-annotated data of 25 variables (incl. the presence of visual, auditory, and interactive elements, and journalistic news values), our dataset makes three significant contributions: First, it enables extensive analyses of news characteristics on TikTok. Second, it provides ground truth data to develop and validate automated tools for multimodal content analyses. Third, it offers a comprehensive guide for generating datasets with similar research interests.

Evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment. Reliably measuring abstract and complex phenomena such as safety' androbustness' requires strong construct validity, that is, having measures that represent what matters to the phenomenon. With a team of 29 expert reviewers, we conduct a systematic review of 445 LLM benchmarks from leading conferences in natural language processing and machine learning. Across the reviewed articles, we find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims. To address these shortcomings, we provide eight key recommendations and detailed actionable guidance to researchers and practitioners in developing LLM benchmarks.