Prior work evaluating emotion and affective understanding in large language models (LLMs) typically rely on predetermined label sets or focus on a singular evaluation task (e.g., emotion detection).We consider affective states, referring to the much broader variety of terms people use to label their emotional experiences.We evaluate multilingual language models' understanding of affective states in English and Spanish through three different tasks: 1) identification, where models predict an affective state given text, 2) expression, where models generate text expressing a given affective state, and 3) verification, where models report whether a given term refers to an affective state.We show that performance on one task is not necessarily predictive of performance on another.Using these three tasks, we then begin to explore when and why models struggle to understand particular affective states compared to others.We examine systematic patterns in the affective state terms that are well and poorly understood by models, characterizing the working emotion vocabulary of LLMs.

Nicholas Deas, Blake Vente, Amith Ananthram, Jessica A Grieser, Desmond U. Patton, Shana Kleiner, James R. Shepard Iii, Kathleen McKeown. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Global partisan hostility and polarization has increased, and this polarization is heightened around presidential elections. Models capable of generating accurate summaries of diverse perspectives can help reduce such polarization by exposing users to alternative perspectives. In this work, we introduce a novel dataset and task for independently summarizing each political perspective in a set of passages from opinionated news articles. For this task, we propose a framework for evaluating different dimensions of perspective summary performance. We benchmark 11 summarization models and LLMs of varying sizes and architectures through both automatic and human evaluation. While recent models like GPT-4o perform well on this task, we find that all models struggle to generate summaries that are faithful to the intended perspective. Our analysis of summaries focuses on how extraction behavior is impacted by features of the input documents.

We introduce and study artificial impressionspatterns in LLMs' internal representations of prompts that resemble human impressions and stereotypes based on language.We fit linear probes on generated prompts to predict impressions according to the two-dimensional Stereotype Content Model (SCM).Using these probes, we study the relationship between impressions and downstream model behavior as well as prompt features that may inform such impressions.We find that LLMs inconsistently report impressions when prompted, but also that impressions are more consistently linearly decodable from their hidden representations.Additionally, we show that artificial impressions of prompts are predictive of the quality and use of hedging in model responses.We also investigate how particular content, stylistic, and dialectal features in prompts impact LLM impressions.

LLMs can be unpredictable, as even slight alterations to the prompt can cause the output to change in unexpected ways. Thus, the ability of models to accurately explain their behavior is critical, especially in high-stakes settings. Counterfactual simulatability measures how well an explanation allows users to infer the model’s output on related counterfactuals and has been previously studied for yes/no question answering. We provide a general framework for extending this method to generation tasks, using news summarization and medical suggestion as example use cases. We find that while LLM explanations do enable users to better predict their outputs on counterfactuals in the summarization setting, there is significant room for improvement for medical suggestion. Furthermore, our results suggest that evaluating counterfactual simulatability may be more appropriate for skill-based tasks as opposed to knowledge-based tasks.

In the field of emotion analysis, much NLP research focuses on identifying a limited number of discrete emotion categories, often applied across languages.These basic sets, however, are rarely designed with textual data in mind, and culture, language, and dialect can influence how particular emotions are interpreted.In this work, we broaden our scope to a practically unbounded set of affective states, which includes any terms that humans use to describe their experiences of feeling.We collect and publish MASIVE, a dataset of Reddit posts in English and Spanish containing over 1,000 unique affective states each.We then define the new problem of affective state identification for language generation models framed as a masked span prediction task.On this task, we find that smaller fine-tuned multilingual models outperform much larger LLMs, even on region-specific Spanish affective states.Additionally, we show that pre-training on MA-SIVE improves model performance on existing emotion benchmarks.Finally, through machine translation experiments, we find that native speaker-written data is vital to good performance on this task.

Warning: This paper contains content and language that may be considered offensive to some readers.While biases disadvantaging African American Language (AAL) have been uncovered in models for tasks such as speech recognition and toxicity detection, there has been little investigation of these biases for language generation models like ChatGPT.We evaluate how well LLMs understand AAL in comparison to White Mainstream English (WME), the encouraged "standard" form of English taught in American classrooms.We measure large language model performance on two tasks: a counterpart generation task, where a model generates AAL given WME and vice versa, as well as a masked span prediction (MSP) task, where models predict a phrase hidden from their input.Using a novel dataset of AAL texts from a variety of regions and contexts, we present evidence of dialectal bias for six pre-trained LLMs through performance gaps on these tasks.

Operational medical environments require reliable hands-free solutions to extract data from audio captured under noisy scenarios during rescue missions and provide timely information. However, approaches using automatic speech recognition (ASR) and natural language processing (NLP) techniques are complex as these conversations have a wide range of noise, involve medical terms from multiple speakers, and occur in high-stress environments, among others. These are further complicated by the lack of large training datasets for operational medical scenarios. To address these issues, we developed a platform that enables resilient hands-free data collection, preserves complete documentation through stages of care, and presents the information in near real-time, critical for the medical operation. Our work uniquely focused on systematic evaluation and improvement of a deep neural network-based ASR system by leveraging realistic testing data obtained from medical simulations of battlefield scenarios, which to our knowledge have not been addressed in any prior work. The system performance is shown to improve significantly using multi-style training, language model adaptation for the medical domain, speech enhancement, and NLP techniques.