Large language models (LLMs) are increasingly deployed with hierarchical instruction schemes, where certain instructions (e.g., system-level directives) are expected to take precedence over others (e.g., user messages). Yet, we lack a systematic understanding of how effectively these hierarchical control mechanisms work. We introduce a systematic evaluation framework based on constraint prioritization to assess how well LLMs enforce instruction hierarchies. Our experiments across six state-of-the-art LLMs reveal that models struggle with consistent instruction prioritization, even for simple formatting conflicts. We find that the widely-adopted system/user prompt separation fails to establish a reliable instruction hierarchy, and models exhibit strong inherent biases toward certain constraint types regardless of their priority designation. Interestingly, we also find that societal hierarchy framings (e.g., authority, expertise, consensus) show stronger influence on model behavior than system/user roles, suggesting that pretraining-derived social structures function as latent behavioral priors with potentially greater impact than post-training guardrails.

Measuring the quality of public deliberation requires evaluating not only civility or argument structure, but also the informational progress of a conversation.We introduce a framework for Conversational Information Gain (CIG) that evaluates each utterance in terms of how it advances collective understanding of the target topic.To operationalize CIG, we model an evolving semantic memory of the discussion: the system extracts atomic claims from utterances and incrementally consolidates them into a structured memory state.Using this memory, we score each utterance along three interpretable dimensions: Novelty, Relevance, and Implication Scope.We annotate 80 segments from two moderated deliberative settings (TV debates and community discussions) with these dimensions and show that memory-derived dynamics (e.g., the number of claim updates) correlate more strongly with human-perceived CIG than traditional heuristics such as utterance length or TF-IDF.We develop effective LLM-based CIG predictors paving the way for information-focused conversation quality analysis in dialogues and deliberative success. 1

Yanbei Jiang, Amr Keleg, Ryandito Diandaru, Jey Han Lau, Lea Frermann, Biaoyan Fang, Fajri Koto. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Abstract When a news article describes immigration as an “economic burden” or a “humanitarian crisis,” it selectively emphasizes certain aspects of the issue. Although this framing shapes how the public interprets such issues, audiences do not absorb frames passively but actively reorganize the presented information. While this relationship between source content and audience response is well-documented in the social sciences, NLP approaches often ignore it, analyzing frames in articles and responses in isolation. We present the first computational framework for large-scale analysis of framing across source content (news articles) and audience responses (reader comments). Methodologically, we refine frame labels and develop a framework that reconstructs primary frames in articles and comments from sentence-level predictions, and aligns articles with topically relevant comments. Applying our framework across eleven topics and two news outlets, we find that frame reuse in comments correlates highly across outlets, and that readers often selectively engage with frames of the articles. We release a frame classifier that performs well on both articles and comments, a dataset of article and comment sentences manually labeled for frames, and a large-scale dataset of articles and comments with predicted frame labels.1

As the impact of large language models increases, understanding the moral values they reflect becomes ever more important.Assessing the nature of moral values as understood by these models via direct prompting is challenging due to potential leakage of human norms into model training data, and their sensitivity to prompt formulation.Instead, we propose to use word associations, which have been shown to reflect moral reasoning in humans, as lowlevel underlying representations to obtain a more robust picture of LLMs' moral reasoning.We study moral differences in associations from western English-speaking communities and LLMs trained predominantly on English data.First, we create a large dataset of LLMgenerated word associations, resembling an existing data set of human word associations.Next, we propose a novel method to propagate moral values based on seed words derived from Moral Foundation Theory through the human and LLM-generated association graphs.Finally, we compare the resulting moral conceptualizations, highlighting detailed but systematic differences between moral values emerging from English speakers and LLM associations. 1

Human Interest (HI) framing is a narrative strategy that injects news stories with a relatable, emotional angle and a human face to engage the audience. In this study we investigate the use of HI framing across different English-speaking cultures in news articles about climate change. Despite its demonstrated impact on the public’s behaviour and perception of an issue, HI framing has been under-explored in NLP to date. We perform a systematic analysis of HI stories to understand its role in climate change reporting in English-speaking countries from four continents. Our findings reveal key differences in how climate change is portrayed across countries, encompassing aspects such as narrative roles, article polarity, pronoun prevalence, and topics. We also demonstrate that these linguistic aspects boost the performance of fine-tuned pre-trained language models on HI story classification.

Ming-Bin Chen, Lea Frermann, Jey Han Lau. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Abstract The task of image captioning has recently been gaining popularity, and with it the complex task of evaluating the quality of image captioning models. In this work, we present the first survey and taxonomy of over 70 different image captioning metrics and their usage in hundreds of papers, specifically designed to help users select the most suitable metric for their needs. We find that despite the diversity of proposed metrics, the vast majority of studies rely on only five popular metrics, which we show to be weakly correlated with human ratings. We hypothesize that combining a diverse set of metrics can enhance correlation with human ratings. As an initial step, we demonstrate that a linear regression-based ensemble method, which we call EnsembEval, trained on one human ratings dataset, achieves improved correlation across five additional datasets, showing there is a lot of room for improvement by leveraging a diverse set of metrics.1

Framing studies how individuals and societies make sense of the world, by communicating or representing complex issues through schema of interpretation.The framing of information in the mass media influences our interpretation of facts and corresponding decisions, so detecting and analysing it is essential to understand biases in the information we consume.Despite that, framing is still mostly examined manually, on a case-by-case basis, while existing largescale automatic analyses using NLP methods are not mature enough to solve this task.In this survey we show that despite the growing interest to framing in NLP its current approaches do identify aspects related to the framing of, rather than simply conveying, the message.To this end, we bring together definitions of frames and framing adopted in different disciplines; examine cognitive, linguistic, and communicative aspects a frame contains beyond its topical content.We survey recent work on computational frame detection, and discuss how framing aspects and frame definitions are (or should) be reflected in NLP approaches. 1

Despite increasing interest in the automatic detection of media frames in NLP, the problem is typically simplified as single-label classification and adopts a topic-like view on frames, evading modelling the broader document-level narrative.In this work, we revisit a widely used conceptualization of framing from the communication sciences which explicitly captures elements of narratives, including conflict and its resolution, and integrate it with the narrative framing of key entities in the story as heroes, victims or villains.We adapt an effective annotation paradigm that breaks a complex annotation task into a series of simpler binary questions, and present an annotated data set of English news articles, and a case study on the framing of climate change in articles from news outlets across the political spectrum.Finally, we explore automatic multi-label prediction of our frames with supervised and semisupervised approaches, and present a novel retrieval-based method which is both effective and transparent in its predictions.We conclude with a discussion of opportunities and challenges for future work on document-level models of narrative framing. 1

Subtle changes in word choice in communication can evoke very different associations with the involved actors.For instance, a company 'employing workers' evokes a more positive connotation than the one 'exploiting' them.This concept is called connotation.This paper investigates whether pre-trained language models (PLMs) encode such subtle connotative information about power differentials between involved entities.We design a probing framework for power connotation, building on Sap et al. (2017)'s operationalization of connotation frames.We show that zero-shot prompting of PLMs leads to above chance prediction of power connotation, however fine-tuning PLMs using our framework drastically improves their accuracy.Using our fine-tuned models, we present a case study of power dynamics in US news reporting on immigration, showing the potential of our framework as a tool for understanding subtle bias in the media. 1

Uri Berger, Gabriel Stanovsky, Omri Abend, Lea Frermann. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Kemal Kurniawan, Lea Frermann, Philip Schulz, Trevor Cohn. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Aili Shen, Xudong Han, Trevor Cohn, Timothy Baldwin, Lea Frermann. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Humans use countless basic, shared facts about the world to efficiently navigate in their environment.This commonsense knowledge is rarely communicated explicitly, however, understanding how commonsense knowledge is represented in different paradigms is important for both deeper understanding of human cognition and for augmenting automatic reasoning systems.This paper presents an in-depth comparison of two large-scale resources of general knowledge: ConceptNet, an engineered relational database, and SWOW a knowledge graph derived from crowd-sourced word associations.We examine the structure, overlap and differences between the two graphs, as well as the extent to which they encode situational commonsense knowledge.We finally show empirically that both resources improve downstream task performance on commonsense reasoning benchmarks over text-only baselines, suggesting that large-scale word association data, which have been obtained for several languages through crowd-sourcing, can be a valuable complement to curated knowledge graphs. 1

Cross-lingual transfer is a leading technique for parsing low-resource languages in the absence of explicit supervision.Simple 'direct transfer' of a learned model based on a multilingual input encoding has provided a strong benchmark.This paper presents a method for unsupervised cross-lingual transfer that improves over direct transfer systems by using their output as implicit supervision as part of self-training on unlabelled text in the target language.The method assumes minimal resources and provides maximal flexibility by (a) accepting any pre-trained arc-factored dependency parser; (b) assuming no access to source language data; (c) supporting both projective and non-projective parsing; and (d) supporting multi-source transfer.With English as the source language, we show significant improvements over state-of-the-art transfer models on both distant and nearby languages, despite our conceptually simpler approach.We provide analyses of the choice of source languages for multi-source transfer, and the advantage of non-projective parsing.Our code is available online. 1

Class imbalance is a common challenge in many NLP tasks, and has clear connections to bias, in that bias in training data often leads to higher accuracy for majority groups at the expense of minority groups.However there has traditionally been a disconnect between research on class-imbalanced learning and mitigating bias, and only recently have the two been looked at through a common lens.In this work we evaluate long-tail learning methods for tweet sentiment and occupation classification, and extend a margin-loss based approach with methods to enforce fairness.We empirically show through controlled experiments that the proposed approaches help mitigate both class imbalance and demographic biases. 1

Bias is pervasive in NLP models, motivating the development of automatic debiasing techniques.Evaluation of NLP debiasing methods has largely been limited to binary attributes in isolation, e.g., debiasing with respect to binary gender or race, however many corpora involve multiple such attributes, possibly with higher cardinality.In this paper we argue that a truly fair model must consider 'gerrymandering' groups which comprise not only single attributes, but also intersectional groups.We evaluate a form of bias-constrained model which is new to NLP, as well an extension of the iterative nullspace projection technique which can handle multiple protected attributes.

Shima Khanehzar, Trevor Cohn, Gosia Mikolajczak, Andrew Turpin, Lea Frermann. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Most general-purpose extractive summarization models are trained on news articles, which are short and present all important information upfront.As a result, such models are biased by position and often perform a smart selection of sentences from the beginning of the document.When summarizing long narratives, which have complex structure and present information piecemeal, simple position heuristics are not sufficient.In this paper, we propose to explicitly incorporate the underlying structure of narratives into general unsupervised and supervised extractive summarization models.We formalize narrative structure in terms of key narrative events (turning points) and treat it as latent in order to summarize screenplays (i.e., extract an optimal sequence of scenes).Experimental results on the CSI corpus of TV screenplays, which we augment with scene-level summarization labels, show that latent turning points correlate with important aspects of a CSI episode and improve summarization performance over general extractive algorithms, leading to more complete and diverse summaries.Victim: Mike Kimble, found in a Body Farm.Died 6 hours ago, unknown cause of death.CSI discover cow tissue in Mike's body.Cross-contamination is suggested.Probable cause of death: Mike's house has been set on fire.CSI finds blood: Mike was murdered, fire was a cover up.First suspects: Mike's fiance, Jane and her ex-husband, Russ.CSI finds photos in Mike's house of Jane's daughter, Jodie, posing naked.Mike is now a suspect of abusing Jodie.Russ allows CSI to examine his gun.CSI discovers that the bullet that killed Mike was made of frozen beef that melt inside him.They also find beef in Russ' gun.Russ confesses that he knew that Mike was abusing Jody, so he confronted and killed him.CSI discovers that the naked photos were taken on a boat, which belongs to Russ.CSI discovers that it was Russ who was abusing his daughter based on fluids found in his sleeping bag and later killed Mike who tried to help Jodie.Russ is given bail, since no jury would convict a protective father.Russ receives a mandatory life sentence.

Automatic summarization is typically treated as a 1-to-1 mapping from document to summary.Documents such as news articles, however, are structured and often cover multiple topics or aspects; and readers may be interested in only some of them.We tackle the task of aspect-based summarization, where, given a document and a target aspect, our models generate a summary centered around the aspect.We induce latent document structure jointly with an abstractive summarization objective, and train our models in a scalable synthetic setup.In addition to improvements in summarization over topic-agnostic baselines, we demonstrate the benefit of the learnt document structure: we show that our models (a) learn to accurately segment documents by aspect; (b) can leverage the structure to produce both abstractive and extractive aspectbased summaries; and (c) that structure is particularly advantageous for summarizing long documents.All results transfer from synthetic training documents to natural news articles from CNN/Daily Mail and RCV1.

Nikos Papasarantopoulos, Lea Frermann, Mirella Lapata, Shay B. Cohen. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Maria Barrett, Ana Valeria González-Garduño, Lea Frermann, Anders Søgaard. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

In this paper we argue that crime drama exemplified in television programs such as CSI: Crime Scene Investigation is an ideal testbed for approximating real-world natural language understanding and the complex inferences associated with it. We propose to treat crime drama as a new inference task, capitalizing on the fact that each episode poses the same basic question (i.e., who committed the crime) and naturally provides the answer when the perpetrator is revealed. We develop a new dataset based on CSI episodes, formalize perpetrator identification as a sequence labeling problem, and develop an LSTM-based model which learns from multi-modal data. Experimental results show that an incremental inference strategy is key to making accurate guesses as well as learning from representations fusing textual, visual, and acoustic input.

Automatically understanding the plot of novels is important both for informing literary scholarship and applications such as summarization or recommendation.Various models have addressed this task, but their evaluation has remained largely intrinsic and qualitative.Here, we propose a principled and scalable framework leveraging expert-provided semantic tags (e.g., mystery, pirates) to evaluate plot representations in an extrinsic fashion, assessing their ability to produce locally coherent groupings of novels (micro-clusters) in model space.We present a deep recurrent autoencoder model that learns richly structured multi-view plot representations, and show that they i) yield better microclusters than less structured representations; and ii) are interpretable, and thus useful for further literary analysis or labelling of the emerging micro-clusters.

Word meanings change over time and an automated procedure for extracting this information from text would be useful for historical exploratory studies, information retrieval or question answering. We present a dynamic Bayesian model of diachronic meaning change, which infers temporal word representations as a set of senses and their prevalence. Unlike previous work, we explicitly model language change as a smooth, gradual process. We experimentally show that this modeling decision is beneficial: our model performs competitively on meaning change detection tasks whilst inducing discernible word senses and their development over time. Application of our model to the SemEval-2015 temporal classification benchmark datasets further reveals that it performs on par with highly optimized task-specific systems.

Categories such as ANIMAL or FURNITURE are acquired at an early age and play an important role in processing, organizing, and conveying world knowledge.Theories of categorization largely agree that categories are characterized by features such as function or appearance and that feature and category acquisition go hand-in-hand, however previous work has considered these problems in isolation.We present the first model that jointly learns categories and their features.The set of features is shared across categories, and strength of association is inferred in a Bayesian framework.We approximate the learning environment with natural language text which allows us to evaluate performance on a large scale.Compared to highly engineered pattern-based approaches, our model is cognitively motivated, knowledge-lean, and learns categories and features which are perceived by humans as more meaningful.

Models of category learning have been extensively studied in cognitive science and primarily tested on perceptual abstractions or artificial stimuli. In this paper we focus on categories acquired from natural language stimuli, that is words (e.g., chair is a member of the FURNITURE category). We present a Bayesian model which, unlike previous work, learns both categories and their features in a single process. Our model employs particle filters, a sequential Monte Carlo method commonly used for approximate probabilistic inference in an incremental setting. Comparison against a state-of-the-art graph-based approach reveals that our model learns qualitatively better categories and demonstrates cognitive plausibility during learning.

Scripts representing common sense knowledge about stereotyped sequences of events have been shown to be a valuable resource for NLP applications.We present a hierarchical Bayesian model for unsupervised learning of script knowledge from crowdsourced descriptions of human activities.Events and constraints on event ordering are induced jointly in one unified framework.We use a statistical model over permutations which captures event ordering constraints in a more flexible way than previous approaches.In order to alleviate the sparsity problem caused by using relatively small datasets, we incorporate in our hierarchical model an informed prior on word distributions.The resulting model substantially outperforms a state-of-the-art method on the event ordering task.