The False Belief Test (FBT) has been the main method for assessing Theory of Mind (ToM) and related socio-cognitive competencies.For Large Language Models (LLMs), the reliability and explanatory potential of this test have remained limited due to issues like data contamination, insufficient model details, and inconsistent controls.We address these issues by testing 17 open-weight models on a balanced set of 192 FBT variants (Trott et al., 2023) using Bayesian Logistic regression to identify how model size and post-training affect socio-cognitive competence.We find that scaling model size benefits performance, but not strictly.A cross-over effect reveals that explicating propositional attitudes (X thinks) fundamentally alters response patterns.Instruction tuning partially mitigates this effect, but further reasoning-oriented finetuning amplifies it.In a case study analysing social reasoning ability throughout OLMo 2 training, we show that this cross-over effect emerges during pre-training, suggesting that models acquire stereotypical response patterns tied to mental-state vocabulary that can outweigh other scenario semantics.Finally, vector steering allows us to isolate a think vector as the causal driver of observed FBT behaviour.

Misinformation can be countered with factchecking, but the process is costly and slow.Identifying checkworthy claims is the first step, where automation can help scale fact-checkers' efforts.However, detection methods struggle with content that is (1) multimodal, (2) from diverse domains, and (3) synthetic.We introduce HINTSOFTRUTH, a public dataset for multimodal checkworthiness detection with 27K real-world and synthetic image/claim pairs.The mix of real and synthetic data makes this dataset unique and ideal for benchmarking detection methods.We compare fine-tuned and prompted Large Language Models (LLMs).We find that well-configured lightweight textbased encoders perform comparably to multimodal models but the former only focus on identifying non-claim-like content.Multimodal LLMs can be more accurate but come at a significant computational cost, making them impractical for large-scale applications.When faced with synthetic data, multimodal models perform more robustly.

Michiel van der Meer, Piek Vossen, Catholijn M. Jonker, Pradeep K. Murukannaiah. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Active Learning (AL) addresses the high costs of collecting human annotations by strategically annotating the most informative samples.However, for subjective NLP tasks, incorporating a wide range of perspectives in the annotation process is crucial to capture the variability in human judgments.We introduce Annotator-Centric Active Learning (ACAL), which incorporates an annotator selection strategy following data sampling.Our objective is two-fold:(1) to efficiently approximate the full diversity of human judgments, and (2) to assess model performance using annotator-centric metrics, which value minority and majority perspectives equally.We experiment with multiple annotator selection strategies across seven subjective NLP tasks, employing both traditional and novel, human-centered evaluation metrics.Our findings indicate that ACAL improves data efficiency and excels in annotator-centric performance evaluations.However, its success depends on the availability of a sufficiently large and diverse pool of annotators to sample from.

Large-scale survey tools enable the collection of citizen feedback in opinion corpora. Extracting the key arguments from a large and noisy set of opinions helps in understanding the opinions quickly and accurately. Fully automated methods can extract arguments but (1) require large labeled datasets that induce large annotation costs and (2) work well for known viewpoints, but not for novel points of view. We propose HyEnA, a hybrid (human + AI) method for extracting arguments from opinionated texts, combining the speed of automated processing with the understanding and reasoning capabilities of humans. We evaluate HyEnA on three citizen feedback corpora. We find that, on the one hand, HyEnA achieves higher coverage and precision than a state-of-the-art automated method when compared to a common set of diverse opinions, justifying the need for human insight. On the other hand, HyEnA requires less human effort and does not compromise quality compared to (fully manual) expert analysis, demonstrating the benefit of combining human and artificial intelligence.

Disagreements are common in online discussions.Disagreement may foster collaboration and improve the quality of a discussion under some conditions.Although there exist methods for recognizing disagreement, a deeper understanding of factors that influence disagreement is lacking in the literature.We investigate a hypothesis that differences in personal values are indicative of disagreement in online discussions.We show how state-of-the-art models can be used for estimating values in online discussions and how the estimated values can be aggregated into value profiles.We evaluate the estimated value profiles based on human-annotated agreement labels.We find that the dissimilarity of value profiles correlates with disagreement in specific cases.We also find that including value information in agreement prediction improves performance.

The pursuit of values drives human behavior and promotes cooperation. Existing research is focused on general values (e.g., Schwartz) that transcend contexts. However, context-specific values are necessary to (1) understand human decisions, and (2) engineer intelligent agents that can elicit and align with human values. We propose Axies, a hybrid (human and AI) methodology to identify context-specific values. Axies simplifies the abstract task of value identification as a guided value annotation process involving human annotators. Axies exploits the growing availability of value-laden text corpora and Natural Language Processing to assist the annotators in systematically identifying context-specific values. We evaluate Axies in a user study involving 80 human subjects. In our study, six annotators generate value lists for two timely and important contexts: Covid-19 measures and sustainable Energy. We employ two policy experts and 72 crowd workers to evaluate Axies value lists and compare them to a list of general (Schwartz) values. We find that Axies yields values that are (1) more context-specific than general values, (2) more suitable for value annotation than general values, and (3) independent of the people applying the methodology. Supplementary Information: The online version contains supplementary material available at 10.1007/s10458-022-09550-0.