Large language models (LLMs) and small language models (SLMs) are being adopted at remarkable speed, although their safety still remains a serious concern. With the advent of multilingual S/LLMs, the question now becomes a matter of scale: can we expand multilingual safety evaluations of these models with the same velocity at which they are deployed? To this end, we introduce RTP-LX, a human-transcreated and human-annotated corpus of toxic prompts and outputs in 28 languages. RTP-LX follows participatory design practices, and a portion of the corpus is especially designed to detect culturally-specific toxic language. We evaluate 10 S/LLMs on their ability to detect toxic content in a culturally-sensitive, multilingual scenario. We find that, although they typically score acceptably in terms of accuracy, they have low agreement with human judges when scoring holistically the toxicity of a prompt; and have difficulty discerning harm in context-dependent scenarios, particularly with subtle-yet-harmful content (e.g. microaggressions, bias). We release this dataset to contribute to further reduce harmful uses of these models and improve their safe deployment.

Fangru Lin, Shaoguang Mao, Emanuele La Malfa, Valentin Hofmann, Adrian de Wynter, Xun Wang, Si-Qing Chen, Michael J. Wooldridge, Janet B. Pierrehumbert, Furu Wei. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Warning: this paper discusses content related, but not limited to, violence, sex, and suicide.Loneliness, or the lack of fulfilling relationships, significantly impacts a person's mental and physical well-being and is prevalent worldwide.Previous research suggests that large language models (LLMs) may help mitigate loneliness.However, we argue that the use of widespread LLMs in services like ChatGPT is more prevalent-and riskier, as they are not designed for this purpose.To explore this, we analysed user interactions with ChatGPT outside of its marketed use as a task-oriented assistant.In dialogues classified as lonely, users frequently (37%) sought advice or validation, and received good engagement.However, Chat-GPT failed in sensitive scenarios, like responding appropriately to suicidal ideation or trauma.We also observed a 35% higher incidence of toxic content, with women being 22× more likely to be targeted than men.Our findings underscore ethical and legal questions about this technology, and note risks like radicalisation or further isolation.We conclude with recommendations to research and industry to address loneliness.

Misgendering is the act of referring to someone by a gender that does not match their chosen identity.It marginalizes and undermines a person's sense of self, causing significant harm.English-based approaches have clear-cut approaches to avoiding misgendering, such as the use of the pronoun "they".However, other languages pose unique challenges due to both grammatical and cultural constructs.In this work we develop methodologies to assess and mitigate misgendering across 42 languages and dialects using a participatory-design approach to design effective and appropriate guardrails across all languages.We test these guardrails in a standard LLM-based application (meeting transcript summarization), where both the data generation and the annotation steps followed a human-in-the-loop approach.We find that the proposed guardrails are very effective in reducing misgendering rates across all languages in the summaries generated, and without incurring loss of quality.Our human-in-the-loop approach demonstrates a method to feasibly scale inclusive and responsible AI-based solutions across multiple languages and cultures.We release the guardrails and synthetic dataset encompassing 42 languages, along with human and LLM-judge evaluations, to encourage further research on this subject.

We show that GPT-4’s reasoning and planning capabilities extend to the 1993 first-person shooterDoom. This large language model (LLM) is able to run and play the game with only a few instructions, plus a textual description–generated by the model itself from screenshots–about the state of the game being observed. We find that GPT-4 can play the game to a passable degree: it is able to manipulate doors, combat enemies, and perform pathing. More complex prompting strategies involving multiple model calls provide better results. While further work is required to enable the LLM to play the game as well as its classical, reinforcement learning-based counterparts, we note that GPT-4 required no training, leaning instead on its own reasoning and observational capabilities. We hope our work pushes the boundaries on intelligent, LLM-based agents in video games. We conclude by discussing the ethical implications of our work.

We evaluate two large language models (LLMs) ability to perform argumentative reasoning. We experiment with argument mining (AM) and argument pair extraction (APE), and evaluate the LLMs’ ability to recognize arguments under progressively more abstract input and output (I/O) representations (e.g., arbitrary label sets, graphs, etc.). Unlike the well-known evaluation of prompt phrasings, abstraction evaluation retains the prompt’s phrasing but tests reasoning capabilities. We find that scoring-wise the LLMs match or surpass the SOTA in AM and APE, and under certain I/O abstractions LLMs perform well, even beating chain-of-thought–we call this symbolic prompting. However, statistical analysis on the LLMs outputs when subject to small, yet still human-readable, alterations in the I/O representations (e.g., asking for BIO tags as opposed to line numbers) showed that the models are not performing reasoning. This suggests that LLM applications to some tasks, such as data labelling and paper reviewing, must be done with care.

We prove that three strategy video games from theSid Meier's Civilizationseries:Sid Meier's Civilization: Beyond Earth,Sid Meier's Civilization V, andSid Meier's Civilization VI, are Turing-complete. We achieve this by building three universal Turing machines--one for each game--using only the elements present in the games, and using their internal rules and mechanics as the transition function. We use these machines to prove the undecidability of these games under the unlimited turns, infinite map assumptions; and conclude by providing a sample execution of an algorithm--the three-state Busy Beaver--with one of our machines.