Online Social Networks (OSNs) widely adopt content moderation to mitigate the spread of abusive and toxic discourse. Nonetheless, the real effectiveness of moderation interventions remains unclear due to the high cost of data collection and limited experimental control. The latest developments in Natural Language Processing pave the way for a new evaluation approach. Large Language Models (LLMs) can be successfully leveraged to enhance Agent-Based Modeling and simulate human-like social behavior with unprecedented degree of believability. Yet, existing tools do not support simulation-based evaluation of moderation strategies. We fill this gap by designing a LLM-powered simulator of OSN conversations enabling a parallel, counterfactual simulation where toxic behavior is influenced by moderation interventions, keeping all else equal. We conduct extensive experiments, unveiling the psychological realism of OSN agents, the emergence of social contagion phenomena and the superior effectiveness of personalized moderation strategies.

Abstract The emergence of simulators powered by Large Language Models (LLMs) has enabled the realistic modeling of complex social phenomena while significantly reducing the costs and challenges of real-world data collection. Despite their promise, assessing the reliability of these simulators remains an open challenge. Existing validation methods often focus on isolated domains and operate at fixed levels of granularity, limiting their generalizability. In this work, we introduce simvale ( sim ulator va lidation with l atent e mbeddings), a generalizable, multi-domain framework for quantitatively assessing LLM-based social simulators. simvale leverages both interpretable features and latent representations to yield global and local assessments about the fidelity of a simulator to real-world dynamics, or about the effects of controlled interventions. We demonstrate the effectiveness of simvale through a case study evaluating a simulator’s ability to reproduce Online Social Network behavioral patterns and capture the impact of moderation interventions.