Michal Štefánik, Timothee Mickus, Marek Kadlčík, Bertram Højer, Michal Spiegel, Raúl Vázquez, Aman Sinha, Josef Kuchař, Philipp Mondorf, Pontus Stenetorp. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

A fundamental question in interpretability research is to what extent neural networks, particularly language models, implement reusable functions through subnetworks that can be composed to perform more complex tasks.Recent advances in mechanistic interpretability have made progress in identifying circuits, which represent the minimal computational subgraphs responsible for a model's behavior on specific tasks.However, most studies focus on identifying circuits for individual tasks without investigating how functionally similar circuits relate to each other.To address this gap, we study the modularity of neural networks by analyzing circuits for highly compositional subtasks within a transformer-based language model.Specifically, given a probabilistic context-free grammar, we identify and compare circuits responsible for ten modular string-edit operations.Our results indicate that functionally similar circuits exhibit both notable node overlap and crosstask faithfulness.Moreover, we demonstrate that the circuits identified can be reused and combined through set operations to represent more complex functional model capabilities.

The ability of large language models (LLMs) to validate their output and identify potential errors is crucial for ensuring robustness and reliability. However, current research indicates that LLMs struggle with self-correction, encountering significant challenges in detecting errors. While studies have explored methods to enhance self-correction in LLMs, relatively little attention has been given to understanding the models’ internal mechanisms underlying error detection. In this paper, we present a mechanistic analysis of error detection in LLMs, focusing on simple arithmetic problems. Through circuit analysis, we identify the computational subgraphs responsible for detecting arithmetic errors across four smaller-sized LLMs. Our findings reveal that all models heavily rely on \textit{consistency heads}\textemdash{}attention heads that assess surface-level alignment of numerical values in arithmetic solutions. Moreover, we observe that the models’ internal arithmetic computation primarily occurs in higher layers, whereas validation takes place in middle layers, before the final arithmetic results are fully encoded. This structural dissociation between arithmetic computation and validation seems to explain why smaller-sized LLMs struggle to detect even simple arithmetic errors.

Large language models readily memorize arbitrary training instances, such as label noise, yet they perform strikingly well on reasoning tasks.In this work, we investigate how language models memorize label noise, and why such memorization in many cases does not heavily affect generalizable reasoning capabilities.Using two controllable synthetic reasoning datasets with noisy labels, four-digit addition (FDA) and two-hop relational reasoning (THR), we discover a reliance of memorization on generalizable reasoning mechanisms: models continue to compute intermediate reasoning outputs even when retrieving memorized noisy labels, and intervening reasoning adversely affects memorization.We further show that memorization operates through distributed encoding, i.e., aggregating various inputs and intermediate results, rather than building a look-up mechanism from inputs to noisy labels.Moreover, our FDA case study reveals memorization occurs via outlier heuristics, where existing neuron activation patterns are slightly shifted to fit noisy labels.Together, our findings suggest that memorization of label noise in language models builds on, rather than overrides, the underlying reasoning mechanisms, shedding lights on the intriguing phenomenon of benign memorization.1

Deductive reasoning plays a pivotal role in the formulation of sound and cohesive arguments.It allows individuals to draw conclusions that logically follow, given the truth value of the information provided.Recent progress in the domain of large language models (LLMs) has showcased their capability in executing deductive reasoning tasks.Nonetheless, a significant portion of research primarily assesses the accuracy of LLMs in solving such tasks, often overlooking a deeper analysis of their reasoning behavior.In this study, we draw upon principles from cognitive psychology to examine inferential strategies employed by LLMs, through a detailed evaluation of their responses to propositional logic problems.Our findings indicate that LLMs display reasoning patterns akin to those observed in humans, including strategies like supposition following or chain construction.Moreover, our research demonstrates that the architecture and scale of the model significantly affect its preferred method of reasoning, with more advanced models tending to adopt strategies more frequently than less sophisticated ones.Importantly, we assert that a model's accuracy, that is the correctness of its final conclusion, does not necessarily reflect the validity of its reasoning process.This distinction underscores the necessity for more nuanced evaluation procedures in the field. Problem 1:Statements: 1.Either there is a white marble in the box or else there is a black marble in the box, but not both.2. Either there is a black marble in the box or else there is a pink marble in the box, but not both.3.There is a pink marble in the box if and only if there is a gray marble in the box.Conclusion: If there is a white marble in the box then there is a gray marble in the box.Response: Sure, I'd be happy to help!Here's my reasoning process for this problem:1. First, I'll start by analyzing the given statements: From Statement 1, we know that the box contains either a white marble or a black marble, but not both.From Statement 2, we know that the box contains either a black marble or a pink marble, but not both.From Statement 3, we know that if there is a pink marble in the box, then there must be a gray marble in the box.2. Now, let's consider the conclusion: If there is a white marble in the box then there is a gray marble in the box.Can we logically deduce this conclusion from the given statements?Well, if there is a white marble in the box, then by Statement 1, there cannot be a black marble in the box.If there is no black marble in the box, then by Statement 2, there must be a pink marble in the box.But if there is a pink marble in the box, then by Statement 3, there must be a gray marble in the box.So, yes, we can logically deduce the conclusion that if there is a white marble in the box, then there is a gray marble in the box.

Knights and knaves problems represent a classic genre of logical puzzles where characters either tell the truth or lie.The objective is to logically deduce each character's identity based on their statements.The challenge arises from the truth-telling or lying behavior, which influences the logical implications of each statement.Solving these puzzles requires not only direct deductions from individual statements, but the ability to assess the truthfulness of statements by reasoning through various hypothetical scenarios.As such, knights and knaves puzzles serve as compelling examples of suppositional reasoning.In this paper, we introduce TruthQuest, a benchmark for suppositional reasoning based on the principles of knights and knaves puzzles.Our benchmark presents problems of varying complexity, considering both the number of characters and the types of logical statements involved.Evaluations on TruthQuest show that large language models like Llama 3 and Mixtral-8x7B exhibit significant difficulties solving these tasks.A detailed error analysis of the models' output reveals that lower-performing models exhibit a diverse range of reasoning errors, frequently failing to grasp the concept of truth and lies.In comparison, more proficient models primarily struggle with accurately inferring the logical implications of potentially false statements.