Hallucinations in large language models are a widespread problem, yet the mechanisms behind whether models will hallucinate are poorly understood, limiting our ability to solve this problem. Using sparse autoencoders as an interpretability tool, we discover that a key part of these mechanisms is entity recognition, where the model detects if an entity is one it can recall facts about. Sparse autoencoders uncover meaningful directions in the representation space, these detect whether the model recognizes an entity, e.g. detecting it doesn't know about an athlete or a movie. This shows that models can have self-knowledge: internal representations about their own capabilities. These directions are causally relevant: capable of steering the model to refuse to answer questions about known entities, or to hallucinate attributes of unknown entities when it would otherwise refuse. We demonstrate that despite the sparse autoencoders being trained on the base model, these directions have a causal effect on the chat model's refusal behavior, suggesting that chat finetuning has repurposed this existing mechanism. Furthermore, we provide an initial exploration into the mechanistic role of these directions in the model, finding that they disrupt the attention of downstream heads that typically move entity attributes to the final token.

Information flows by routes inside the network via mechanisms implemented in the model.These routes can be represented as graphs where nodes correspond to token representations and edges to computations.We automatically build these graphs in a top-down manner, for each prediction leaving only the most important nodes and edges.In contrast to the existing workflows relying on activation patching, we do this through attribution: this allows us to efficiently uncover existing circuits with just a single forward pass.Unlike with patching, we do not need a human to carefully design prediction templates, and we can extract information flow routes for any prediction (not just the ones among the allowed templates).As a result, we can analyze model behavior in general, for specific types of predictions, or different domains.We experiment with Llama 2 and show that some attention head roles are overall important, e.g.previous token heads and subword merging heads.Next, we find similarities in Llama 2 behavior when handling tokens of the same part of speech.Finally, we show that some model components can be specialized on domains such as coding or multilingual texts.

Language Generation Models produce words based on the previous context.Although existing methods offer input attributions as explanations for a model's prediction, it is still unclear how prior words affect the model's decision throughout the layers.In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation.Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines.Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable.Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions.

Multilingual neural machine translation architectures mainly differ in the number of sharing modules and parameters applied among languages. In this paper, and from an algorithmic perspective, we explore whether the chosen architecture, when trained with the same data, influences the level of gender bias. Experiments conducted in three language pairs show that language-specific encoder-decoders exhibit less bias than the shared architecture. We propose two methods for interpreting and studying gender bias in machine translation based on source embeddings and attention. Our analysis shows that, in the language-specific case, the embeddings encode more gender information, and their attention is more diverted. Both behaviors help in mitigating gender bias.

In Neural Machine Translation (NMT), each token prediction is conditioned on the source sentence and the target prefix (what has been previously translated at a decoding step).However, previous work on interpretability in NMT has mainly focused solely on source sentence tokens' attributions.Therefore, we lack a full understanding of the influences of every input token (source sentence and target prefix) in the model predictions.In this work, we propose an interpretability method that tracks input tokens' attributions for both contexts.Our method, which can be extended to any encoder-decoder Transformer-based model, allows us to better comprehend the inner workings of current NMT models.We apply the proposed method to both bilingual and multilingual Transformers and present insights into their behaviour.

The Transformer architecture aggregates input information through the self-attention mechanism, but there is no clear understanding of how this information is mixed across the entire model.Additionally, recent works have demonstrated that attention weights alone are not enough to describe the flow of information.In this paper, we consider the whole attention block -multi-head attention, residual connection, and layer normalization-and define a metric to measure token-to-token interactions within each layer.Then, we aggregate layer-wise interpretations to provide input attribution scores for model predictions.Experimentally, we show that our method, ALTI (Aggregation of Layer-wise Token-to-token Interactions), provides more faithful explanations and increased robustness than gradient-based methods.Grad ℓ2 went here just before a movie .the service was fast but that ' s it .i ordered the mango and shrimp quesadilla .my friend ordered nachos .the food was not good .i and my friend could not finish our food and we had stomach aches immediately .IG ℓ2 went here just before a movie .the service was fast but that ' s it .i ordered the mango and shrimp quesadilla .my friend ordered nachos .the food was not good .i and my friend could not finish our food and we had stomach aches immediately .ALTI went here just before a movie .the service was fast but that ' s it .i ordered the mango and shrimp quesadilla .my friend ordered nachos .the food was not good .i and my