Recent interpretability work on large language models (LLMs) has been increasingly dominated by a feature-discovery approach with the help of proxy modules. Then, the quality of features learned by, e.g., sparse auto-encoders (SAEs), is evaluated. This paradigm naturally raises a critical question: do such learned features have better properties than those already represented within the original model parameters, and unfortunately, only a few studies have made such comparisons systematically so far. In this work, we revisit the interpretability of feature vectors stored in feed-forward (FF) layers, given the perspective of FF as key-value memories, with modern interpretability benchmarks. Our extensive evaluation revealed that SAE and FFs exhibits a similar range of interpretability, although SAEs displayed an observable but minimal improvement in some aspects. Furthermore, in certain aspects, surprisingly, even vanilla FFs yielded better interpretability than the SAEs, and features discovered in SAEs and FFs diverged. These bring questions about the advantage of SAEs from both perspectives of feature quality and faithfulness, compared to directly interpreting FF feature vectors, and FF key-value parameters serve as a strong baseline in modern interpretability research.

Large language models (LLMs) take advantage of step-by-step reasoning instructions, e.g., chain-of-thought (CoT) prompting.Building on this, their ability to perform CoT-style reasoning robustly is of interest from a probing perspective.In this study, we inspect the stepby-step reasoning ability of LLMs with a focus on negation, which is a core linguistic phenomenon that is difficult to process.In particular, we introduce several controlled settings (e.g., reasoning on fictional entities) to evaluate the logical reasoning abilities of the models.We observed that dozens of modern LLMs were not robust against lexical negation (e.g., plausi-ble→implausible) when performing CoT-style reasoning, and the results highlight unique limitations in each LLM family.https://github.com/muyo8692/ stepbystep-reasoning-vs-negation Setting Few-shot exemplars Target example If fails at this setting BASE Is a sentence "A does B" plausible?A is a C player.B happens in C/X.So the answer is yes/no.Is a sentence "D does E" plausible?D is a F player.E happens in F/Y.So the answer is __ CoT-style reasoning fails. FIC Is a sentence "A does B" plausible?A is a C player.B happens in C/X.So the answer is yes/no.Is a sentence "α does β" plausible?α is a γ player.β happens in γ/χ.So the answer is __ Reasoning cannot be abstracted to fictional texts. FICNEG Is a sentence "A does B" implausible?A is a C player.B happens in C/X.So the answer is yes/no.Is a sentence "α does β" implausible?α is a γ player.β happens in γ/χ.So the answer is __ Abstract CoT-style reasoning is only achieved on the affirmative domain. FICNEG-O Is a sentence "A does B" plausible?A is a C player.B happens in C/X.So the answer is yes/no.Is a sentence "α does β" implausible?α is a γ player.β happens in γ/χ.So the answer is __ Model cannot handle domain shift in terms of negation.Table 1: General task format in each setting.Few-shot exemplars are first shown to a model, and then the model answers to the target example given its question and reasoning chain.Symbols, e.g., A and α, are replaced with certain real or fictional entities in the actual input.The REAL setting indicates that the entity choices reflect the factual reality, and FIC.indicates that the entity choices do not reflect factual reality, e.g., Is "Judy Tate was safe at first." plausible?Judy Tate is a turboglide player.Getting out at first happens in turboglide.So the answer is yes.Refer to Appendix A for the exact input. Format: SyllogismWe evaluated the LLMs' ability to judge the validity of particular types of syllogisms.Here, we utilized three settings to ensure the robustness of the results (Section 3); however, we consider the following SPORTS TASK (SP) format as an example to explain the settings.The base format of the syllogism is as follows:Premise1: PERSON is a SPORT player.Premise2: ACTION happens in the SPORT. Conclusion: PERSON does ACTION.Here, the transitivity of reasoning (A>B, B>C, then A>C) is targeted. C.3 Answer Distribution