Circuit graph discovery has emerged as a fundamental approach to elucidating the skill mechanistic of language models. Despite the output faithfulness of circuit graphs, they suffer from atomic ablation, which causes the loss of causal dependencies between connected components. In addition, their discovery process, designed to preserve output faithfulness, inadvertently captures extraneous effects other than an isolated target skill. To alleviate these challenges, we introduce skill paths, which offer a more refined and compact representation by isolating individual skills within a linear chain of components. To enable skill path extracting from circuit graphs, we propose a three-step framework, consisting of decomposition, pruning, and post-hoc causal mediation. In particular, we offer a complete linear decomposition of the transformer model which leads to a disentangled computation graph. After pruning, we further adopt causal analysis techniques, including counterfactuals and interventions, to extract the final skill paths from the circuit graph. To underscore the significance of skill paths, we investigate three generic language skills—Previous Token Skill, Induction Skill, and In-Context Learning Skill—using our framework. Experiments support two crucial properties of these skills, namely stratification and inclusiveness.

Temporal Action Segmentation (TAS) is fundamental to video understanding, aiming to recognize and segment actions within untrimmed videos. While over-segmentation errors have received considerable attention, the equally critical issue of out of-context errors stemming from the neglect of implicit causal relationships between actions remains under-explored. However, constraining these unknown causal relationships at the frame level—including identifying causal links and strengths—presents a significant challenge, leading to notable research gaps. To tackle these challenges, we define a Temporal Action Causal Model (TACM) for the TAS task as declarative guidance, which efficiently injects the causal relationship between actions into frame variables in a generated manner. To this end, we propose the Causal Action Segmentation Refiner (CaASR), a refinement framework designed to mitigate out-of-context errors in segmentation results of backbone models. By reconstructing the causal generation process of post-segmentation frame variables, CaASR constrains causal relationships between actions and ensures the reliability of generated causal relationships. Extensive experiments demonstrate that CaASR significantly enhances both segmentation performance and interpretability across various backbone models, including state-of-the-art methods.

Large language models (LLMs) are widely recognized for their exceptional capacity to capture semantics meaning.Yet, there remains no established metric to quantify this capability.In this work, we introduce a quantitative metric, Information Emergence (IE), designed to measure LLMs' ability to extract semantics from input tokens.We formalize "semantics" as the meaningful information abstracted from a sequence of tokens and quantify this by comparing the entropy reduction observed for a sequence of tokens (macro-level) and individual tokens (micro-level).To achieve this, we design a lightweight estimator to compute the mutual information at each transformer layer, which is agnostic to different tasks and language model architectures.We apply IE in both synthetic in-context learning (ICL) scenarios and natural sentence contexts.Experiments demonstrate informativeness and patterns about semantics.While some of these patterns confirm the conventional prior linguistic knowledge, the rest are relatively unexpected, which may provide new insights.

Fine-grained classification via LLMs is susceptible to more complex label biases compared to traditional classification tasks. Existing bias mitigation strategies, such as retraining, post-hoc adjustment, and parameter-efficient fine-tuning (PEFT) are primarily effective for simple classification biases, such as stereotypes, but fail to adequately address prediction propensity and discriminative ability biases. In this paper, we analyze these two bias phenomena and observe their progressive accumulation from intermediate to deeper layers within LLMs. To mitigate this issue, we propose a bias-aware optimization framework that incorporates two distinct label balance constraints with a PEFT strategy targeting an intermediate layer. Our approach adjusts less than 1% of the model’s parameters while effectively curbing bias amplification in deeper layers. Extensive experiments conducted across 12 datasets and 5 LLMs demonstrate that our method consistently outperforms or matches the performance of full-parameter fine-tuning and LoRA, achieving superior results with lower perplexity.

Circuit discovery has gradually become one of the prominent methods for mechanistic interpretability, and research on circuit completeness has also garnered increasing attention. Methods of circuit discovery that do not guarantee completeness not only result in circuits that are not fixed across different runs but also cause key mechanisms to be omitted. The nature of incompleteness arises from the presence of OR gates within the circuit, which are often only partially detected in standard circuit discovery methods. To this end, we systematically introduce three types of logic gates: AND, OR, and ADDER gates, and decompose the circuit into combinations of these logical gates. Through the concept of these gates, we derive the minimum requirements necessary to achieve faithfulness and completeness. Furthermore, we propose a framework that combines noising-based and denoising-based interventions, which can be easily integrated into existing circuit discovery methods without significantly increasing computational complexity. This framework is capable of fully identifying the logic gates and distinguishing them within the circuit. In addition to the extensive experimental validation of the framework's ability to restore the faithfulness, completeness, and sparsity of circuits, using this framework, we uncover fundamental properties of the three logic gates, such as their proportions and contributions to the output, and explore how they behave among the functionalities of language models.

Existing methods, including large language models (LLMs), excel at capturing semantic correlations between utterances, but often struggle to accurately distinguish specific causal relationships. This limitation poses a significant challenge for reasoning-intensive tasks in affective computing, where precise identification of emotional triggers and their effects is crucial. Our preliminary work demonstrated the potential of introducing i.i.d. noise terms within Structural Causal Models (SCMs) for the Emotion-Cause Pair Extraction (ECPE) task. However, this approach relied on end-to-end learning of high-dimensional latent representations, which hindered both scalability to LLMs and model interpretability. To address these issues, we conceptualize i.i.d. noise terms as token-level implicit causes—natural language expressions that reflect a speaker's underlying emotions, intentions, or situational context. Building on this insight, we introduce ICE (Implicit-Cause-Enhanced), an instruction-based framework that leverages implicit causes to enhance causal reasoning in LLMs. First, we design prompts that heuristically guide LLMs to generate implicit causes, which are then iteratively refined via an external evaluation mechanism. Second, by incorporating these implicit causes as intermediate reasoning steps, ICE improves the accuracy of emotion-cause pair prediction. Moreover, we distill the rationales produced by ICE into lightweight generative models, demonstrating that even small models can benefit from implicit-cause-driven reasoning. Extensive experiments in both instruction-based and distillation-based settings confirm the effectiveness, robustness, and interpretability of our approach.

Reasoning, a crucial aspect of NLP research, has not been adequately addressed by prevailing models including Large Language Model. Conversation reasoning, as a critical component of it, remains largely unexplored due to the absence of a welldesigned cognitive model. In this paper, inspired by intuition theory on conversation cognition, we develop a conversation cognitive model (CCM) that explains how each utterance receives and activates channels of information recursively. Besides, we algebraically transformed CCM into a structural causal model (SCM) under some mild assumptions, rendering it compatible with various causal discovery methods. We further propose a probabilistic implementation of the SCM for utterance-level relation reasoning. By leveraging variational inference, it explores substitutes for implicit causes, addresses the issue of their unobservability, and reconstructs the causal representations of utterances through the evidence lower bounds. Moreover, we constructed synthetic and simulated datasets incorporating implicit causes and complete cause labels, alleviating the current situation where all available datasets are implicit-causesagnostic. Extensive experiments demonstrate that our proposed method significantly outperforms existing methods on synthetic, simulated, and real-world datasets. Finally, we analyze the performance of CCM under latent confounders and propose theoretical ideas for addressing this currently unresolved issue.

Our investigation into the Affective Reasoning in Conversation (ARC) task highlights the challenge of causal discrimination.Almost all existing models, including large language models (LLMs), excel at capturing semantic correlations within utterance embeddings but fall short in determining the specific causal relationships.To overcome this limitation, we propose the incorporation of i.i.d.noise terms into the conversation process, thereby constructing a structural causal model (SCM).It explores how distinct causal relationships of fitted embeddings can be discerned through independent conditions.To facilitate the implementation of deep learning, we introduce the cogn frameworks to handle unstructured conversation data, and employ an autoencoder architecture to regard the unobservable noise as learnable "implicit causes."Moreover, we curate a synthetic dataset that includes i.i.d.noise.Through comprehensive experiments, we validate the effectiveness and interpretability of our approach.Our code is available in https://github.com/Zodiark-ch/ mater-of-our-EMNLP2023-paper.