The storage and recall of factual associations in auto-regressive transformer language models (LMs) have drawn a great deal of attention, inspiring knowledge editing by directly modifying the located model weights. Most editing works achieve knowledge editing under the guidance of existing interpretations of knowledge recall that mainly focus on subject knowledge. However, these interpretations are seriously flawed, neglecting relation information and leading to the *over-generalizing* problem for editing. In this work, we discover a novel relation-focused perspective to interpret the knowledge recall of transformer LMs during inference and apply it on single knowledge editing to avoid over-generalizing. Experimental results on the dataset supplemented with a new R-Specificity criterion demonstrate that our editing approach significantly alleviates over-generalizing while remaining competitive on other criteria, breaking the domination of subject-focused editing for future research.

Due to the demand for efficient fine-tuning of large language models, Low-Rank Adaptation (LoRA) has been widely adopted as one of the most effective parameter-efficient fine-tuning methods. Nevertheless, while LoRA improves efficiency, there remains room for improvement in accuracy. Herein, we adopt a novel perspective to assess the characteristics of LoRA ranks. The results reveal that different ranks within the LoRA modules not only exhibit varying levels of importance but also evolve dynamically throughout the fine-tuning process, which may limit the performance of LoRA. Based on these findings, we propose BeamLoRA, which conceptualizes each LoRA module as a beam where each rank naturally corresponds to a potential sub-solution, and the fine-tuning process becomes a search for the optimal sub-solution combination. BeamLoRA dynamically eliminates underperforming sub-solutions while expanding the parameter space for promising ones, enhancing performance with a fixed rank. Extensive experiments across three base models and 12 datasets spanning math reasoning, code generation, and commonsense reasoning demonstrate that BeamLoRA consistently enhances the performance of LoRA, surpassing the other baseline methods.

Parameter-efficient fine-tuning (PEFT) has become a common method for fine-tuning large language models, where a base model can serve multiple users through PEFT module switching.To enhance user experience, base models require periodic updates.However, once updated, PEFT modules fine-tuned on previous versions often suffer substantial performance degradation on newer versions.Re-tuning these numerous modules to restore performance would incur significant computational costs.Through a comprehensive analysis of the changes that occur during base model updates, we uncover an interesting phenomenon: continual training primarily affects task-specific knowledge stored in Feed-Forward Networks (FFN), while having less impact on the task-specific pattern in the Attention mechanism.Based on these findings, we introduce Trans-PEFT, a novel approach that enhances the PEFT module by focusing on the task-specific pattern while reducing its dependence on certain knowledge in the base model.Further theoretical analysis supports our approach.Extensive experiments across 7 base models and 12 datasets demonstrate that Trans-PEFT trained modules can maintain performance on updated base models without re-tuning, significantly reducing maintenance overhead in real-world applications 1 .

The high costs of customizing large language models (LLMs) fundamentally limit their adaptability to user-specific needs.Consequently, LLMs are increasingly offered as cloud-based services, a paradigm that introduces critical limitations: providers struggle to support personalized customization at scale, while users face privacy risks when exposing sensitive data.To address this dual challenge, we propose Customized Black-box Prompt Tuning (CBP-Tuning), a novel framework that facilitates efficient local customization while preserving bidirectional privacy.Specifically, we design a two-stage framework: (1) a prompt generator trained on the server-side to capture domainspecific and task-agnostic capabilities, and (2) user-side gradient-free optimization that tailors soft prompts for individual tasks.This approach eliminates the need for users to access model weights or upload private data, requiring only a single customized vector per task while achieving effective adaptation.Furthermore, the evaluation of CBP-Tuning in the commonsense reasoning, medical and financial domain settings demonstrates superior performance compared to baselines, showcasing its advantages in task-agnostic processing and privacy preservation.

Open-world attribute extraction is one of the most important tasks of information extraction aiming to mine all the valuable attributes of entities and their corresponding values from unstructured texts, usually in the form of (entity, attribute, value) triplets. However, existing methods have difficulty extracting attribute triplets from open-world sparse corpora where the attribute names are not previously given, especially in the few- shot scenario with only few manual annotations available. To solve the above problems, we propose a two-stage Meta-pattern-Enhanced Generative Few-shot Attribute Extraction (MEGFAE) framework which can be used to discover utmost valuable attribute triplets from open-world sparse corpora in a generative manner. For evaluation on open-world sparse corpora, we introduce a benchmark dataset called OSN-51511The dataset is available in https://github.com/sunshower-liu/OSN-515.. Experimental results verifies the effectiveness of our framework and inspires future explorations on the text mining on sparse corpora.

Parameter-Efficient Tuning (PET) has shown remarkable performance by fine-tuning only a small number of parameters of the pre-trained language models (PLMs) for the downstream tasks, while it is also possible to construct backdoor attacks due to the vulnerability of pretrained weights.However, a large reduction in the number of attackable parameters in PET will cause the user's fine-tuning to greatly affect the effectiveness of backdoor attacks, resulting in backdoor forgetting.We find that the backdoor injection process can be regarded as multitask learning, which has a convergence imbalance problem between the training of clean and poisoned data.And this problem might result in forgetting the backdoor.Based on this finding, we propose a gradient control method to consolidate the attack effect, comprising two strategies.One controls the gradient magnitude distribution cross layers within one task and the other prevents the conflict of gradient directions between tasks.Compared with previous backdoor attack methods in the scenario of PET, our method improves the effect of the attack on sentiment classification and spam detection respectively, which shows that our method is widely applicable to different tasks.