Supervised fine-tuning (SFT) is widely adopted for tailoring large language models (LLMs) to specific downstream tasks. However, the substantial computational demands of LLMs hinder iterative exploration of fine-tuning datasets and accurate evaluation of individual sample importance. To address this challenge, we introduce Meta-LoRA, a memory-efficient method for automatic sample reweighting. Meta-LoRA learns to reweight fine-tuning samples by minimizing the loss on a small, high-quality validation set through an end-to-end bi-level optimization framework based on meta-learning. To reduce memory usage associated with computing second derivatives, we approximate the bi-level optimization using gradient similarity between training and validation datasets, replacing bi-dimensional gradient similarity with the product of one-dimensional activation states and their corresponding gradients. Further memory optimization is achieved by refining gradient computations, selectively applying them to the low-rank layers of LoRA, which results in as little as 4% additional memory usage. Comprehensive evaluations across benchmark datasets in mathematics, coding, and medical domains demonstrate Meta-LoRA’s superior efficacy and efficiency. The source code is available at https://github.com/liweicheng-ai/meta-lora.

Zero-shot Relation Triplet Extraction (ZeroRTE) aims to extract relation triplets from texts containing unseen relation types. This capability benefits various downstream information retrieval (IR) tasks. The primary challenge lies in enabling models to generalize effectively to unseen relation categories. Existing approaches typically leverage the knowledge embedded in pre-trained language models to accomplish the generalization process. However, these methods focus solely on fitting the training data during training, without specifically improving the model's generalization performance, resulting in limited generalization capability. For this reason, we explore the integration of bi-level optimization (BLO) with pre-trained language models for learning generalized knowledge directly from the training data, and propose a generative meta-learning framework which exploits the 'learning-to-learn' ability of meta-learning to boost the generalization capability of generative models.

ABSTRACT Few‐shot relation extraction has attracted significant attention due to its potential to identify new relations when training samples are scarce. Previous studies demonstrate that meta‐learning techniques can significantly enhance models' adaptability in few‐shot scenarios. Most existing meta‐learning methods generalize to unseen relations by constructing effective prototype representations for each relation. However, meta‐learning techniques often require a large amount of labeled data during the meta‐training stage, which contradicts the initial purpose of addressing the issue of insufficient labeled data. Current methods usually fail to model both intra‐class similarity and inter‐class difference sufficiently, resulting in fuzzy class prototypes when distinguishing between similar but slightly different relations. Therefore, we propose a novel task, low‐resource few‐shot relation extraction (LR‐FSRE), that explores minimizing the usage of labeled data while maximizing the acquisition of task meta‐knowledge, and we design a novel graph‐based adaptive discrimination network that optimizes relation prototypes in both tasks. Specifically, we effectively represent class prototypes using the topological information between instance and relation‐level representation. Then, the relation discrimination network considers the difference between similar classes to further improve the recognition ability of relation classes. Finally, a debiased optimization strategy is employed to optimize the learning processes of task‐general and task‐specific knowledge. Experimental results show that our proposed framework outperforms the state‐of‐the‐art methods in FSRE and LR‐FSRE tasks, and achieves significant improvement in accuracy across challenging cross‐domain and similarity relation discrimination scenarios.

Signed graphs can model friendly or antagonistic relations where edges are annotated with a positive or negative sign. The main downstream task in signed graph analysis is $\textit{link sign prediction}$. Signed Graph Neural Networks (SGNNs) have been widely used for signed graph representation learning. While significant progress has been made in SGNNs research, two issues (i.e., graph sparsity and unbalanced triangles) persist in the current SGNN models. We aim to alleviate these issues through data augmentation ($\textit{DA}$) techniques which have demonstrated effectiveness in improving the performance of graph neural networks. However, most graph augmentation methods are primarily aimed at graph-level and node-level tasks (e.g., graph classification and node classification) and cannot be directly applied to signed graphs due to the lack of side information (e.g., node features and label information) in available real-world signed graph datasets. Random $\textit{DropEdge} $is one of the few $\textit{DA}$ methods that can be directly used for signed graph data augmentation, but its effectiveness is still unknown. In this paper, we first provide the generalization bound for the SGNN model and demonstrate from both experimental and theoretical perspectives that the random $\textit{DropEdge}$ cannot improve the performance of link sign prediction. Therefore, we propose a novel signed graph augmentation method, $\underline{S}$igned $\underline{G}$raph $\underline{A}$ugmentation framework (SGA). Specifically, SGA first integrates a structure augmentation module to detect candidate edges solely based on network information. Furthermore, SGA incorporates a novel strategy to select beneficial candidates. Finally, SGA introduces a novel data augmentation perspective to enhance the training process of SGNNs. Experiment results on six real-world datasets demonstrate that SGA effectively boosts the performance of diverse SGNN models, achieving improvements of up to 32.3\% in F1-micro for SGCN on the Slashdot dataset in the link sign prediction task.

The shortage of labeled data has been a long-standing challenge for relation extraction (RE) tasks. Semi-supervised RE (SSRE) is a promising way through annotating unlabeled samples with pseudolabels as additional training data. However, some pseudolabels on unlabeled data might be erroneous and will bring misleading knowledge into SSRE models. For this reason, we propose a novel adversarial multi-teacher distillation (AMTD) framework, which includes multi-teacher knowledge distillation and adversarial training (AT), to capture the knowledge on unlabeled data in a refined way. Specifically, we first develop a general knowledge distillation (KD) technique to learn not only from pseudolabels but also from the class distribution of predictions by different models in existing SSRE methods. To improve the robustness of the model, we further empower the distillation process with a language model-based AT technique. Extensive experimental results on two public datasets demonstrate that our framework significantly promotes the performance of the base SSRE methods.

The performance of relation extraction (RE) is hindered by the lack of sufficient labeled data. Semisupervised methods can offer to help hands with this problem by augmenting high-quality unlabeled samples into the training data. However, existing semisupervised RE methods either need a set of manually defined rules or rely on the classifier trained on the small labeled data, i.e., the former requires the heavy intervention of human knowledge, and the latter is bound to the number and the quality of the labeled data. In this article, we present a novel semisupervised RE method that involves small human efforts and is robust to the size of the initial set of labeled data. Specifically, we adopt only two simple rules to build the lexical and semantic graphs which connect the labeled samples with the unlabeled ones. In this way, the graphs are much easier to construct yet keep the ability to transfer knowledge from labeled samples to unlabeled ones. We then develop a graph interaction module to fully exploit the reference information in lexical and semantic graphs, which is used to jointly recognize the high-quality unlabeled samples with the classifier. We conduct extensive experimental results on two public datasets. The results demonstrate that our proposed method significantly outperforms the state-of-the-art baselines.

Few-shot relation extraction (FSRE) has been a challenging problem since it only has a handful of training instances.Existing models follow a 'one-for-all' scheme where one general large model performs all individual N-way-Kshot tasks in FSRE, which prevents the model from achieving the optimal point on each task.In view of this, we propose a model generation framework that consists of one general model for all tasks and many tiny task-specific models for each individual task.The general model generates and passes the universal knowledge to the tiny models which will be further fine-tuned when performing specific tasks.In this way, we decouple the complexity of the entire task space from that of all individual tasks while absorbing the universal knowledge.Extensive experimental results on two public datasets demonstrate that our framework reaches a new state-of-the-art performance for FRSE tasks.Our code is available at: https://github.com/NLPWM-WHU/GM_GEN.

Manual annotation of labeled data for relation extraction is time-consuming and labor-intensive. Semi-supervised methods can offer helping hands for this problem and have aroused great research interests. Existing works focus on mapping the unlabeled samples to the classes to augment the labeled dataset. However, it is hard to find an overall good mapping function, especially for the samples with complicated syntactic components in one sentence. To tackle this limitation, we propose to build the connection between the unlabeled data and the labeled ones rather than directly mapping the unlabeled samples to the classes. Specifically, we first use two kinds of information to construct a reference graph, including entity reference and verb reference. The goal is to lexically connect the unlabeled sample(s) to the labeled one(s). Then, we develop a Multi-head Reference Graph (MRefG) model to exploit the reference information for better recognizing high-quality unlabeled samples. The effectiveness of our method is demonstrated by extensive comparison experiments with the state-of-the-art baselines. To tackle this limitation, we propose to build the connection between the unlabeled data and the labeled ones rather than directly mapping the unlabeled samples to the classes. Specifically, we first use two kinds of information to construct a reference graph, including entity reference and verb reference. The goal is to lexically connect the unlabeled sample(s) to the labeled one(s). Then, we develop a Multi-head Reference Graph (MRefG) model to exploit the reference information for better recognizing high-quality unlabeled samples. The effectiveness of our method is demonstrated by extensive comparison experiments with the state-of-the-art baselines.