It has become a popular paradigm to transfer the knowledge of large-scale pre-trained models to various downstream tasks via fine-tuning the entire model parameters. However, with the growth of model scale and the rising number of downstream tasks, this paradigm inevitably meets the challenges in terms of computation consumption and memory footprint issues. Recently, Parameter-Efficient Fine-Tuning (PEFT) (e.g., Adapter, LoRA, BitFit) shows a promising paradigm to alleviate these concerns by updating only a portion of parameters. Despite these PEFTs having demonstrated satisfactory performance in natural language processing, it remains under-explored for the question: whether these techniques could be transferred to graph-based tasks with Graph Transformer Networks (GTNs)? Therefore, in this paper, we fill this gap by providing extensive benchmarks with traditional PEFTs on a range of graph-based downstream tasks. Our empirical study shows that it is sub-optimal to directly transfer existing PEFTs to graph-based tasks due to the issue of feature distribution shift. To address this issue, we propose a novel structure-aware PEFT approach, named G-Adapter, which leverages graph convolution operation to introduce graph structure information (e.g., graph adjacency matrix) as an inductive bias to guide the updating process. Further, we propose Bregman proximal point optimization to alleviate feature distribution shift by preventing the model from aggressive update. Extensive experiments demonstrate that G-Adapter obtains state-of-the-art performance compared to counterparts on nine graph benchmark datasets based on diverse pre-trained GTNs, and delivers tremendous memory footprint efficiency compared to the conventional paradigm.

To fully leverage the advantages of large-scale pre-trained language models (PLMs) on downstream tasks, it has become a ubiquitous adaptation paradigm to fine-tune the entire parameters of PLMs.However, this paradigm poses issues of inefficient updating and resource overconsuming for fine-tuning in data-scarce and resource-limited scenarios, because of the large scale of parameters in PLMs.To alleviate these concerns, in this paper, we propose a parameterefficient fine-tuning method HiFi, that is, only the highly informative and strongly correlated attention heads for the specific task are finetuned.To search for those significant attention heads, we develop a novel framework to analyze the effectiveness of heads.Specifically, we first model the relationship between heads into a graph from two perspectives of information richness and correlation, and then apply PageRank algorithm to determine the relative importance of each head.Extensive experiments on the GLUE benchmark demonstrate the effectiveness of our method, and show that HiFi obtains state-of-the-art performance over the prior baselines.

Recently pre-trained language models have shown great success in natural language processing. To fully leverage these models on downstream tasks, full fine-tuning has become a ubiquitous training paradigm. However, it is inefficient and resource-intensive to fine-tune the entire parameters of the model since it always employs a huge scale of parameters (e.g., GPT-3 has 175B parameters). To address these issues, we propose a simple yet effective method to fine-tune the pre-trained models, named Gradient activation mapping Masking (GamMa). Specifically, GamMa first detects the class-specific parameters from the perspective of gradient activation mapping, and then aggregates them to work out the particular parameters that can greatly influence the model predictive boundaries. During the fine-tuning process, only these significant parameters are updated necessarily. Besides, GamMa takes several advantages over the previous approaches: (i) it does not require modifying the original model structure; (ii) neither extra parameters nor training budgets are introduced. Extensive experiments on the GLUE benchmark demonstrate that our method outperforms the current state-of-the-art counterparts.