Various approaches are proposed to help under-resourced security researchers to detect and analyze software vulnerabilities. It is still incredibly time-consuming and labor-intensive for security researchers to fix such reported vulnerabilities due to the increasing size and complexity of modern software systems. The time lag between the reporting and fixing of a security vulnerability causes software systems to suffer from significant exposure to possible attacks. Very recently, some techniques propose to apply pretrained models to fix security vulnerabilities and have proved their success in improving repair accuracy. However, the effectiveness of existing pre-trained models has not been systematically compared and little is known about their advantages and disadvantages. To bridge this gap, we perform the first extensive study on applying various pre-trained models to automated vulnerability repair. The experimental results on two vulnerability datasets show that all studied pre-trained models consistently outperform the state-ofthe- art technique VRepair with a prediction accuracy of 32.94$\sim$44.96%. We also investigate the impact of three major phases (i.e., data pre-processing, model training and repair inference) in the vulnerability repair workflow. Inspired by the findings, we construct a simplistic vulnerability repair approach that adopts the transfer learning from bug fixing. Surprisingly, such a simplistic approach can further improve the prediction accuracy of pre-trained models by 9.40% on average. Besides, we provide additional discussion from different aspects (e.g., code representation and a preliminary study with ChatGPT) to illustrate the capacity and limitation of pre-trained model-based techniques. Finally, we further pinpoint various practical guidelines (e.g., the improvement of fine-tuning) for advanced pre-trained model-based vulnerability repair in the near future. Our study highlights the promising future of adopting pre-trained models to patch real-world security vulnerabilities and reduce the manual debugging effort of security experts in practice.

Automated program repair (APR) aims to fix software bugs without manual debugging efforts and plays a crucial role in software development and maintenance. Template-based APR has been widely investigated and shown promising results. However, it is challenging for template-based APR to select the appropriate donor code, which is an important repair ingredient for generating candidate patches. Inappropriate donor code may cause plausible but incorrect patch generation even with correct fix patterns, limiting the repair performance. In this paper, we aim to revisit template-based APR, and propose Gamma, to directly leverage large pre-trained language models for donor code generation. Our main insight is that instead of retrieving donor code in the local buggy file, we can directly predict the correct code tokens based on the context code snippets and repair patterns by a cloze task. Specifically, (1) Gamma revises a variety of fix templates from state-of-the-art template-based APR techniques (i.e., TBar) and transforms them into mask patterns. (2) Gamma adopts a pre-trained language model to predict the correct code for masked code as a fill-in-the-blank task. Although our idea is general and can be built on various existing pre-trained language models, we have implemented Gamma as a practical APR tool based on the recent UniXcoder model. The experimental results demonstrate that Gamma correctly repairs 82 bugs on Defects4J-v1.2, which achieves 20.59% (14 bugs) and 26.15% (17 bugs) improvement over the previous state-of-the-art template-based approach TBar and learning-based one Recoder. Furthermore, Gamma repairs 45 bugs and 22 bugs from the additional Defects4J-v2.0 and QuixBugs, indicating the generalizability of Gamma in addressing the dataset overfitting issue. We also prove that adopting other pre-trained language models can provide substantial advancement, e.g., CodeBERT-based and ChatGPT-based Gamma is able to fix 80 and 67 bugs on Defects4J-v1.2, indicating the scalability of Gamma. Overall, our study highlights the promising future of adopting pre-trained models to generate correct patches on top of fix patterns in practice.