Abstract Compilation errors are unavoidable during the debugging process of novice students. Compiler error messages can help novices to localize and remove errors, but these messages are difficult to understand for students. Previous studies have investigated the compilation error categorization by analyzing compiler error messages, but the categorization cannot cover all kinds of errors, which limits the evaluation of compilation error studies. Therefore, a comprehensive categorization for compilation errors is needed for evaluating the performance of models or tools related to compilation errors. In this study, we first propose a new compilation error categorization, which is based on the smallest unit of the program, tokens. The experiments on 29,573 programs from three datasets show that our proposed compilation error categorization can cover more types of errors and the distribution of the error categorization are significantly different between the datasets. Then, based on our proposed categorization, we develop a neural network model CLACER (CLAssification of Compilation ERrors) for predicting the compilation errors. The results indicate that CLACER can improve the compiler's error localization accuracy and predicts the compilation error effectively. Moreover, based on the proposed categorization, we conduct empirical studies to evaluate the performance of three repairing tools (i.e., DeepFix, RLAssist, and MACER). The comparison results illustrate that DeepFix and RLAssist can fix more errors in the category of delimiter than errors in other categories. Furthermore, MACER performs better than DeepFix and RLAssist because it has a sufficient repairing pattern set for the errors. We also provide some suggestions for improving the repairing tools in the future.

Compilation errors happen during the debugging process of novice students. Compiler error messages help novices to localize and remove errors, but these messages are difficult to understand for students. Some computing education researchers analyzed the compiler error messages generated by novice’s attempts to compile their programs. However, some important questions remain open. For example, the existing compilation error category cannot cover all programs due to the simple static analysis and program repair patterns. And existing prediction models for classifying compilation errors are unsatisfactory because of the inappropriate neural networks. In this paper, we first propose a new category of compilation error based on the program tokens, which is the smallest unit of the program. Then we develop a neural network model CLACER (ClAssification of Compilation ERrors) based on TextCNN. CLACER performs better on extracting semantic features and statistical features from compiler error messages. To verify the effectiveness of our proposed category and corresponding method CLACER, we choose 16,926 student programs as our experimental subjects. Final experimental results indicate that our proposed classification category covers 16.5% more programs than the state-of-the-art category TEGCER. Moreover, CLACER improves the compiler’s localization effectiveness and with a 4.25% improvement on the TEGCER category. Further analysis shows that CLACER has a promising prediction performance for different error classes, and TextCNN is more suitable for constructing the compilation error classification model.