With the growing reliance on automated code completion tools in software development, the need for comprehensive evaluation benchmarks has become critical. Existing benchmarks focus more on code completion in function and class level by pro-viding text descriptions to prompt the model. By contrast, such descriptive prompt is commonly unavailable in real development and code completion can occur in wider range of situations such as in the middle of a function or a code block. These limitations makes existing evaluation benchmarks poorly align with the practical scenarios of code completion tools. In this paper, we propose RepoMasterEval, a novel benchmark for evaluating code completion models constructed from real-world repositories. Each benchmark datum is generated by masking a code snippet (ground truth) from one source code file with existing test suites. To improve test accuracy of model generated code, we employ mutation testing to measure the effectiveness of the test cases and we manually crafted new test cases for those test suites with low mutation score. Our empirical evaluation on 10 state-of-the-art models shows that test argumentation is critical in improving the accuracy of the benchmark and RepoMasterEval is able to report variance in model performance in real-world scenarios. The deployment of RepoMasterEval also revealed that the benchmark is useful to give accurate feedback during model training and the score is in high correlation with the model’s performance in practice.

With the blossoming of online social networks (OSN), personality analysis based on OSN texts has gained much research attention in recent years. The previous methods mainly focus on human-designed features extracted through psychological dictionaries or semantic features extracted through language models. However, the shallow statistics features can not fully convey the personality information and the language models can not capture enough psychological background knowledge. Besides, the lack of large labeled datasets has been a serious obstacle impending further research. To tackle these problems, we propose a personality analysis model, namely PerKG, which combines personality knowledge graph and heterogeneous graph representation learning to exploit external knowledge from psycholinguistics and learn the group-level information to predict users’ personalities accurately. Specifically, we construct a personality knowledge graph based on existing psycholinguistics knowledge. And then, for each user, we align the user information with the knowledge graph to obtain the personality heterogeneous graph. Finally, the personality vector of each entity node is learned for prediction by designing a walk strategy on the personality heterogeneous graph. Detailed experimentation shows that our proposed PerKG architecture can effectively improve the performance and alleviate the label sparsity problem of personality analysis.