As Large Language Models (LLMs) have become capable of generating long and descriptive code summaries, accurate and reliable evaluation of factual consistency has become a critical challenge.However, previous evaluation methods are primarily designed for short summaries of isolated code snippets.Consequently, they struggle to provide fine-grained evaluation of multi-sentence functionalities and fail to accurately assess dependency context commonly found in real-world code summaries.To address this, we propose ReFEree, a referencefree and fine-grained method for evaluating factual consistency in real-world code summaries.We define factual inconsistency criteria specific to code summaries and evaluate them at the segment level using these criteria along with dependency information.These segment-level results are then aggregated into a fine-grained score.We construct a code summarization benchmark with human-annotated factual consistency labels.The evaluation results demonstrate that ReFEree achieves the highest correlation with human judgment among 13 baselines, improving 15-18% over the previous state-of-the-art.Our code and data are available at https: //github.com/bsy99615/ReFEree.git.

YunSeok Choi, CheolWon Na, Jee-Hyong Lee. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Automatic processing of source code, such as code clone detection and software vulnerability detection, is very helpful to software engineers.Large pre-trained Programming Language (PL) models (such as CodeBERT, Graph-CodeBERT, CodeT5, etc.), show very powerful performance on these tasks.However, these PL models are vulnerable to adversarial examples that are generated with slight perturbation.Unlike natural language, an adversarial example of code must be semantic-preserving and compilable.Due to the requirements, it is hard to directly apply the existing attack methods for natural language models.In this paper, we propose DIP (Dead code Insertion based Blackbox Attack for Programming Language Model), a high-performance and efficient black-box attack method to generate adversarial examples using dead code insertion.We evaluate our proposed method on 9 victim downstream-task large code models.Our method outperforms the state-of-the-art black-box attack in both attack efficiency and attack quality, while generated adversarial examples are compiled preserving semantic functionality.