Recently, large language models (LLMs) have shown strong potential in code generation tasks. However, there are still gaps before they can be fully applied in actual software development processes. Accurately assessing the code generation capabilities of large language models has become an important basis for evaluating and improving the models. Some existing works have constructed datasets to evaluate the capabilities of these models. However, the current evaluation process may encounter the illusion of “Specialist in Familiarity”, primarily due to three gaps: the exposure of target code, case timeliness, and dependency availability. The fundamental reason for these gaps is that the code in current datasets may have been extensively exposed and exercised during the training phase, and due to the continuous training and development of LLM, their timeliness has been severely compromised. The key to solve the problem is to, as much as possible, evaluate the LLMs using code that they have not encountered before. Thus, the fundamental idea in this paper is to draw on the concept of code obfuscation, changing code at different levels while ensuring the functionality and output. To this end, we build a code-obfuscation based benchmark OBFusEvAL. We first collect 1,354 raw cases from five real-world projects, including function description and code. Then we use three-level strategy (symbol, structure and semantic) to obfuscate descriptions, code and context dependencies. We evaluate four LLMs on Obfu-sevaland compared the effectiveness of different obfuscation strategy. We use official test suites of these projects to evaluate the generated code. The results show that after obfuscation, the average decrease ratio of test pass rate can up to 62.5%.

High-precision static analysis can effectively detect Null Pointer Dereference (NPD) vulnerabilities in C language, but the performance overhead is significant. In recent years, researchers have attempted to enhance the efficiency of static analysis by leveraging multicore resources. However, due to complex dependencies in the analysis process, the parallelization of static value-flow NPD analysis for large-scale software still faces significant challenges. It is difficult to achieve a good balance between detection efficiency and accuracy, which impacts its application.This paper presents PANDA, the first parallel detector for high-precision static value-flow NPD analyzer in the C language. The core idea of PANDA is to utilize dependency analysis to ensure high precision while decoupling the strong dependencies between static value-flow analysis steps. This transforms the traditionally challenging-to-parallelize NPD analysis into two parallelizable algorithms: function summarization and combined query-based vulnerability analysis. PANDA introduces a task-level parallel framework and enhances it with a dynamic scheduling method to parallel schedule the above two key steps, significantly improving the performance and scalability of memory vulnerability detection.Fully implemented within the LLVM framework (version 15.0.7), PANDA demonstrates a significant advantage in balancing accuracy and efficiency compared to current popular open-source detection tools. In precision-targeted benchmark tests, PANDA maintains a false positive rate within 3.17% and a false negative rate within 5.16%; in historical CVE detection rate tests, its recall rate far exceeds that of comparative open-source tools. In performance evaluations, compared to its serial version, PANDA achieves up to an 11.23-fold speedup on a 16-node server, exhibiting outstanding scalability.

Taint analysis of value flows, as a static analysis technique, has gained widespread application in the fields of software security and vulnerability mining. However, when dealing with complex programs, it still faces challenges in terms of precision and performance. This research proposes P-DATA, a parallel framework implementing dependency-aware taint analysis. P-DATA employs modeling to capture data and control dependencies, reducing false positives over tools like Clang Static Analyzer and SVF. To accelerate the analysis, P-DATA leverages a task-level parallel framework introducing Preemption of Computational Resources (PCR) and Asynchronous Taint Source Registration, lead to impressive scalability and efficiency. Evaluations demonstrate P-DATA's ability to significantly expedite taint analysis for large programs using multi-core resources, achieving over 25X speedup on 32 cores. P-DATA makes notable contributions by boosting precision, efficiency and scalability of security-critical program analysis through advanced dependency modeling and paral-lelization techniques. It provides an extensible high-performance framework benefiting static analysis advancement.

Memory vulnerability detection aims to identify software defects that can compromise memory safety. However, existing methods often struggle to achieve both high precision and efficiency. This paper presents a high-precision memory vulnerability detection approach based on value flow analysis and parallel computing. We first construct a static semantic representation called SVFG to enable precise detection of memory vulnerabilities such as null pointer dereference and use-after-free. We then perform dependency-aware path feasibility analysis using an SMT solver to reduce false positives. Finally, we develop a task-level parallel framework to accelerate the constraint solving process and improve efficiency.We evaluate our approach on the Juliet test set of over 2,000 test cases and 7 open-source projects. Experimental results show that our dependency-aware analysis can achieve 0.5%-2.05% false positive rates, outperforming traditional approaches and existing tools. Our task-level parallel framework can achieve up to 3.25x speedup with 4 computing nodes.Our study demonstrates that combining value flow analysis and parallel computing is a promising way to enable highly precise and efficient detection of memory vulnerabilities. For future work, we plan to integrate pointer analysis to support more complex codes, and optimize the granularity of parallelism to improve scalability. Overall, this paper presents a static analysis based method to address the inherent trade-off between precision and efficiency in memory vulnerability detection.

Code search aims to search for relevant code snippets through queries, which has become an essential requirement to assist programmers in software development. With the availability of large and rapidly growing source code repositories covering various languages, multilingual code search can leverage more training data to learn complementary information across languages. Contrastive learning can naturally understand the similarity between functionally equivalent code across different languages by narrowing the distance between objects with the same function while keeping dissimilar objects further apart. Some works exist addressing monolingual code search problems with contrastive learning, however, they mainly exploit every specific programming language’s textual semantics or syntactic structures for code representation. Due to the high diversity of different languages in terms of syntax, format, and structure, these methods limit the performance of contrastive learning in multilingual training. To bridge this gap, we propose a unified semantic graph representation approach toward multilingual code search called MulCS. Specifically, we first design a general semantic graph construction strategy across different languages by Intermediate Representation (IR). Furthermore, we introduce the contrastive learning module integrated into a gated graph neural network (GGNN) to enhance query-multilingual code matching. The extensive experiments on three representative languages illustrate that our method outperforms state-of-the-art models by 10.7% to 77.5% in terms of MRR on average.

When programmers retrieve a code method and want to reuse it, they need to understand the usage patterns of the retrieved method. However, it is difficult to obtain usage information of the retrieved method since this method may only have a brief comment and few available usage examples. In this paper, we propose an approach, called LUPIN (cLone-based Usage Pattern mIniNg), to mine the usage patterns of these methods, which do not widely appeared in the code repository. The key idea of LUPIN is that the cloned code of the target method may have a similar usage pattern, and we can collect more usage information of the target method from cloned code usage examples. From the amplified usage examples, we mine the usage pattern of the target method by frequent subsequence mining after program slicing and code normalization. Our evaluation shows that LUPIN can mine four categories of usage patterns with an average precision of 0.65.