We evaluate ChatGPT’s ability to detect and fix cryptographic API misuses. First, we show that GPT-4o, when guided by engineered prompts, achieves F1 scores above 0.90 on two established benchmarks. Next, we assess the model on Java samples collected from GitHub repositories and Android apps. On the GitHub samples, it attains an F1 score of 0.84 and a Matthews Correlation Coefficient (MCC) of 0.81; on the Android samples, it achieves the F1 of 0.85 but a slightly lower MCC of 0.76. We found that factors beyond the misuse type such as file, method, and variable names; file size; code comments/context; the complexity of API usage (e.g., basic vs. interprocedural, field-sensitive, path-sensitive); and the number of misuses per file, affect ChatGPT’s detection performance. For comparison, we apply a leading crypto misuse detector to the Android apps and observe lower performance (F1 = 0.81, MCC = 0.68). We conclude that ChatGPT is a reliable tool for uncovering cryptographic misuses. Moreover, when it flags a misuse, it can suggest effective fixes. We reported identified cryptographic API misuses in GitHub repositories, each with GPT-generated fix suggestions. Maintainers responded to 23 misuses, with 21 confirming the issues and agreeing with the proposed fixes. However, several declined to implement changes due to legacy compatibility, hardware limitations, or because the code serves as a tutorial or sample. In two cases, developers did not consider the identified misuses to be security issues, as the code was intended for non-sensitive data. • ChatGPT has a promising performance in cryptographic misuse detection • ChatGPT, guided by engineered prompts, outperforms the leading static crypto misuse detector • ChatGPT’s recommendations for crypto misuses are mostly correct and secure.

Many developers rely on Large Language Models (LLMs) to facilitate software development. Nevertheless, these models have exhibited limited capabilities in the security domain. We introduce LLMSecGuard, a framework to offer enhanced code security through the synergy between static code analyzers and LLMs. LLMSecGuard is open source and aims to equip developers with code solutions that are more secure than the code initially generated by LLMs. This framework also has a benchmarking feature, aimed at providing insights into the evolving security attributes of these models.

The correct adoption of cryptography APIs is challenging for mainstream developers, often resulting in widespread API misuse. Meanwhile, cryptography misuse detectors have demonstrated inconsistent performance and remain largely inaccessible to most developers. We investigated the extent to which ChatGPT can detect cryptography misuses and compared its performance with that of the state-of-the-art static analysis tools. Our investigation, mainly based on the CryptoAPI-Bench benchmark, demonstrated that ChatGPT is effective in identifying cryptography API misuses, and with the use of prompt engineering, it can even outperform leading static cryptography misuse detectors.

Cryptography misuses are prevalent in the wild. Crypto APIs are hard to use for developers, and static analysis tools do not detect every misuse. We developed SafEncrypt, an API that streamlines encryption tasks for Java developers. It is built on top of the native Java Cryptography Architecture, and it shields developers from crypto complexities and erroneous low-level details. Experiments showed that SafEncrypt is suitable for developers with varying levels of experience.

Cryptography is known as a challenging topic for developers. We studied StackOverflow posts to identify the problems that developers encounter when using Java Cryptography Architecture (JCA) for symmetric encryption. We investigated security risks that are disseminated in these posts, and we examined whether ChatGPT helps avoid cryptography issues. We found that developers frequently struggle with key and IV generations, as well as padding. Security is a top concern among developers, but security issues are pervasive in code snippets. ChatGPT can effectively aid developers when they engage with it properly. Nevertheless, it does not substitute human expertise, and developers should remain alert.

Background. C# maintains type safety and security by not allowing direct dangerous pointer arithmetic. To improve performance for special cases, pointer arithmetic is provided via an unsafe context. Programmers can use the C# unsafe keyword to encapsulate a code block, which can use pointer arithmetic. In the Common Language Runtime (CLR), unsafe code is referred to as unverifiable code. It then becomes the responsibility of the programmer to ensure the encapsulated code snippet is not dangerous. Naturally, this raises concern on whether such trust is misused by programmers when they promote the use of C# unsafe context. Aim. We aim to analyze the prevalence and vulnerabilities of share code examples using C# unsafe keyword in Stack Overflow (SO) code sharing platform. Method. By using some regular expressions and manual checks, we extracted C# unsafe code relevant posts from SO and categorized them into some software development scenarios. Results. In the entire SO data dump of September 2018, we find 2,283 C# snippets with the unsafe keyword. Among those posts, 27% of posts are about Image processing, where unsafe codes are mainly used for performance reasons. The second most popular category by 21% of the codes in the posts is used for 'Interoperability' reasons. That is 'unsafe' is used to enable 'Interoperability' between C# managed codes and unmanaged codes. The 'stackalloc' operator is the third category with 9% of unsafe code posts. The stackalloc operator allocates a block of memory on the stack. Since C# 7.2, Microsoft recommends against using 'stackalloc' in unsafe context whenever possible. Manual inspection shows 67 code snippets with dangerous functions that can introduce vulnerability if not used with caution (e.g., buffer overflow). Finally, 35% of 'Interoperability' posts have 'P/Invoke' tag were used outside NativeMethods class, which is in contrast to Microsoft design suggestion. Conclusion. Our study leads to 7 main findings, and these findings show the importance of cautiously using this feature.