A less complex and more straightforward program is a crucial factor that enhances its maintainability and makes writing secure and bug-free programs easier. However, due to its heavy workload and the risks of breaking the working programs, programmers are reluctant to do code refactoring, and thus, it also causes the loss of potential learning experiences. To mitigate this, we demonstrate the application of using a large language model (LLM), GPT-3.5, to suggest less complex versions of the user-written Python program, aiming to encourage users to learn how to write better programs. We propose a method to leverage the prompting with few-shot examples of the LLM by selecting the best-suited code refactoring examples for each target programming problem based on the prior evaluation of prompting with the one-shot example. The quantitative evaluation shows that 95.68% of programs can be refactored by generating 10 candidates each, resulting in a 17.35% reduction in the average cyclomatic complexity and a 25.84% decrease in the average number of lines after filtering only generated programs that are semantically correct. Further-more, the qualitative evaluation shows outstanding capability in code formatting, while unnecessary behaviors such as deleting or translating comments are also observed.

A popular language model that can solve introductory programming problems, OpenAI’s Codex, has drawn much attention not only in the natural language processing field but also in the software engineering field. It supports programmers by suggesting the next tokens to write, and it can even generate a whole function definition from a document string. We focus on its capability of automatically solving programming problems through code generation from problem descriptions. We investigate the model’s sensitivity to problem descriptions by formatting and modifying them. The experimental results show that the more explicitly formatted problem description enhances the code generation performance from 30.9% (raw) to 39.9% (formatted). Additionally, we observe that code generation relies on information specified in the problem description, such as variable names and constant values, as anonymizing them reduces the performance significantly. Moreover, statistical biases in code generation are identified, such as the generated programs ignoring the problem modification and answering the exact opposite problem. The changes in accuracy across formats suggest that the model does not correctly understand the natural language explaining the problem specification even if the model could solve the programming problems with high accuracy.