Existing code generation benchmarks, such as HumanEval, primarily evaluate universal algorithmic problems that do not address region-localized requirements. In realworld software development, localized knowledge, such as legal systems, cultural practices, and local language-specific processing is often required. To address this gap, we propose SakuraEval, the first region-localized code generation benchmark designed to evaluate Large Language Models, (LLMs,) performance in generating code that requires Japanese-localized knowledge. As a demonstration, of its effectiveness, we evaluated five LLMs with different language backgrounds using SakuraEval alongside HumanEval and JHumanEval. The results showed that all models exhibited significantly different tendencies in SakuraEval compared to existing benchmarks, with notable changes in their rankings that highlight each model’s Japanese-localized capabilities. SakuraEval provides valuable insights into the development of region-aware LLMs and underscores the importance of localized knowledge in code generation tasks.

ChatGPT presents emerging opportunities in software development, yet its capabilities for understanding code remain largely understudied. This study aims to focus on the logical aspect of code comprehension of ChatGPT by examining its performance in detecting and fixing bugs. Our preliminary results suggest that ChatGPT seems to correct code in a different way than human logical steps.