We share our experience in developing Code Understanding Linter, an automated code review tool based on language models of code. We introduce several ideas to make the tool be more practical, including combining two different language models, filtering meaningless outputs from the model, and generating developer-friendly diagnosis messages by interpreting the outputs from the model. On top of those ideas, we describe the design and implementation of an automated code review tool to detect variable-misuse defects in Python codes and suggest how to fix them. We evaluated the tool with a set of code repositories in Samsung Electronics, which contains real-world Python codes. Our experiment proves that our tool can discover hidden defects in the real-world codes, but the false positive rate is far higher than we expected. After manually investigating every false positives, we discuss the limitations of the language models and possible solutions.

Code Review Automation can reduce human efforts during code review by automatically providing valuable information to reviewers. Nevertheless, it is a challenge to automate the process for large-scale companies, such as Samsung Electronics, due to their complexity: various development environments, frequent review requests, huge size of software, and diverse process among the teams.