The rapid evolution of software libraries poses a considerable hurdle for code generation, necessitating continuous adaptation to frequent version updates while preserving backward compatibility. While existing code evolution benchmarks provide valuable insights, they typically lack execution-based evaluation for generating code compliant with specific library versions. To address this, we introduce GitChameleon 2.0, a novel, meticulously curated dataset comprising 328 Python code completion problems, each conditioned on specific library versions and accompanied by executable unit tests. GitChameleon 2.0 rigorously evaluates the capacity of contemporary large language models (LLMs), LLM-powered agents, code assistants, and RAG systems to perform version-conditioned code generation that demonstrates functional accuracy through execution. Our extensive evaluations indicate that state-of-the-art systems encounter significant challenges with this task; enterprise models achieving baseline success rates in the 48-51% range, underscoring the intricacy of the problem. By offering an execution-based benchmark emphasizing the dynamic nature of code libraries, GitChameleon 2.0 enables a clearer understanding of this challenge and helps guide the development of more adaptable and dependable AI code generation methods.

In the context of continually growing large code repositories where code refactoring is an ongoing requirement, we highlight the effectiveness of OpenRewrite as a tool for conducting large-scale code refactoring. OpenRewrite leverages Lossless Semantic Trees (LST) to represent code and applies recipes to search and implement changes. These recipes are openly available and can be executed locally or accessed through the Moderne platform for public repositories. We provide a concise overview of the underlying technology, instructions for utilizing the tool, and we compare its performance against a manual approach and two prominent large language models (LLM): ChatGPT and StarChat-β. Our comparison is based on the execution time of the tool and the accuracy of the implemented changes. Additionally, we present three distinct use cases that demonstrate the versatile applications of the tool. A demonstration of OpenRewrite’s recipe which detects vulnerabilities and automatically fixes them is available at the following link: https://www.youtube.com/watch?v=L1-_cQUX-JA.