Autonomous coding agents are reshaping software development by creating pull requests (PRs) on GitHub, referred to as agentic PRs. In parallel, the review process is also becoming autonomous, thereby making reviewer bots key actors in the assessment of these agentic PRs. However, their influence on PR acceptance and resolution remains unclear. This study empirically investigates the relationship between reviewer-bot feedback and PR outcomes by analyzing how Reviewer Bot Feedback Quality (relevance, clarity, conciseness) and Reviewer Bot Activity Volume (comment count) are associated with PR acceptance and resolution time. We analyze 7,416 reviewer-bot comments on 4,532 PRs from the AI_Dev dataset (a dataset that captured AI agents’ PRs in GitHub projects). Our results show that reviewer-bot comments mainly focus on bug fixes, testing, and documentation, are civil in tone, and are prescriptive in nature. Reviewer bots generally produce clear and concise feedback, though the semantic relevance of comments to underlying code changes is moderate. We find that higher Reviewer Bot Activity volume is associated with longer PR resolution times and lower average feedback quality, showing that as bots generate more comments on a PR, the average pertinence of that feedback appears to degrade. At the same time, Reviewer Bot Feedback Quality shows no meaningful association with workflow outcomes. Our findings suggest that, in agentic PR workflows, reviewer bots should prioritize targeted high-relevance feedback over generating large numbers of comments.

Large Language Models are being extensively used for AI-assisted programming and code generation. The challenge is to ensure that the generated code is not only functionally correct but also safe, reliable and trustworthy. In this direction, we conduct a comprehensive empirical analysis of AI-generated code to assess whether large language models (LLMs) can produce correct and higher-quality code than humans. We evaluate the code quality of 984 code samples generated by GPT-3.5-Turbo and GPT-4 using various prompt types (simple, instructional, and enhanced) against input queries from the HumanEval dataset. We also enhance the HumanEval benchmark by calculating code quality metrics for the human-written code it contains. Code quality metrics are calculated using established tools like Radon, Bandit, Pylint, and Complexipy, with human-written code serving as a baseline for comparison. To quantify performance, we employ the TOPSIS method to rank the models and human code by their proximity to ideal and anti-ideal code quality metrics. Our results demonstrate that GPT-4, when used with advanced prompts, produces code closest to the ideal solution, outperforming human-written code in several key metrics. Our work provides evidence that LLMs, when properly guided, can surpass human developers in generating high-quality code. Our code and datasets are available online.

Accurate detection of semantic code clones has many applications in software engineering but is challenging because of lexical, syntactic, or structural dissimilarities in code. CodeBERT, a popular deep neural network based pre-trained code model, can detect code clones with a high accuracy. However, its performance on unseen data is reported to be lower. A challenge is to interpret CodeBERT's clone detection behavior and isolate the causes of mispredictions. In this paper, we evaluate CodeBERT and interpret its clone detection behavior on the SemanticCloneBench dataset focusing on Java and Python clone pairs. We introduce the use of a black-box model interpretation technique, SHAP, to identify the core features of code that CodeBERT pays attention to for clone prediction. We first perform a manual similarity analysis over a sample of clone pairs to revise clone labels and to assign labels to statements indicating their contribution to core functionality. We then evaluate the correlation between the human and model's interpretation of core features of code as a measure of CodeBERT's trustworthiness. We observe only a weak correlation. Finally, we present examples on how to identify causes of mispredictions for CodeBERT. Our technique can help researchers to assess and fine-tune their models' performance.

Developers spend a lot of time online, searching for code to help them implement their desired features. While code recommenders help improve developers’ productivity, there is currently no support for context-aware code recommendation for opportunistic code reuse on-the-go. Typical code recommendation systems provide recommendations against a search query, whereas a code recommender that supports opportunistic reuse can recommend related code snippets that represent features that the developer may want to implement next. In this paper, we present a novel Context-aware Feature-driven API usage-based Code Recommender (CA-FACER) tool, which is an Intellij IDEA plugin that leverages a developer’s development context to recommend related code snippets. We consider the methods having API usages in a developer’s active project as part of the development context. Our approach uses contextual data from a developer’s active project to find similar projects and recommends code from popular features of those projects. The popular features are identified as frequently occurring API usage based Method Clone Classes. From our experimental evaluation on 120 Android Java projects from GitHub, we observe a 46% improvement of precision using our proposed context-aware approach over a baseline system. Our technique recommends related code examples with an average precision ([email protected]) of 94% and 83% and a success rate of 90% and 95% for initial and evolved development stages respectively. A video demonstration of our tool is available at https://youtu.be/UjuM8WRc318.

Android developers often need to search for example code to complete their development tasks. While existing code search systems for Android can deliver code against a search query, they do not recommend code for features that a developer might later need to implement. In this paper, we present FACER-AS (FACER for Android Studio); an Android Studio plugin, which uses FACER (Feature-driven API usage-based Code Examples Recommender) as its back-end code search and recommendation engine. FACER provides relevant code against natural language queries (Stage 1) and also recommends code of multiple related features (Stage 2) to facilitate opportunistic code reuse. To evaluate FACER-AS, we perform a user study involving one professional Android developer who uses our tool for the development of their ongoing live Android projects. We analyze the developer's usage of our tool over a span of seven days and find that FACER-AS achieves a 79% success rate for retrieving code against user queries (Stage 1) and a 41% success rate for recommending code for related features (Stage 2). We also observe a 43% reuse rate of Stage 1 recommendations and a 45% reuse rate of Stage 2 recommendations. Our tool's performance analysis and the developer's positive feedback show that FACER-AS can help Android developers with their coding activities. A video demonstration of our tool is available at https://youtu.be/3yN-39wP_FU and the source code of our tool is available at https://doi.org/10.5281/zenodo.5176816.

Developers need to be able to find reusable code for desired software features in a way that supports opportunistic programming for increased developer productivity. Our objective is to develop a recommendation system that provides a developer with function recommendations having functionality relevant to her development task. We employ a combination of information retrieval, static code analysis and data mining techniques to build the proposed recommendation system called FACER (Feature-driven API usage-based Code Examples Recommender). We performed an experimental evaluation on 122 projects from GitHub from selected categories to determine the accuracy of the retrieved code for related features. FACER recommended functions with a precision of 54% and 75% when evaluated using automated and manual methods respectively.

Software design patterns provide well-known solutions to recurring problems faced by developers. However, not all developers are proficient with the use of design patterns. To facilitate developers in this regard, we propose a design pattern usage examples retrieval system. The retrieval is based on analyzing the developer's development context to provide design pattern usage examples from projects that have a similar functional domain to that of the developer. In our approach, the Latent Dirichlet Allocation model is used to extract domain keywords from individual projects' source code. The domain keywords are stored in a fact repository and are used to construct the respective projects' profiles. We perform content-based retrieval by using a developer's domain keywords to retrieve only matching projects and their associated design patterns-based code (as design pattern usage examples) from the fact repository. Quantitative and qualitative analysis of the results from internal experiments and a user study indicate that our system provides fairly accurate results.

Teaching refactoring effectively while making students realize the importance and benefits of refactoring is a challenge. In this direction, an experiment was carried out while conducting the course project for the Refactoring and Design Patterns course. This paper discusses the results of the experiment that involved two different project schemes to carry out refactoring activities on the same code base. One scheme was post-enhancement refactoring and the other was pre-enhancement refactoring. The aim of the experiment was to decide which scheme was beneficial in terms of better understanding, appreciation, and implementation of refactoring.