This paper investigates an important question in the era where AI can autonomously generate code and create pull requests: do AI-generated pull requests get rejected more than human-written ones and why do they get rejected? This investigation will help us understand how AI can support the open source community and its limitations. We conduct an in-depth examination of 1,779 rejected PRs (594 AI-generated and 1,185 human-authored), supported by 19,450 review comments and 28,875 review events. We further construct a ten-category taxonomy of rejection reasons grounded in reviewers’ explanations and code-level evidence. Our comparative analysis reveals similarity and differences in failure patterns: AI-generated PRs are more likely to be rejected due to incomplete implementation issues and testing-inadequate issues, human PRs showed a highly similar composition of rejection categories. However, the proportion of the other category is higher for human PRs than for AI PRs. This suggests that rejections of human PRs more often involve heterogeneous or context-specific reasons that do not fall neatly into predefined categories, whereas AI-generated PRs tend to fail in more repetitive and well-defined ways. Failure modes in AI-generated PRs are also more consistent and predictable, with description-level features providing stronger predictive signals compared to human-authored PRs. These findings provide empirical insights for improving AI coding assistants and open new opportunities for designing automated mechanisms and PR-review support tools tailored to AI-generated code.

Machine learning (ML) has gained much attention and has been incorporated into our daily lives. While there are numerous publicly available ML projects on open source platforms such as GitHub, there have been limited attempts in filtering those projects to curate ML projects of high quality. The limited availability of such a high-quality dataset poses an obstacle to understanding ML projects. To help clear this obstacle, we present NICHE, a manually labelled dataset consisting of 572 ML projects. Based on the evidence of good software engineering practices, we label 441 of these projects as engineered and 131 as non-engineered. This dataset can help researchers understand the practices that are adopted in high-quality ML projects. It can also be used as a benchmark for classifiers designed to identify engineered ML projects.

Artificial Intelligence (AI) systems, which benefit from the availability of large-scale datasets and increasing computational power, have become effective solutions to various critical tasks, such as natural language understanding, speech recognition, and image processing. The advancement of these AI systems is inseparable from open-source software (OSS). Specifically, many benchmarks, implementations, and frameworks for constructing AI systems are made open source and accessible to the public, allowing researchers and practitioners to reproduce the reported results and broaden the application of AI systems. The development of AI systems follows a data-driven paradigm and is sensitive to hyperparameter settings and data separation. Developers may encounter unique problems when employing open-source AI repositories.This paper presents an empirical study that investigates the issues in the repositories of open-source AI repositories to assist developers in understanding problems during the process of employing AI systems. We collect 576 repositories from the PapersWithCode platform. Among these repositories, we find 24,953 issues by utilizing GitHub REST APIs. Our empirical study includes three phases. First, we manually analyze these issues to categorize the problems that developers are likely to encounter in open-source AI repositories. Specifically, we provide a taxonomy of 13 categories related to AI systems. The two most common issues are runtime errors (23.18%) and unclear instructions (19.53%). Second, we see that 67.5% of issues are closed. We also find that half of these issues resolve within four days. Moreover, issue management features, e.g., label and assign, are not widely adopted in open-source AI repositories. In particular, only 7.81% and 5.9% of repositories label issues and assign these issues to assignees, respectively. Finally, we empirically show that employing GitHub issue management features and writing issues with detailed descriptions facilitate the resolution of issues. Based on our findings, we make recommendations for developers to help better manage the issues of open-source AI repositories and improve their quality.