Recent advances in large language models (LLMs) and AI agents have enabled automated test generation to become a practical component of modern software development workflows. However, these systems frequently produce tests that fail to compile or execute correctly, raising questions about their robustness in real-world settings. In this study, we conduct an empirical analysis of test failures in the AIDev dataset from the MSR 2026 Mining Challenge. Particularly, we examine 500 merged pull requests containing test-related failures (100 per agent across 5 AI coding agents) and classify each case using a refined taxonomy that distinguishes between compile-time and runtime errors. Our findings reveal that runtime errors dominate (62.6% vs 37.4% compile-time), with assertion failures being the most common issue (28.6%). The analysis exposes distinct failure profiles across agents and provides insights for improving AI-driven testing reliability in large-scale development environments. Our replication package is available at: https://figshare.com/s/005a3942818fba893a7e.

This study investigates the readability of AI-generated pull request (PR) descriptions, a critical factor for enabling programmers to efficiently understand and integrate code changes. We analyze a large-scale and diverse dataset, AIDev, which contains AI-generated PR descriptions spanning multiple PR types, including bug fix, feature, documentation, test, dependency, and refactor changes. Readability is measured using the Flesch Reading Ease metric, complemented by a quantitative rubric-based evaluation to assess completeness. We compare readability and completeness across PR types and AI agents. Our results reveal systematic differences in both readability and completeness that are influenced by the PR type as well as the underlying AI model. These findings provide empirical evidence of current AI models’ performance in generating PR documentation, highlight limitations in existing approaches, and identify opportunities to improve the design of AI-assisted tools to support clearer, more complete, and more effective developer workflows.

Android apps continuously evolve to meet user expectations and thrive in the competitive environment of app stores. Hence, making informed decisions is crucial for the success of upcoming releases. In recent years, researchers have sought to aid developers in release planning by studying, analyzing, and modeling evolutionary information derived from tracking releases of various apps. They have demonstrated how the types of information provided on different platforms can effectively predict and assess the quality or success of an app. Nevertheless, this field needs a comprehensive dataset containing evolutionary information that tracks various releases of open-source Android apps. Existing datasets lack release-level information, are not publicly available, or do not cover a wide range of up-to-date apps. This paper introduces a dataset comprising diverse metadata adapted from GitHub and Google Play, along with impactful metrics extracted from the source code of 8,041 published releases of 1,363 open-source Android apps.

Jupyter notebooks have become indispensable tools for data analysis and processing in various domains. However, despite their widespread use, there is a notable research gap in understanding and analyzing the contents and code metrics of these notebooks. This gap is primarily attributed to the absence of datasets that encompass both Jupyter notebooks and extracted their code metrics. To address this limitation, we introduce DistilKaggle, a unique dataset specifically curated to facilitate research on code metrics in Jupyter notebooks, utilizing the Kaggle repository as a prime source. Through an extensive study, we identify thirty-four code metrics that significantly impact Jupyter notebook code quality. These features such as lines of code cell, mean number of words in markdown cells, performance tier of developer, etc., are crucial for understanding and improving the overall effectiveness of computational notebooks. The DistilKaggle dataset which is derived from a vast collection of notebooks constitutes two distinct datasets: (i) Code Cells and Markdown Cells Dataset which is presented in two CSV files, allowing for easy integration into researchers' workflows as dataframes. It provides a granular view of the content structure within 542,051 Jupyter notebooks, enabling detailed analysis of code and markdown cells; and (ii) The Notebook Code Metrics Dataset focused on the identified code metrics of notebooks. Researchers can leverage this dataset to access Jupyter notebooks with specific code quality characteristics, surpassing the limitations of filters available on the Kaggle website. Furthermore, the reproducibility of the notebooks in our dataset is ensured through the code cells and markdown cells datasets, offering a reliable foundation for researchers to build upon. Given the substantial size of our datasets, it becomes an invaluable resource for the research community, surpassing the capabilities of individual Kaggle users to collect such extensive data. For accessibility and transparency, both the dataset and the code utilized in crafting this dataset are publicly available at https://github.com/ISE-Research/DistilKaggle.

Platforms such as GitHub and GitLab introduce Issue Report Templates (IRTs) to enable more effective issue management and better alignment with developer expectations. However, these templates are not widely adopted in most repositories, and there is currently no tool available to aid developers in generating them. In this work, we introduce GIRT-Model, an assistant language model that automatically generates IRTs based on the developer's instructions regarding the structure and necessary fields. We create GIRT-Instruct, a dataset comprising pairs of instructions and IRTs, with the IRTs sourced from GitHub repositories. We use GIRT-Instruct to instruction-tune a T5-base model to create the GIRT-Model.

GitHub’s issue reports provide developers with valuable information that is essential to the evolution of a software development project. Contributors can use these reports to perform software engineering tasks like submitting bugs, requesting features, and collaborating on ideas. In the initial versions of issue reports, there was no standard way of using them. As a result, the quality of issue reports varied widely. To improve the quality of issue reports, GitHub introduced issue report templates (IRTs), which pre-fill issue descriptions when a new issue is opened. An IRT usually contains greeting contributors, describing project guidelines, and collecting relevant information. However, despite of effectiveness of this feature which was introduced in 2016, only nearly 5% of GitHub repositories (with more than 10 stars) utilize it. There are currently few articles on IRTs, and the available ones only consider a small number of repositories.In this work, we introduce GIRT-DATA, the first and largest dataset of IRTs in both YAML and Markdown format. This dataset and its corresponding open-source crawler tool are intended to support research in this area and to encourage more developers to use IRTs in their repositories. The stable version of the dataset contains 1,084,300 repositories and 50,032 of them support IRTs. The stable version of the dataset and crawler is available here: https://github.com/kargaranamir/girt-data