Automated test generation has a substantial body of work, yet most studies focus on generating tests for complete software units, such as classes, and rely on metrics such as code coverage for assessment. In contrast, modern software development primarily evolves through small, targeted changes introduced in pull requests (PRs). Despite this, the crucial task of generating tests specifically for these PRs has been overlooked, and the performance of state-of-the-art tools for this purpose remains unknown. This study evaluates two distinct approaches for PR-aware test generation: EvoSuite, a leading search-based tool, and GPT-4o, one of the widely used large language models (LLMs). To measure their effectiveness at validating PR-specific changes, we assess their ability to generate fail-to-pass (F2P) test cases, meaning tests that fail on the code before the change and pass on the code after the change. Our evaluation shows that EvoSuite outperformed GPT-4o, producing at least one F2P test for a significantly higher percentage of PRs (36 percent vs. 13 percent). The performance of GPT-4o was significantly hampered by a high rate of compilation errors (63 percent), whereas only 2 percent of EvoSuite's generated tests failed to run. Despite EvoSuite's relative success, our findings indicate that both tools are largely ineffective for this task, as they failed to generate any meaningful change-capturing tests for the large majority of the PRs (64 percent). Although both generators could not achieve a high F2P ratio in our evaluation, and EvoSuite outperformed GPT-4o, we believe that agentic code generation methods may have significant potential for this task. Ultimately, our work highlights a critical gap in tooling and calls for the development of high-performance test generators tailored to the incremental nature of modern software development.

Software artifacts contain crucial information about a project. Analyzing these artifacts and their relationships yields valuable insights. During a software project’s lifecycle, software tracking tools are used to monitor artifacts. Mining metadata from modern software tracking tools provides extensive data for constructing comprehensive software artifact traceability graphs. These graphs aid decision-making in software development. While prior studies have used various software artifact graphs for analysis, comprehensive graphs are underexplored. Moreover, existing studies often lack interactive visualization for exploratory analysis. A unified traceability graph with interactive visualization can illuminate a broader range of issues and enhance understanding through visual cues. This article introduces the Software Artifact Analyzer (SAA) framework, leveraging artifact traceability graphs to support diverse analyses. A sample SAA tool demonstrates framework implementation, evaluated through quantitative and qualitative methods with focus groups and surveys. Participants praised its potential to improve software processes but noted challenges in graph complexity management. Based on the surveys, the tool’s usability score was 74.5 out of 100, which is above average on the System Usability Scale (SUS), indicating its practicality. The SAA framework offers broad applicability by enabling seamless implementation of new software analysis methods, providing project decision-makers with insightful visualizations of the analysis results.

Background: Code comments are important in software development because they directly influence software maintainability and overall quality. Bad practices of code comments lead to code comment smells, negatively impacting software maintenance. Recent research has been conducted on classifying inline code comment smells, yet automatically detecting these still remains a challenge. Objective: We aim to automatically detect and classify inline code comment smells through machine learning (ML) models and a large language model (LLM) to determine how accurately each smell type can be detected. Method: We enhanced a previously labeled dataset, where comments are labeled according to a determined taxonomy, by augmenting it with additional code segments and their associated comments. GPT-4, a large language model, was used to classify code comment smells on both the original and augmented datasets to evaluate its performance. In parallel, we trained and tested seven different machine learning algorithms on the augmented dataset to compare their classification performance against GPT-4. Results: The performance of models—particularly Random Forest, which achieved an overall accuracy of 69%, along with Gradient Boosting and Logistic Regression, each achieving 66% and 65%, respectively —establishes a solid baseline for future research in this domain. The Random Forest model outperformed all other ML models, by achieving the highest Matthew’s Correlation Coefficient (MCC) score of 0.44. The augmented dataset improved the overall classification accuracy of the GPT-4 model’s predictions from 34% to 55%. Conclusion: This study contributes to software maintainability by exploring the automatic detection and classification of inline code comment smells. We have made our augmented dataset and code artifacts available online, offering a valuable resource for developing automated comment smell detection tools.

Context: Code comments play a critical role in improving code readability, maintainability, and collaborative development. However, comments may deviate from best practices due to software evolution, where code changes are not reflected in comments, as well as practitioner-related issues such as vague descriptions, redundancy, or misaligned intent. These issues lead to various comment smells that degrade software quality. While prior studies have explored comment inconsistencies, most are limited in scope, either addressing a narrow subset of smells or focusing solely on detection without considering repair.Objective: This study evaluates the effectiveness of large language models (LLMs) in both detecting and repairing inline code comment smells, using a comprehensive taxonomy of code comment smell types.Method: We extended a prior data set by incorporating repaired versions of smelly comments, resulting in 2,211 unique instances. Four LLMs—GPT-4o-mini, o3-mini, DeepSeek-V3, and Codestral-2501—are evaluated under zero-shot and few-shot prompting strategies. To account for non-deterministic behavior in LLM outputs and ensure robustness, each configuration is executed five times. Detection performance is measured using accuracy, macro F1 score, and Matthews correlation coefficient (MCC); repair is evaluated using SBERT similarity, METEOR, and ROUGE-L. Our multi-stage pipeline feeds detection outputs into the repair phase, where the detection result with the highest macro F1 score is used to simulate the best possible repair scenario. Median scores across runs are reported for comparison.Results: o3-mini with few-shot prompting achieves the highest median detection performance: macro F1 of 0.41, MCC of 0.50, and accuracy of 0.72, exceeding the baseline of GPT-4. For repair, Codestral-2501 in the zero-shot setting yields the best results with a median SBERT score of 0.61, followed by DeepSeek-V3 and GPT-4o-mini at 0.53, and o3-mini at 0.46. Few-shot prompts improve detection, while zero-shot prompts are more effective for repair.Conclusion: Lightweight LLMs such as o3-mini can achieve strong detection performance when guided by effective few-shot prompts. For example, o3-mini with few-shot prompting attains the highest median detection results: macro F1 of 0.41, MCC of 0.50, and accuracy of 0.72, surpassing the GPT-4 baseline. In contrast, repair tasks benefit more from zero-shot prompting, though they introduce challenges such as overfitting and the risk of generating new smells. Our findings support the development of practical tools, including a GitHub-integrated comment repair assistant, and motivate future work on dynamic prompt selection and multilingual benchmark construction.

Reproducing game bugs, particularly crash bugs in continuously evolving games like Minecraft, is a notoriously manual, time-consuming, and challenging process to automate; insights from a key decision maker from Minecraft we interviewed confirm this, highlighting that a substantial portion of crash reports necessitate manual scenario reconstruction. Despite the success of LLM-driven bug reproduction in other software domains, games, with their complex interactive environments, remain largely unaddressed. This paper introduces BugCraft, a novel end-to-end framework designed to automate the reproduction of crash bugs in Minecraft directly from user-submitted bug reports, addressing the critical gap in automated game bug reproduction. BugCraft employs a two-stage approach: first, a Step Synthesizer leverages LLMs and Minecraft Wiki knowledge to transform bug reports into high-quality, structured steps to reproduce (S2R). Second, an Action Model, powered by a vision-based LLM agent and a custom macro API, executes these S2R steps within Minecraft to trigger the reported crash. To facilitate evaluation, we introduce BugCraft-Bench, a curated dataset of Minecraft crash bug reports. On BugCraft-Bench, our framework end-to-end reproduced 34.9% of crash bugs with GPT-4.1, outperforming baseline computer-use models by 37%. BugCraft demonstrates the feasibility of automated reproduction of crash bugs in complex game environments using LLMs, opening promising avenues for game testing and development. Finally, we make our code open at https://bugcraft2025.github.io.

The dependency graph (DG) of a software project offers valuable insights for identifying its key components and, hence has been leveraged in numerous studies. Nevertheless, there is a lack of reusable tools for DG extraction. Existing tools are either outdated and difficult to configure, or fail to provide accurate analysis. However, Integrated Development Environments (IDEs) are designed to address the above issues. This study introduces RefExpo2, a reusable DG extraction tool that supports multiple languages, such as Java, Python, and JavaScript. RefExpo is a plugin based on IntelliJ which is a wellmaintained and reputed IDE. In addition, we compile an initial version of our dataset consisting of 20 Java and Python projects. We evaluated RefExpo’s validity at two levels: specific language features and comparisons against other existing tools, which we refer to as the micro and macro levels. Our evaluation shows RefExpo achieving 92% and 100% recall on micro test suites Judge and PyCG for Python and Java, respectively. In macrolevel experiments, RefExpo outperformed existing tools by at least 31 % and 7 % in finding unique and shared results (nonoverlapping and overlapping with other tools). The installable version of RefExpo is available on the IntelliJ marketplace3. Additionally, a short video describing its functionality is available on YouTube1.1https://youtu.be/eCnPUlj6YgA

In software development, accurately identifying bug-inducing commits (BICs) is crucial for maintaining code integrity and ensuring the reliability of software systems. The complexities involved in pinpointing the exact commits responsible for bugs necessitate a thorough investigation of the underlying issues and limitations of existing tools and algorithms. This study investigates and identifies corner cases in BIC identification, clarifying definitions and examining issues with existing algorithms and tools. By analyzing these cases, we aim to reveal challenges faced by current methods and propose insights for future improvements. We evaluated the SZZ algorithm and two large language models, GPT-40 and Llama 3.1, using a curated repository of corner-case bugs with detailed reports. This setup allowed us to assess the strengths and weaknesses of both traditional algorithms and LLMs. The SZZ algorithm achieved a recall of 0.8 and a precision of 0.36, resulting in an F1 score of 0.5 for corner cases and a recall of 1 and a precision of 0.5 for non-corner cases with an F1 score of 0.67. In comparison, the LLMs showed varied performance: for corner cases, Llama had an MRR of 0.7, while GPT scored 0.5. For non-corner cases, both models performed better, with an MRR of 0.875. Corner cases in BIC identification expose limitations in current methods, emphasizing the need for improved approaches to accurately handle these challenges.

Traceability is important in the software development life cycle for managing the connections among various software artifacts, particularly pull requests (PRs) and issues. Developers often neglect to link these manually, reducing traceability. Existing algorithms to recover these links have limited application in closed-source projects. In this study, we share the experience of using ReLink, a predictive PR-issue linking tool with visualization capabilities, in a closed-source project environment. ReLink stands out due to its availability as a fully functional web application and its semi-automated nature, which enhances usability. ReLink determines missing links between PRs and issues based on a confidence score normalized between 0 and 100, calculated using text similarity and heuristic rules. The tool's effectiveness was evaluated in both an open-source project and an industry-based case study. In the open-source project, ReLink achieved a top-5 accuracy of 0.80 and a precision of 0.77. In the industrial case study, ReLink's effectiveness was validated by practitioners selecting the correct link from five issue suggestions, resulting in a top-5 accuracy of 0.86 and a mean reciprocal rank (MRR) of 0.84. Through this experience, we offer insights into both the benefits and challenges of implementing traceability link recovery in the industry and provide recommendations for practitioners seeking to bridge traceability gaps efficiently.

Abstract Context Software development comprises many processes, including Code Review (CR) and Bug Tracking (BT). Although no perfect practices exist, recognizing bad practices helps avoid detrimental habits in CR and BT. Objectives This study investigates CR and BT process smells in industrial settings using Smellyzer , a detection tool based on established smell taxonomies. We aimed to explore the challenges, insights, and outcomes of applying a smell detection tool in real-world software projects. Method We conducted an evaluative case study involving a large-scale proprietary company, analyzing CR and BT smells in a large software project using Smellyzer . We collected data through pre-surveys, focus groups, questionnaires, follow-up study and tool-based analyses, and then analyzed these data using triangulation and grounded theory methods. Results Practitioners confirmed the presence of CR and BT smells in their workflows and provided detailed feedback on root causes and their impacts on development efficiency. They also rated the tool’s usefulness at identifying CR smells as 5 out of 5 and BT smells as 4.6 out of 5. Its practicality, measured by the System Usability Scale, scored 77.5 for CR and 80.0 for BT smell detection. Conclusions This study highlights the presence of CR and BT smells in industrial settings and their potential to affect workflow efficiency and software quality. While our tool, Smellyzer , proved effective in identifying these smells, the findings emphasize the importance of integrating theoretical frameworks with practical solutions. Future work should explore these smells across diverse project contexts and investigate their influence on development practices over time.

Abstract Context Code reviewing is an essential yet challenging activity due to the potential repercussions associated with changes to the codebase. While version control highlights differences between versions, it may not adequately alert developers to potential side effects of changes. Change impact analysis alongside code differences can guide reviewers in making informed decisions. Objective We aim to enhance the code review process by providing helpful insights to reviewers by analyzing change impact. Method We propose a novel change impact analysis approach with the granularity of pull requests by combining call graph-based dependency analysis and history mining techniques. By utilizing these, we calculate several file metrics and an overall risk score for each pull request. To validate the approach, we conducted two focus group sessions, including a feature feedback survey, a tool demo, a post-demo survey, and focus group discussions. Additionally, we performed experiments on three open-source projects to evaluate the computational feasibility. Results From the focus groups, our approach received an average of 3.66 out of 5.0 for enhancing the code review experience across five aspects. The current risk score formula averaged 3.2 out of 5.0 for accurately representing change impact, with 71% agreement on the selected metrics. Experiments confirmed computational viability, with analysis times ranging from 7.4 to 22.43 seconds. Conclusions Our study underscores the potential benefits of integrating change impact analysis into the code review process. By leveraging this approach, developers can conduct more thorough reviews, enhancing their ability to detect and mitigate potential issues arising from code changes.

ABSTRACT Code review is a widely utilized practice that focuses on improving code via manual inspections. However, this practice is not addressed adequately in a typical software engineering curriculum. We aim to help address the code review practice knowledge gap between the software engineering curricula and the industry with a serious game approach. We determine our learning objectives around the introduction of the code review process. To realize these objectives, we design, build, and test the serious game. We then conduct three case studies with a total of 280 students. We evaluated the results by comparing the student's knowledge and confidence about code review before and after case studies, as well as evaluating how they performed in code review quizzes and game levels themselves. Our analysis indicates that students had a positive experience during gameplay, and an in‐depth examination suggests that playing the game also enhanced their knowledge. We conclude that the game had a positive impact on introducing the code review process. This study represents a step taken toward moving code review education from industry starting positions to higher education. The game and its auxiliary materials are available online.

Code review is a crucial component of the software development life cycle and is adopted as a best practice in the industry. However, like any process, counterproductive practices and pitfalls exist within code review, such as the occurrence of the “Looks Good to Me” (LGTM) smell. LGTM smell occurs when a superficial review is conducted. LGTM review smells can potentially result in the accidental inclusion of low-quality changesets in the codebase, leading to severe bugs, possibly many reopens of the associated issues, and additional time wasted on changes to the changeset. In this study, we aim to explore LGTM smells and examine their potential impacts on the related repository. Given the inherent challenges of automatically detecting LGTM smells in code reviews, this study introduces an alternative approach by categorizing code reviews into two distinct types: comment-free and commented reviews. The primary hypothesis is that comment-free reviews are more prone to LGTM smells due to their lack of detailed examination and discourse. To test this hypothesis, we conduct an empirical analysis on a subset of pull requests (PRs) comprising code reviews from five large-scale software projects. We further investigate the impact of comment-free and commented reviews on key development metrics, specifically focusing on the number of reopens and late commits in PRs. According to the results, 64.7% of the PRs in these projects exhibited comment-free reviews. Our manual analysis reveals that comment-free reviews exhibit the LGTM smell 3.5 times more frequently than the commented reviews. We also observed a statistically significant difference, indicating that comment-free review PRs tend to include more late commits (i.e., commits made after the reviewer's approval) than the commented PRs. However, no statistically significant difference was observed in the reopening ratio of associated issues between comment-free reviews and commented reviews. Our approach provides a novel method for detecting and exploring the impacts of LGTM smell, emphasizing the significance of comprehensive code reviews and setting the stage for future research aimed at automatically identifying LGTM smell occurrences.

Abstract Code comments play a vital role in source code comprehension and software maintainability. It is common for developers to write comments to explain a code snippet, and commenting code is generally considered a good practice in software engineering. However, low-quality comments can have a detrimental effect on software quality or be ineffective for code understanding. This study aims to create a taxonomy of inline code comment smells and determine how frequently each smell type occurs in software projects. We conducted a multivocal literature review to define the initial taxonomy of inline comment smells. Afterward, we manually labeled 2447 inline comments from eight open-source projects where half of them were Java, and another half were Python projects. We created a taxonomy of 11 inline code comment smell types and found out that the smells exist in both Java and Python projects with varying degrees. Moreover, we conducted an online survey with 41 software practitioners to learn their opinions on these smells and their impact on code comprehension and software maintainability. The survey respondents generally agreed with the taxonomy; however, they reported that some smell types might have a positive effect on code comprehension in certain scenarios. We also opened pull requests and issues fixing the comment smells in the sampled projects, where we got a 27% acceptance rate. We share our manually labeled dataset online and provide implications for software engineering practitioners, researchers, and educators.

GitHub Issues is a widely used issue tracking tool in open-source software projects. Originally designed with broad flexibility, its lack of standardization led to incomplete issue reports, impeding software development and maintenance efficiency. To counteract this, GitHub introduced issue templates in 2016, which rapidly became popular. Our study assesses the current use and evolution of these templates in large-scale open-source projects and their impact on issue tracking metrics, including resolution time, number of reopens, and number of issue comments. Employing a comprehensive analysis of 350 templates from 100 projects, we also evaluated over 1.9 million issues for template conformity and impact. Additionally, we solicited insights from open-source software maintainers through a survey. Our findings highlight issue templates’ extensive usage in 99 of the 100 surveyed projects, with a growing preference for YAML-based templates, a more structured template variant. Projects with a template exhibited markedly reduced resolution time (381.02 days to 103.18 days) and reduced issue comment count (4.95 to 4.32) compared to those without. The use of YAML-based templates further significantly decreased resolution time, the number of reopenings, and the discussion extent. Thus, our research underscores issue templates’ positive impact on large-scale open-source projects, offering recommendations for improved effectiveness.

In this study, we assess bug life cycles to identify bottlenecks in the bug tracking processes, and examine the effectiveness of bug tracking system usage practices linked to bug states and state transitions. To achieve this, we examined the bug life cycles of three open-source software projects which use Bugzilla as their bug tracking system. In total, we have analyzed 106.196 bugs gathered from these projects. We started by looking at the temporal and quantitative aspects of these projects’ bug life cycles. After that, we collected data about how bug life cycles differ over time. Finally, we inspected the frequency of reopened and state-looping bugs in these projects. After our analysis, we have deduced that the presented temporal and quantitative analysis of bug life cycles is useful for finding bottlenecks and undesired behaviors in the bug tracking processes. We also inferred that examining the changes in bug life cycles over time can provide insights into how bug tracking practices changed throughout the project’s lifetime, and it can be used as a parameter to assess whether the bug tracking system usage has improved. Lastly, we deducted that analyzing undesired state trails’ frequency provides insights into the performance of bug tracking processes. Based on the insights gained from analyzing bug life cycles with the presented methods, we believe that decision makers can improve their workflow by introducing or removing new states to the bug life cycle and adding new rules and restrictions to their bug tracking process.

Code reviews, while effective, can be crippled by process smells if not performed correctly. A typical process smell that harms the efficacy of code reviews is the ‘Looks Good To Me’ (LGTM) smell, wherein a reviewer approves a code review task without reviewing the code attentively. Low-quality code reviews can be harmful, as they can cause bugs to slip into a product codebase leading to potentially severe consequences. In this paper, we propose an innovative solution to potentially minimize the occurrence of the LGTM smell commonly found in code reviews. We built a tool that is a proof-of-concept implementation of our solution, which incorporates the concept of mutation testing into code reviews. It provides a platform where pull request authors can apply mutations to the pull request code in GitHub. Reviewer attention and review efficacy are measured based on their mutation score. To the best of our knowledge, our proof of concept implementation is the first-ever code review tool that uses the concept of mutation testing. We validated our proposed solution with eight developers and received promising results.

Bus factor (BF) is a metric that tracks knowledge distribution in a project. It is the minimal number of engineers that have to leave for a project to stall. Despite the fact that there are several algorithms for calculating the bus factor, only a few tools allow easy calculation of bus factor and convenient analysis of results for projects hosted on Git-based providers. We introduce Bus Factor Explorer, a web application that provides an interface and an API to compute, export, and explore the Bus Factor metric via treemap visualization, simulation mode, and chart editor. It supports repositories hosted on GitHub and enables functionality to search repositories in the interface and process many repositories at the same time. Our tool allows users to identify the files and subsystems at risk of stalling in the event of developer turnover by analyzing the VCS history. The application and its source code are publicly available on GitHub at https://github.com/JetBrains-Research/bus-factor-explorer. The demonstration video can be found on YouTube: https://youtu.be/uIoV79N14z8

Software projects experience the departure of developers due to various reasons. As developers are one of the main sources of knowledge in software projects, their absence will inevitably result in a certain degree of knowledge depletion. Bus Factor (BF) is a metric to evaluate how this knowledge loss can affect the project’s continuity. Conventionally, BF is calculated as the smallest set of developers, removing over half the project knowledge upon departure. Current state-of-the-art approaches measure developers’ knowledge by the number of authored files, utilizing version control system (VCS) information. However, numerous studies have shown that files in software projects have different significance. In this study, we explore how weighting files according to their significance affects the performance of two prevailing BF estimators. We derive significance scores by computing five well-known graph metrics from the project’s dependency graph: PageRank, In-/Out-/All-Degree, and Betweenness Centralities. Furthermore, we introduce BFSig , a prototype of our approach. Finally, we present a new dataset comprising reported BF scores collected by surveying software practitioners from five prominent Github repositories. Our results indicate that BFSig outperforms the baselines by up to an 18% reduction in terms of Normalized Mean Absolute Error (NMAE). Moreover, BFSig yields 18% fewer False Negatives in identifying potential risks associated with low BF. Besides, our respondent confirmed BFSig versatility by showing its ability to assess the BF of the project’s subfolders. In conclusion, we believe to estimate BF from authorship, software components of higher importance should be assigned heavier weight. Currently, BFSig exclusively explores the topological characteristics of these components. Nevertheless, considering attributes such as code complexity and bug proneness could potentially enhance the performance of BFSig.

Abstract Bug handling is an essential part of the software development process. Ideally, in a bug‐tracking system, bugs are reported, fixed, verified, and closed. In some cases, bugs have to be reopened mostly due to an incorrect fix. However, instead of reopening the existing bug report, users may submit a new report for a previously reported bug, which results in duplicate bug reports. Users might report duplicate bugs if they miss the previous bug report or if the previous bug is unresolved (i.e., in progress) and therefore cannot be reopened. These duplicate bug reports may cost extra maintenance efforts in triaging and bug fixing. There have been several studies on characterizing reopened bugs and duplicate bug reports; however, to the best of our knowledge, there has been no prior work on understanding the dynamics of their interaction, which ismissed reopenbugs. Our study is based on analyzing the difference between duplicate and nonduplicate bugs and further categorizing the duplicate bugs. In this regard, we categorize duplicate bugs according to the original bug's resolution status at the duplicate's creation time as Master‐Unresolved bugs and Master‐Resolved (Missed Reopen) bugs to distinguish their properties. We compare these two types of bugs in terms of their relationship to their master bug, bug surface time, bug fix time, bug's severity, and the number of users involved. We perform case studies using the Eclipse and Mozilla projects' bug repositories that include more than 165,500 and 394,000 bug reports, respectively. Additionally, we investigate the perceived importance, impact, and causes of duplicate bugs, as well as the difference between nonduplicate and duplicate bugs and its categories for practitioners in the software industry by conducting a survey.

Abstract Today's software industry is global, virtual, and depending more than ever on strong and reliable processes. Stakeholders and infrastructure are distributed across the globe, posing challenges that go beyond those with co‐located teams and servers. Software Engineering continues to be a complex undertaking, with projects challenged to meet expectations, especially regarding costs. We know that Software Engineering is an ever‐changing discipline, with the result that firms and their employees must regularly embrace new methods, tools, technologies, and processes. In 2020, the International Conference on Global Software Engineering (ICGSE) and the International Conference on Systems and Software Processes (ICSSP) joined forces aiming to create a holistic understanding of the software landscape both from the perspective of human and infrastructure distribution and also the processes to support software development. Unfortunately, these challenges have become even more personal to many more in 2020 due to the disruption introduced by the COVID‐19 pandemic, which forced both conferences to be held virtually. As an outcome of the joint event, we selected a set of the best papers from the two conferences, which were invited to submit extended versions to this Special Issue in the Journal of Software: Maintenance and Evolution. Dedicated committees were established to identify the best papers. Eight papers were invited and ultimately, seven of these invited papers have made it into this Special Issue.

Abstract On June 26–28, 2020, the International Conference on Software and Systems Processes (ICSSP 2020) and the International Conference on Global Software Engineering (ICGSE 2020) were held in virtual settings during the first year of the COVID pandemic. Several submissions to the joint event have been selected for inclusion in this special issue, focusing on impactful and timely contributions to machine learning (ML). At present, many in our field are enthusiastic about the potential of ML, yet some risks should not be casually overlooked or summarily dismissed. Each ML implementation is subtly different from any other implementation, and the risk profile varies greatly based on the approach adopted and the implementation context. The ICSSP/ICGSE 2020 Program Committees have encouraged submissions that explore the risks and benefits associated with ML so that the important discussion regarding ML efficacy and advocacy can be further elaborated. Four contributions have been included in this special issue.

Abstract Context Tangled commits are changes to software that address multiple concerns at once. For researchers interested in bugs, tangled commits mean that they actually study not only bugs, but also other concerns irrelevant for the study of bugs. Objective We want to improve our understanding of the prevalence of tangling and the types of changes that are tangled within bug fixing commits. Methods We use a crowd sourcing approach for manual labeling to validate which changes contribute to bug fixes for each line in bug fixing commits. Each line is labeled by four participants. If at least three participants agree on the same label, we have consensus. Results We estimate that between 17% and 32% of all changes in bug fixing commits modify the source code to fix the underlying problem. However, when we only consider changes to the production code files this ratio increases to 66% to 87%. We find that about 11% of lines are hard to label leading to active disagreements between participants. Due to confirmed tangling and the uncertainty in our data, we estimate that 3% to 47% of data is noisy without manual untangling, depending on the use case. Conclusion Tangled commits have a high prevalence in bug fixes and can lead to a large amount of noise in the data. Prior research indicates that this noise may alter results. As researchers, we should be skeptics and assume that unvalidated data is likely very noisy, until proven otherwise.

Together with many success stories, promises such as the increase in production speed and the improvement in stakeholders’ collaboration have contributed to making agile a transformation in the software industry in which many companies want to take part. However, driven either by a natural and expected evolution or by contextual factors that challenge the adoption of agile methods as prescribed by their creator(s), software processes in practice mutate into hybrids over time. Are these still agile? In this article, we investigate the question: what makes a software development method agile? We present an empirical study grounded in a large-scale international survey that aims to identify software development methods and practices that improve or tame agility. Based on 556 data points, we analyze the perceived degree of agility in the implementation of standard project disciplines and its relation to used development methods and practices. Our findings suggest that only a small number of participants operate their projects in a purely traditional or agile manner (under 15 percent). That said, most project disciplines and most practices show a clear trend towards increasing degrees of agility. Compared to the methods used to develop software, the selection of practices has a stronger effect on the degree of agility of a given discipline. Finally, there are no methods or practices that explicitly guarantee or prevent agility. We conclude that agility cannot be defined solely at the process level. Additional factors need to be taken into account when trying to implement or improve agility in a software company. Finally, we discuss the field of software process-related research in the light of our findings and present a roadmap for future research.

Reviewer selection in modern code review is crucial for effective code reviews. Several techniques exist for recommending reviewers appropriate for a given pull request (PR). Most code reviewer recommendation techniques in the literature build and evaluate their models based on datasets collected from real projects using open-source or industrial practices. The techniques invariably presume that these datasets reliably represent the “ground truth.”

Bug tracking is the process of monitoring and reporting malfunctions or issues found in software. While there is no consensus on a formally specified bug tracking process, some certain rules and best practices for an optimal bug tracking process are accepted by many companies and open-source software (OSS) projects. Despite slight variations between different platforms, the primary aim of all these rules and practices is to perform a more efficient bug tracking process. Practitioners’ noncompliance with the best practices not only impedes the benefits of the bug tracking process but also negatively affects the other phases of the life cycle of software development.In this study, based on the results of a multivocal literature review, we analyzed 60 sources in academic and gray literature and propose a taxonomy of 12 bad practices in the bug tracking process, that is bug tracking process smells. To quantitatively analyze these process smells, we inspect bug reports collected from six projects. Among these projects, four of them are Jira-based (MongoDB Core Server, Evergreen, Confluence Server & Data Center, Jira Server & Data Center) and the other two are Bugzilla-based (GCC and Wireshark). We observed that a considerable amount of bug tracking process smells exist in all projects with varying ratios.

Code comments play a vital role in source code comprehension and software maintainability. It is common for developers to write comments for explaining a code snippet. However, low-quality comments can have a detrimental effect on software quality or be ineffective for code understanding. This study aims to create a taxonomy of inline code comment smells and determine how commonly each smell type occurs in software projects. We conducted a multivocal literature review for defining the initial taxonomy of inline comment smells. Afterward, we manually labeled 899 inline comments from three open-source Java projects. We created a taxonomy of 11 inline code comment smell types and found out that the smells exist in practice with varying degrees.

Code review is a systematic inspection of the code-base. It ensures that the software satisfies the required functionalities and standards; thus, it is an essential stage in the modern development process and is used frequently in the industry. For a successful review, the reviewer should be able to identify defects; therefore, the process is highly dependent on the awareness of the reviewer. Currently, code review is done by comparing the line-by-line differences on the codebase. However, this does not give structural information, such as design pattern changes or dependency changes between services.To address this problem, we propose an augmented next-generation code review experience using visual and holistic approaches to streamline the reviewing process. The proposed code review experience shows the structural representation of the change by visualizing it into a UML-like relationship diagram. Along with this diagram, the risk percentage for each commit is highlighted to emphasize the differences that possibly can be most affected by the change. An artifact map is also produced that includes issue-commit relation for bug tracking, to make reviewers aware of the frequent issues that are brought up. This map also highlights the change frequency of the files, for focusing the reviewers on the possibly vulnerable parts of the software for better quality reviews. The proposed visualizations have been implemented in a plugin-based proof-of-concept tool integrated within GitHub, supporting the visual reviewing of changes in codebases developed with Java language.

Bug handling is an essential part of the software development process. Ideally, in a bug tracking system, bugs are reported, fixed, verified, and closed. In some cases, bugs have to be reopened mostly due to an incorrect fix. However, instead of reopening the existing bug report, users may submit a new report on a previously reported bug, which causes duplicate bug reports. Additionally, users might report duplicate bugs if they are unable to reopen the previously reported bugs due to the bug being unresolved (i.e., in progress) and when they miss previously reported bug reports. These duplicate bug reports may cost extra maintenance efforts in triaging and fixing bugs.There have been several studies on characterizing reopened bugs and duplicate bug reports, however, to the best of our knowledge, there has been no prior work on understanding the dynamics of their intersection, which is missed reopen bugs. Our study is based on analyzing the differences between duplicate and non-duplicate bugs, and further categorizing the duplicated bugs. In this regard, we categorize duplicate bugs according to their creation time with respect to their master's resolution status as Master-Unresolved bugs and Master-Resolved (Missed Reopen bugs) to distinguish their properties. We compare these two different types of bugs in terms of various aspects such as their relationships to their master bugs, bug surface time, bug fix time, bug's severity, and the number of users involved. We perform case studies using the Eclipse and Mozilla projects' bug repositories that include more than 165,500 and 394,000 bug reports respectively.

We propose a complementary board game that aims to familiarize students with usage of tools in software engineering, their functionality and how they fit together in order to form a tool-stack for modern development paradigms like Agile and DevOps for an Application Lifecycle Management course.

Code review, a manual inspection of source code by developers other than the author, is a frequently used practice for improving code quality in the software development life-cycle. Employing a tool-based review of code changes has become the norm for a wide variety of open source and industrial systems. Despite its widespread usage and importance, software development practices such as code review are often not addressed in typical Software Engineering education. To address this knowledge gap, we propose to use a serious game approach for teaching code review practices. In this study, we define our learning objectives and design a code review serious game along with its companion quizzes. Then we conduct a small preliminary experiment in order to procure feedback. Using the results of the experiment and participant interviews, we improve our game prototype for integration into a software engineering course while optimizing the initial experiment for student's benefit. We document the process, lessons learned and the future directions of the game. The results we gather indicate that the game is ready to be used in a software engineering course setting.

The application of code review in a development environment is essential, but this skill is not taught very often in an educational context despite its wide usage. To streamline the teaching process of code review, we propose a browser based "Code Review Serious Game" (CRSG) with high accessibility, progressive level difficulty and an evolvable foundation for prospective improvements or changes. The application is built as a serious game to reinforce the learning experience of its users by immersing them in its story and theme, helping them learn while having fun. The effectiveness of the game components are measured with a case study of 132 students of 2 software engineering courses. The promising result of this case study suggests CRSG can indeed be used effectively to teach code review. The demo video for the game can be accessed at https://youtu.be/FLnr3p4bhOg, and CRSG itself at: https://github.com/barisardic/crsg.

Context:Knowledge level and productivity of the software engineering (SE) workforce are the subject of regular discussions among practitioners, educators, and researchers. There have been many efforts to measure and improve the knowledge gap between SE education and industrial needs. Objective:Although the existing efforts for aligning SE education and industrial needs have provided valuable insights, there is a need for analyzing the SE topics in a more “fine-grained” manner; i.e., knowing that SE university graduates should know more about requirements engineering is important, but it is more valuable to know the exact topics of requirements engineering that are most important in the industry. Method:We achieve the above objective by assessing the knowledge gaps of software engineers by designing and executing an opinion survey on levels of knowledge learned in universities versus skills needed in industry. We designed the survey by using the SE knowledge areas (KAs) from the latest version of the Software Engineering Body of Knowledge (SWEBOK v3), which classifies the SE knowledge into 12 KAs, which are themselves broken down into 67 subareas (sub-KAs) in total. Our analysis is based on (opinion) data gathered from 129 practitioners, who are mostly based in Turkey. Results:Based on our findings, we recommend that educators should include more materials on software maintenance, software configuration management, and testing in their SE curriculum. Based on the literature as well as the current trends in industry, we provide actionable suggestions to improve SE curriculum to decrease the knowledge gap.

Background: Selecting the ideal code reviewer in modern code review is a crucial first step to perform effective code reviews. There are several algorithms proposed in the literature for recommending the ideal code reviewer for a given pull request. The success of these code reviewer recommendation algorithms is measured by comparing the recommended reviewers with the ground truth that is the assigned reviewers selected in real life. However, in practice, the assigned reviewer may not be the ideal reviewer for a given pull request.Aims: In this study, we investigate the validity of ground truth data in code reviewer recommendation studies.Method: By conducting an informal literature review, we compared the reviewer selection heuristics in real life and the algorithms used in recommendation models. We further support our claims by using empirical data from code reviewer recommendation studies.Results: By literature review, and accompanying empirical data, we show that ground truth data used in code reviewer recommendation studies is potentially problematic. This reduces the validity of the code reviewer datasets and the reviewer recommendation studies. Conclusion: We demonstrated the cases where the ground truth in code reviewer recommendation studies are invalid and discussed the potential solutions to address this issue.

Large-scale software development organizations generally carry out multiple software development projects simultaneously. Teams use various software development processes and tools to implement these projects. In this context, the main challenges of the practitioners are (1) keeping track of the status of a single project where hybrid set of tools exist for different software life cycle activities (2) effectively monitoring a consolidated status of multiple projects that use hybrid processes and tools. To address these challenges, it is vital to have a unified view of these projects independent from these hybrid processes and tools. To this end, we report on our preliminary experiences on the development of a unified project monitoring solution and a corresponding tool support based on the Essence framework's language and kernel. Our solution provides an up-to-date and unified view of projects by collecting data from various tools automatically as well as allowing manual data entry.

Today, one of the challenges in software engineering is utilising application lifecycle management (ALM) tools effectively in software development. In particular, it is hard for software developers to engage with the work items that are appointed to themselves in these ALM tools. In this study, the authors have focused on bug tracking in ALM where one of the most important metrics is mean time to resolution that is the average time to fix a reported bug. To improve this metric, they developed a serious game application based on an auction‐based reward mechanism. The ultimate aim of this approach is to create an incentive structure for software practitioners to find and resolved bugs that are auctioned where participants are encouraged to solve and test more bugs in less time and improve quality of software development in a competitive environment. They conduct hypothesis tests by performing a Monte Carlo simulation. The preliminary results of this research support the idea that using a gamification approach for an issue tracking system enhances the productivity and decreases mean time to resolution.