Abstract Social coding platforms like GitHub facilitate collaborative software development through pull requests (PRs), which generate discussions that significantly impact code quality, requirements, and design. Such conversations become a rich source of insights for improving development practices and predicting project outcomes and are subject to several human aspects that have been linked to code quality and PR acceptance. Sentiment analysis is one of the many ways to try to understand these human aspects. However, PR discussions are multifaceted, often involving technical jargon and aspects which limits the utility of general-purpose sentiment analysis tools. This has led to the creation of SE-specific tools, but recent studies have also observed that they demonstrate limited effectiveness. Thus, this study explores the potential of using large language models (LLMs) for this purpose, given their enhanced contextual understanding and ability to process technical language. We evaluated ten LLMs across proprietary and open-source categories, using two complementary datasets: a curated Gold dataset and the PRemo dataset, which captures real-world PR discussions. The models were assessed under zero-shot, few-shot and chain-of-thought prompting techniques on 8,913 messages. In addition, we establish baselines by evaluating fine-tuned transformer-based models. Results show that GPT-4o achieved the highest overall performance across the LLMs, though smaller models, such as Mistral Small and Deepseek-R1 32B delivered competitive results. Transformer-based models achieved excellent performance on the Gold dataset but exhibited degradation on the PRemo dataset. Finally, we conducted a qualitative analysis of misclassified instances, revealing recurring challenges related to technical terminology, sentiment-charged keywords, message length, and contextual ambiguity. These findings suggest that model selection should balance performance requirements against practical constraints, rather than defaulting to the largest available models.

Open software development platforms, such as GitHub, foster developers’ collaboration in coding tasks through pull requests (PR). PRs serve as a mechanism for code contributions and structured discussions. These discussions often involve developers exchanging messages that contain questions. Despite the importance of questions, existing studies have not gathered insights on what are recurring types of questions in GitHub PR discussions. In this paper, we address this gap by leveraging the well-established 5W2H framework, which consists of seven questions classes. Our goal is to analyze the distribution of these question types within GitHub PRs and assess whether language models, specifically GPT-3.5 and GPT-4o, can accurately classify them. We conducted our study using a dataset derived from nine GitHub projects from the Netflix and Google ecosystems. Our findings reveal that both GPT-3.5 and GPT-4o perform well in identifying What and Who classes, which were the most frequently occurring question types in developers’ discussions. However, both models struggled with less common categories, failing to detect How Much, and showing inconsistencies in classifying Where, When, Why, and How. These findings suggest that improving classification accuracy may require incorporating domain-specific context and refining prompt engineering techniques.

Modern software development relies on cloud-based collaborative platforms (e.g., GitHub and GitLab). In these platforms, developers often employ a pull-based development approach, proposing changes via pull requests and engaging in communication via asynchronous message exchanges. Since communication is key for software development, studies have linked different types of sentiments embedded in the communication to their effects on software projects, such as bug-inducing commits or the non-acceptance of pull requests. In this context, sentiment analysis tools are paramount to detect the sentiment of developers’ messages and prevent potentially harmful impact. Unfortunately, existing state-of-the-art tools vary in terms of the nature of their data collection and labeling processes. Yet, there is no comprehensive study comparing the performance and generalizability of existing tools utilizing a dataset that was designed and systematically curated to this end, and in this specific context. Therefore, in this study, we design a methodology to assess the effectiveness of existing sentiment analysis tools in the context of pull request discussions. For that, we created a dataset that contains ≈ 1.8K manually labeled messages from 36 software projects. The messages were labeled by 19 experts (neuroscientists and software engineers), using a novel and systematic manual classification process designed to reduce subjectivity. By applying these existing tools to the dataset, we observed that while some tools ]perform acceptably, their performance is far from ideal, especially when classifying negative messages. This is interesting since negative sentiment is often related to a critical or unfavorable opinion. We also observed that some messages have characteristics that can make them harder to classify, causing disagreements between the experts and possible misclassifications by the tools, requiring more attention from researchers. Our contributions include valuable resources to pave the way to develop robust and mature sentiment analysis tools that capture/anticipate potential problems during software development.

Refactoring is a non-trivial maintenance activity. Developers spend time and effort refactoring code to remove structural problems, i.e., code smells. Recent studies indicated that developers often apply composite refactoring (composite, for short), i.e., two or more interrelated refactorings. However, prior studies revealed that only 10% of composite refactorings are considered complete, i.e., those fully removing code smells. Many incomplete refactorings can even replace or introduce smells, requiring additional effort for their removal later in the project. Moreover, existing refactoring recommendations are not well-detailed and do not alert developers about these possible side effects. To address these gaps, we conducted a large-scale study involving more than 250k refactorings from 42 software projects, including both open-source and closed-source projects. Our goal is to investigate how the most common complete composites are combined and their side effects in the practice. Our results reveal that the current recommendation to apply Extract Method(s) with fine-grained refactoring types needs refinements. We found that certain fine-grained refactorings like Change Variable Types and Change Return Types can introduce up to 45% of Brain Methods when combined with Extract Method(s). Moreover, Ex-tract Method(s) and Move Method(s), a common recommendation to remove Feature Envy, may inadvertently introduce about 30% of Lazy Classes and approximately 70% of Data Classes. Despite these potential side effects, existing refactoring catalogs and tools' recommenders do not alert developers about these side effects. Finally, we consolidate our findings into a catalog to provide clear guidance for developers and researchers on effectively applying composite refactorings to fully remove code smells.

Robustness is critical for ensuring that software functions correctly under adverse conditions. Exception-handling mechanisms in programming languages enable developers to deal with these adverse conditions. However, implementing exception-related code can present significant challenges to developers. We investigated exception-related code contributions across Java projects in the Apache ecosystem. We analyzed exception-related pull requests (exception-PRs), which were detected using a validated heuristic. We produced a comprehensive dataset of 988 exception-PRs. We observed no statistically significant differences in complexity metrics between exception-PRs and non-exception-PRs. We also found no significant differences in developers' behavior metrics, indicating consistent engagement regardless of whether the pull request addressed exception-related code. A manual analysis revealed that most exception-PRs focused on system improvements rather than bug fixes, suggesting proactive efforts to enhance software robustness. Moreover, the most frequently addressed aspects of exceptional code in these exception-PRs were: (i) the external representation of adverse situations to end-users (more than 40% of the PRs) and (ii) the implementation of effective error-handling actions (nearly 35% of the PRs) to promote program recoverability. Interestingly, a significant proportion of exception-PRs simultaneously addressed multiple aspects. By understanding the nature and characteristics of exception-PRs, we expect to better support developers in managing erroneous conditions and improving software robustness.

Background: Code development is done collaboratively in platforms such as GitHub and GitLab, following a pull-based development model. In this model, developers actively communicate and share their knowledge through conversations. Pull request conversations are affected by social aspects such as communication dynamics among developers, discussion content, and organizational dynamics. Despite prior studies indicating that social aspects indeed impact software quality, it is still unknown to what extent social aspects influence design decay during software development. Thus, since social aspects are intertwined with design and implementation decisions, there is a need for investigating how social aspects contribute to avoiding, reducing, or accelerating design decay. Aims: To fill this gap, we performed a study aimed at investigating the effects of pull request conversation on design decay. Method: We investigated 10,746 pull request conversations from 11 open-source systems, characterizing in terms of three different social aspects: discussion content, organizational and communication dynamics. We considered 18 social metrics to these three social aspects, and analyzed how they associate with design decay. We used a statistical approach to assess which social metrics are able to discriminate between impactful and unimpactful pull requests. Then, we employed a multiple logistic regression model to evaluate the influence of each social metric per social aspect in the presence of each other on design decay. Finally, we also observed how the combination of all social metrics influences the design decay. Results: Our findings reveal that social metrics related to the size and duration of a discussion, the presence of design-related keywords, the team size, and gender diversity can be used to discriminate between design impactful and unimpactful pull requests. Organizational growth and gender diversity prevent decay. Each software community has its unique aspects that can be used to detect and prevent design decay. Also, design improvements can be accomplished by timely feedback, engaged communication, and design-oriented discussions with the contribution of multiple participants who provide significant comments. Conclusion: The social aspects related to pull request conversations are useful indicators of design decay.

Modern programming languages, such as Java, use exception-handling mechanisms to guarantee the robustness of software systems. Although important, the quality of exception code is usually poor and neglected by developers. Indiscriminate robustness changes (e.g., the addition of empty catch blocks) can indicate design decisions that negatively impact the internal quality of software systems. As it is known in the literature, multiple occurrences of poor code structures, namely code smells, are strong indicators of design problems. Still, existing studies focus mainly on the correlation of maintainability smells with design problems. However, using only these smells may not be enough since developers need more context (e.g., system domain) to identify the problems in certain scenarios. Moreover, these studies do not explore how changes in the exceptional code of the methods combined with maintainability smells can give complementary evidence of design problems. By covering both regular and exception codes, the developer can have more context about the system and find complementary code smells that reinforce the presence of design problems. This work aims to leverage the identification of design problems by tracking poor robustness changes combined with maintainability smells. We investigated the correlation between robustness changes and maintainability smells on the commit history of more than 160k methods from different releases of 10 open-source software systems. We observed that maintainability smells can be worsened or even introduced when robustness changes are performed. This scenario mainly happened for the smells Feature Envy, Long Method, and Dispersed Coupling. We also analyzed the co-occurrence between robustness and maintainability smells. We identified that the empty catch block and catch throwable robustness smells were the ones that co-occurred the most with maintainability smells related to the Concern Overload and Misplaced Concern design problems. The contribution of our work is to reveal that poor exception code, usually neglected by developers, negatively impacts the quality of methods and classes, signaled by the maintainability smells. Therefore, existing code smell detecting tools can be enhanced to leverage robustness changes to identify design problems.

Developers constantly perform code changes throughout the lifetime of a project. These changes may induce the introduction of design problems (design decay) over time, which may be reduced or accelerated by interacting with different factors (e.g., refactorings) that underlie each change. However, existing studies lack evidence about how these factors interact and influence design decay. Thus, this paper reports a study aimed at investigating whether and how (associations of) process and developer factors influence design decay. We studied seven software systems, containing an average of 45K commits in more than six years of project history. Design decay was characterized in terms of five internal quality attributes: cohesion, coupling, complexity, inheritance, and size. We observed and characterized 12 (sub-)factors and how they associate with design decay. To this end, we employed association rule mining. Moreover, we also differentiate between the associations found on modules with varying levels of decay. Process- and developer-related factors played a key role in discriminating these different levels of design decay. Then, we focused on analyzing the effects of potentially interacting factors regarding slightly- and largely-decayed modules. Finally, we observed diverging decay patterns in these modules. For example, individually, the developer-related sub-factor that represented first-time contributors, as well as the process-related one that represented the size of a change did not have negative effects on the changed classes. However, when analyzing specific factor interactions, we saw that changes in which both of these factors interacted tended to have a negative effect on the code, leading to decay.

Recent studies have revealed that developers often apply composite refactorings (or, simply, composites). A composite consists of two or more interrelated refactorings applied together. Previous studies investigated the effect of composites on code smells. A composite is considered “complete” whenever it completely removes one target code smell. They proposed descriptions of complete composites with recommendations to remove certain code smell types, such as Long Methods and Feature Envies. These studies also present different recommendations to remove the same code smell type. However, these studies: (i) are limited to composites only consisting of a small subset of Fowler's refactoring types, (ii) do not detail the scenarios in which each recommendation can be applied to remove the code smell, and (iii) fail in reporting possible side effects of the described composites, such as adversely introducing certain smell types. This paper aims to cover these limitations by performing a systematic analysis of 618 complete composites on removing four common smell types identified in 20 software projects. Our results indicated that: (i) 64% complete composites consisted of refactoring types not covered by existing descriptions of complete composites, and (ii) 36% complete composites formed by Extract Methods can introduce Feature Envies and Intensive Couplings. This information is not documented by existing descriptions, and it can alert developers about alternatives to remove Feature Envy, mainly in methods that are fully envious. These results suggest existing descriptions of complete composites should be either revisited or enhanced to explicitly highlight known side effects. We present a catalog of composites with details about side effects, recommendations to remove or minimize them, and some scenarios in which each recommendation can be applied to remove the code smell. Our catalog can be useful to improve existing tooling support for refactorings, such as IDEs, informing about possible side effects when refactorings are composed.

Companies have adopted modern code review as a key technique for continuously monitoring and improving the quality of software changes. One of the main motivations for this is the early detection of design impactful changes, to prevent that design-degrading ones prevail after each code review. Even though design degradation symptoms often lead to changes' rejections, practices of modern code review alone are actually not sufficient to avoid or mitigate design decay. Software design degrades whenever one or more symptoms of poor structural decisions, usually represented by smells, end up being introduced by a change. Design degradation may be related to both technical and social aspects in collaborative code reviews. Unfortunately, there is no study that investigates if code review stakeholders, e.g, reviewers, could benefit from approaches to distinguish and predict design impactful changes with technical and/or social aspects. By analyzing 57,498 reviewed code changes from seven open-source systems, we report an investigation on prediction of design impactful changes in modern code review. We evaluated the use of six ML algorithms to predict design impactful changes. We also extracted and assessed 41 different features based on both social and technical aspects. Our results show that Random Forest and Gradient Boosting are the best algorithms. We also observed that the use of technical features results in more precise predictions. However, the use of social features alone, which are available even before the code review starts (e.g., for team managers or change assigners), also leads to highly-accurate prediction. Therefore social and/or technical prediction models can be used to support further design inspection of suspicious changes early in a code review process. Finally, we provide an enriched dataset that allows researchers to investigate the context behind design impactful changes during the code review process.

Program refactoring consists of code changes applied to improve the internal structure of a program and, as a consequence, its comprehensibility. Recent studies indicate that developers often perform composite refactorings, i.e., a set of two or more interrelated single refactorings. Recent studies also recommend certain patterns of composite refactorings to fully remove poor code structures, i.e, code smells, thus further improving the program comprehension. However, other recent studies report that composite refactorings often fail to fully remove code smells. Given their failure to achieve this purpose, these composite refactorings are considered incomplete, i.e, they are not able to entirely remove a smelly structure. Unfortunately, there is no study providing an in-depth analysis of the incompleteness nature of many composites and their possibly partial impact on improving, maybe decreasing, internal quality attributes. This paper identifies the most common forms of incomplete composites, and their effect on quality attributes, such as coupling and cohesion, which are known to have an impact on program comprehension. We analyzed 353 incomplete composite refactorings in 5 software projects, two common code smells (Feature Envy and God Class), and four internal quality attributes. Our results reveal that incomplete composite refactorings with at least one Extract Method are often (71%) applied without Move Methods on smelly classes. We have also found that most incomplete composite refactorings (58%) tended to at least maintain the internal structural quality of smelly classes, thereby not causing more harm to program comprehension. We also discuss the implications of our findings to the research and practice of composite refactoring.

When creating their programs, developers usually have a preferred or standardized style of their own to write code, known as coding style. Such code is usually stored in a version control repository, through which collaborative work usually takes place. However, in such a setting, isolated attempts of standardization can lead to several coding styles coexisting in the same project, causing the opposite effect to that intended. Besides increasing the effort required to understand code, coding style conflicts may also clutter repository history as developers change existing styles to their usual preferences. To overcome this problem, we propose an approach to support the definition of a repository coding style while allowing developers to use their preferred coding style. To illustrate our approach, we built the RECoSt tool and applied it using real excerpts of a popular open source project. Our proposed approach intends to help developers keep their projects' coding style standardized without having to abandon the style they are familiar with.