Service-based systems (SBS) depend on loosely coupled services that interact via standardized protocols, making testing challenging due to external dependencies. Mocking and service virtualization techniques have been proposed to simulate these dependencies and support automated testing, yet existing research lacks a unified synthesis of their approaches, effectiveness, and limitations.

AI coding agents have been increasingly integrated into software development workflows to help developers with various coding tasks. However, their effectiveness in real bug-fixing tasks remains uncertain. In this paper, we report on an analysis of 9,052 pull requests (PRs) from five agents—Codex, Copilot, Cursor, Claude, and Devin—alongside human-authored PRs from the same projects. We found that agentic PRs have lower acceptance rates than human-authored PRs (71.37% vs. 87.62%), although substantial variation exists between agents. To understand the reasons behind the rejection of agentic PRs, we manually examined 201 rejected agentic PRs and derived five categories of rejection factors. The most frequent issues are related to code quality, solution suitability, and failure to address reviewer feedback, often leading maintainers to abandon the review process. In many cases, agentic PRs are also superseded by other PRs. These findings shed light on current limitations of AI agents and highlight key challenges in developing more reliable and context-aware automated contributors.

Context: Diversity can impact team communication, productivity, cohesiveness, and creativity. Analyzing the existing knowledge about diversity in open source software (OSS) projects can provide directions for future research and raise awareness about barriers and biases against underrepresented groups in OSS. Objective: This study aims to analyze the knowledge about minority groups in OSS projects. We investigated which groups were studied in the OSS literature, the study methods used, their implications, and their recommendations to promote the inclusion of minority groups in OSS projects. Method: To achieve this goal, we performed a systematic literature review study that analyzed 42papers that directly study underrepresented groups in OSS projects. Results: Most papers focus on gender (62.3%), while others like age or ethnicity are rarely studied. The neurodiversity dimension, have not been studied in the context of OSS. Our results also reveal that diversity in OSS projects faces several barriers but brings significant benefits, such as promoting safe and welcoming environments. Conclusion: Most analyzed papers adopt a myopic perspective that sees gender as strictly binary. Dimensions of diversity that affect how individuals interact and function in an OSS project, such as age, tenure, and ethnicity, have received very little attention.

The use of generative AI-based coding assistants like ChatGPT and Github Copilot is a reality in contemporary software development. Many of these tools are provided as remote APIs. Using third-party APIs raises data privacy and security concerns for client companies, which motivates the use of locallydeployed language models. In this study, we explore the tradeoff between model accuracy and energy consumption, aiming to provide valuable insights to help developers make informed decisions when selecting a language model. We investigate the performance of 18 families of LLMs in typical software development tasks on two real-world infrastructures, a commodity GPU and a powerful AI-specific GPU. Given that deploying LLMs locally requires powerful infrastructure which might not be affordable for everyone, we consider both full-precision and quantized models. Our findings reveal that employing a big LLM with a higher energy budget does not always translate to significantly improved accuracy. Additionally, quantized versions of large models generally offer better efficiency and accuracy compared to full-precision versions of medium-sized ones. Apart from that, not a single model is suitable for all types of software development tasks.

Context:Code understandability plays a crucial role in software development, as developers spend between 58% and 70% of their time reading source code. Improving code understandability can lead to enhanced productivity and save maintenance costs.Problem:Experimental studies aim to establish what makes code more or less understandable in a controlled setting, but ignore that what makes code easier to understand in the real world also depends on extraneous elements such as developers’ background and project culture and guidelines. Not accounting for the influence of these factors may lead to results that are sound but have little external validity.Goal:We aim to investigate how developers improve code understandability during software development through code review comments. Our assumption is that code reviewers are specialists in code quality within a project.Method and Results:We manually analyzed 2,401 code review comments from Java open-source projects on GitHub and found that over 42% of all comments focus on improving code understandability, demonstrating the significance of this quality attribute in code reviews. We further explored a subset of 385 comments related to code understandability and identified eight categories of code understandability concerns, such as incomplete or inadequate code documentation, bad identifier, and unnecessary code. Among the suggestions to improve code understandability, 83.9% were accepted and integrated into the codebase. Among these, only two (less than 1%) ended up being reverted later. We also identified types of patches that improve code understandability, ranging from simple changes (e.g., removing unused code) to more context-dependent improvements (e.g., replacing method calling chains by existing API). Finally, we investigated the potential coverage of four well-known linters to flag the identified code understandability issues. These linters cover less than 30% of these issues, although some of them could be easily added as new rules.Implications:Our findings motivate and provide practical insight for the construction of tools to make code more understandable, e.g., understandability improvements are rarely reverted and thus can be used as reliable training data for specialized ML-based tools. This is also supported by our dataset, which can be used to train such models. Finally, our findings can also serve as a basis to develop evidence-based code style guides.

Deep Learning (DL) frameworks such as PyTorch and TensorFlow include runtime infrastructures responsible for executing trained models on target hardware, managing memory, data transfers, and multi-accelerator execution, if applicable. Additionally, it is a common practice to deploy pre-trained models on environments distinct from their native development settings. This led to the introduction of interchange formats such as ONNX, which includes its runtime infrastructure, and ONNX Runtime, which work as standard formats that can be used across diverse DL frameworks and languages. Even though these runtime infrastructures have a great impact on inference performance, no previous paper has investigated their energy efficiency. In this study, we monitor the energy consumption and inference time in the runtime infrastructures of three well-known DL frameworks as well as ONNX, using three various DL models. To have nuance in our investigation, we also examine the impact of using different execution providers. We find out that the performance and energy efficiency of DL are difficult to predict. One framework, MXNet, outperforms both PyTorch and TensorFlow for the computer vision models using batch size 1, due to efficient GPU usage and thus low CPU usage. However, batch size 64 makes PyTorch and MXNet practically indistinguishable, while TensorFlow is outperformed consistently. For BERT, PyTorch exhibits the best performance. Converting the models to ONNX yields significant performance improvements in the majority of cases. Finally, in our preliminary investigation of execution providers, we observe that TensorRT always outperforms CUDA.

Existing studies on the use of Large Language Models (LLMs) in software development leverage methodologies that limit their scalability and require intensive manual data collection and analysis, for example, due to the use of video data or think-aloud protocols. We propose the use of a specialized tool capable of automatically collecting fine-grained, relevant data during experiments and case studies. It enables researchers to understand for example how often participants accept or reject suggestions made by LLMs and what kinds of prompts are more likely to trigger accepted suggestions, even in studies targeting a large number of participants. We implement this idea as a Visual Studio Code plugin named AthenaLLM 1. It mimics the functionalities of GitHub Copilot and offers seamless integration with OpenAI API models like GPT-4 and GPT-3.5, and compatibility with other models providing an OpenAI-compatible API, e.g., Vicuna [6]. It automatically collects data at a fine level of granularity and covers both the interactions of developers with their IDE, e.g., all changes made in the code, and the products of such interactions, e.g., the generated code, when accepted. Thus, the proposed approach also reduces bias that the experimental process itself may introduce, e.g., due to the need for participants to verbalize their thoughts. In this paper we discuss how AthenaLLM could enable researchers to go both broader (in terms of number of participants) and deeper (in terms of the kinds of research questions that can be tackled).

Mobile app development frameworks lower the effort to write and deploy apps across different execution platforms. At the same time, their use may limit native optimizations and impose overhead, increasing resource usage. In this paper, we analyze the resource usage of Android benchmarks and apps based on three mobile app development frameworks, Flutter, React Native, and Ionic, comparing them to functionally equivalent, native variants written in Java. These frameworks, besides being in widespread use, represent three different approaches for developing multiplatform apps: Flutter supports the deployment of apps that are compiled and run fully natively, React Native runs interpreted JavaScript code combined with native views for different platforms, and Ionic is based on web apps, which means that it does not depend on platform-specific details. We measure the energy consumption, execution time, and memory usage of ten optimized, CPU-intensive benchmarks, to gauge overhead in a controlled manner, and two applications, to measure their impact when running commonly mobile app functionalities. Our results show that cross-platform and hybrid frameworks can be competitive in CPU-intensive applications. In five of the ten benchmarks, at least one framework-based version exhibits lower energy consumption and execution time than its native counterpart, up to a reduction of 81% in energy and 83% in execution time. Furthermore, in three other benchmarks, framework-based and native versions achieved similar results. Overall, Flutter, usually imposes the least overhead in execution time and energy, while React Native imposes the highest in all the benchmarks. However, in an app that continuously animates multiple images on the screen, without interaction, the React Native version uses the least CPU and energy, up to a reduction of 96% in energy compared to the second-best framework-based version. These findings highlight the importance of analyzing expected application behavior before committing to a specific framework.

Software programs can be written in different but functionally equivalent ways. Even though previous research has compared specific formatting elements to find out which alternatives affect code legibility, seeing the bigger picture of what makes code more or less legible is challenging. We aim to find which formatting elements have been investigated in empirical studies and which alternatives were found to be more legible for human subjects. We conducted a systematic literature review and identified 15 papers containing human-centric studies that directly compared alternative formatting elements. We analyzed and organized these formatting elements using a card-sorting method. We identified 13 formatting elements (e.g., indentation) and 33 levels of formatting elements (e.g., two-space indentation), which are about formatting styles, spacing, block delimiters, long or complex code lines, and word boundary styles. While some levels were found to be statistically better than other equivalent ones in terms of code legibility, e.g., appropriate use of indentation with blocks, others were not, e.g., formatting layout. For identifier style, we found divergent results, where one study found a significant difference in favor of camel case, while another study found a positive result in favor of snake case. The number of identified papers, some of which are outdated, and the many null and contradictory results emphasize the relative lack of work in this area and underline the importance of more research. There is much to be understood about how formatting elements influence code legibility before the creation of guidelines and automated aids to help developers make their code more legible.

The best practices in the industry for Software Architecture (SA) documentation are not always followed, despite it being known that SA documentation can positively influence different aspects of software development. Open-Source Software (OSS) projects often operate in a different manner compared to proprietary software projects. This study investigates contemporary SA documentation practices in OSS projects to gain insights into: (1) what architectural elements are described, (2) what the different description formats and types are, and (3) what the maturity of the architecture description is. The SA description documentation of six OSS projects is identified, classified, and evaluated: VLC, OpenEHR, openKM, GIMP, Audacity, and Home Assistant. The results show that natural language is widely used in describing the architecture, sometimes accompanied by diagrams of informal models. The majority of documentation was found on websites and wikis. The maturity was evaluated by applying the Architectural Capability Model (ArchCaMo). Out of the five maturity levels, most projects did not get past the first level. Only one project reached the second level, and one project showed potential for level three as it was the only project with explicitly documented SA design decisions.

Code review is a popular software engineering practice. Success of code reviews can be threatened by confusion experienced by code reviewers. For instance, on the one hand, research has studied the reasons for confusion in code reviews, and on the other hand, it also has analyzed source code patterns, so called "atoms of confusion", that have been shown to lead to misunderstanding in the lab setting. However, to the best of our knowledge, there is no research which tried to investigate the possible cause and effect relationship between atoms of confusion and confusion in code reviews. Another important aspect still not studied is how those atoms of confusion evolve across pull requests. In this emerging results paper, we report an exploratory case study to provide a deeper understanding of atoms of confusion, more specifically, whether atoms of confusion are related to confusion in code reviews and how they persist across pull requests. With the help of an existing tool for the detection of atoms of confusion, and a manual analysis of code reviews comments, we observed that statistical analysis did not show any relationship between atoms of confusion and presence of confusion comments in code reviews. Additionally, we found evidence that atoms of confusion are mostly not being removed in pull requests. Based on the results, we formulate hypotheses on atoms of confusion in the code review context, that should be confirmed or rejected by future studies.

Conventionally, callbacks and inversion of control have been the main tools to structure event-driven applications. Sadly, those patterns constitute a well-known source of design problems. The Reactive Programming (RP) paradigm has arisen as an approach to mitigate these problems. Yet, little evidence has been provided regarding the advantages of RP, and concerns have also arisen about the API usability of RP libraries given their disparate number of operators. In this work, we conduct a study on GitHub (GH) and Stack Overflow (SO) and explore three Reactive Extensions (Rx) libraries (RxJava, RxJS, and RxSwift) with the most GH projects to understand how much the vast Rx operators are being used. Also, we examine Rx SO posts to complement the results from the GH exploration by understanding the problems faced by RP developers and how they relate with the operators' frequencies found in open source projects. Results reveal that, in spite of its API size, the great majority of the Rx operators are actually being used (95.2%), with only a few, mostly related to RxJava, not being utilized. Also, we unveil 23 topics from SO with more posts concerning the Stream Abstraction (36.4%). Posts related to Dependency Management, Introductory Questions, and iOS Development figure as relevant topics to the community. The findings herein present can not only stimulate advancements in the field by understanding the usage of RP API and the main problems faced by developers, but also help newcomers in identifying the most important operators and the areas that are the most likely to be relevant for a RP application.

Although Open Source Software (OSS) maintainers devote a significant proportion of their work to coding tasks, great maintainers must excel in many other activities beyond coding. Maintainers should care about fostering a community, helping new members to find their place, while also saying "no" to patches that although are well-coded and well-tested, do not contribute to the goal of the project. To perform all these activities masterfully, maintainers should exercise attributes that software engineers (working on closed source projects) do not always need to master. This paper aims to uncover, relate, and prioritize the unique attributes that great OSS maintainers might have. To achieve this goal, we conducted 33 semi-structured interviews with well-experienced maintainers that are the gatekeepers of notable projects such as the Linux Kernel, the Debian operating system, and the GitLab coding platform. After we analyzed the interviews and curated a list of attributes, we created a conceptual framework to explain how these attributes are connected. We then conducted a rating survey with 90 OSS contributors. We noted that "technical excellence" and "communication" are the most recurring attributes. When grouped, these attributes fit into four broad categories: management, social, technical, and personality. While we noted that "sustain a long term vision of the project" and being "extremely careful" seem to form the basis of our framework, we noted through our survey that the communication attribute was perceived as the most essential one.

Abstract Context Code review is a widely used technique of systematic examination of code changes which aims at increasing software quality. Code reviews provide several benefits for the project, including finding bugs, knowledge transfer, and assurance of adherence to project guidelines and coding style. However, code reviews have a major cost: they can delay the merge of the code change, and thus, impact the overall development process. This cost can be even higher if developers do not understand something, i.e., when developers faceconfusionduring the code review. Objective This paper studies the phenomenon ofconfusionin code reviews. Understanding confusion is an important starting point to help reducing the cost of code reviews and enhance the effectiveness of this practice, and hence, improve the development process. Method We conducted two complementary studies. The first one aimed at identifying the reasons for confusion in code reviews, its impacts, and the coping strategies developers use to deal with it. Then, we surveyed developers to identify the most frequently experienced reasons for confusion, and conducted a systematic mapping study of solutions proposed for those reasons in the scientific literature. Results From the first study, we build a framework with 30 reasons for confusion, 14 impacts, and 13 coping strategies. The results of the systematic mapping study shows 38 articles addressing the most frequent reasons for confusion. From those articles, we found 13 different solutions for confusion proposed in the literature, and five impacts were established related to the most frequent reasons for confusion. Conclusions Based on the solutions identified in the mapping study, or the lack of them, we propose an actionable guideline for developers on how to cope with confusion during code reviews; we also make several suggestions how tool builders can support code reviews. Additionally, we propose a research agenda for researchers studying code reviews.

Reading code is an essential activity in software maintenance and evolution. Several studies with human subjects have investigated how different factors, such as the employed programming constructs and naming conventions, can impact code readability, i.e., what makes a program easier or harder to read and apprehend by developers, and code legibility, i.e., what influences the ease of identifying elements of a program. These studies evaluate readability and legibility by means of different comprehension tasks and response variables. In this paper, we examine these tasks and variables in studies that compare programming constructs, coding idioms, naming conventions, and formatting guidelines, e.g., recursive vs. iterative code. To that end, we have conducted a systematic literature review where we found 54 relevant papers. Most of these studies evaluate code readability and legibility by measuring the correctness of the subjects' results (83.3%) or simply asking their opinions (55.6%). Some studies (16.7%) rely exclusively on the latter variable. There are still few studies that monitor subjects' physical signs, such as brain activation regions (5%). Moreover, our study shows that some variables are multi-faceted. For instance, correctness can be measured as the ability to predict the output of a program, answer questions about its behavior, or recall parts of it. These results make it clear that different evaluation approaches require different competencies from subjects, e.g., tracing the program vs. summarizing its goal vs. memorizing its text. To assist researchers in the design of new studies and improve our comprehension of existing ones, we model program comprehension as a learning activity by adapting a preexisting learning taxonomy. This adaptation indicates that some competencies, e.g., tracing, are often exercised in these evaluations whereas others, e.g., relating similar code snippets, are rarely targeted.

Exception handling is a feature provided by most mainstream programming languages, and typically involves constructs to throw and handle error signals. On the one hand, early work has argued extensively about the benefits of exception handling, such as promoting modularity by defining how exception handlers can be implemented and maintained independently of the normal behavior of the system and easing but localization. On the other hand, some studies argue that exception handling can make the programming languages unnecessarily complex and promote the introduction of subtle bugs in programs. In 2015 we published a paper describing a study investigating the prevalence and nature of exception handling bugs in two large, widely adopted Java systems. This study also confronted its findings about real exception handling bugs with the perceptions of developers about those bugs, also accounting for bugs not related to exception handling. The goal of this reflection paper is to investigate the state of the art in exception handling research, with a particular emphasis on exception handling bugs, and how our paper has influenced other studies in the area. We found that our paper was cited by 33 articles, and all themes for future work we raised in our paper have been tackled by other studies in the short span of five years.

Background: APIs that implement I/O operations are the building blocks of many well-known, non-trivial software systems. These APIs are used for a great variety of programming tasks, from simple file management operations, to database communications and implementation of network protocols.Aims: Despite their ubiquity, there are few studies that focus on comprehending their energy behaviors in order to aid developers interested in building energy-conscious software systems. The goal of this work is two-fold. We first aim to characterize the landscape of the Java I/O programming APIs. After better comprehending their energy variations, our second goal is to refactor software systems that use energy inefficient I/O APIs to their efficient counterparts.Method: To achieve the first goal, we instrumented 22 Java micro-benchmarks that perform I/O operations. To achieve our second goal, we extensively experimented with three benchmarks already optimized for performance and five macro-benchmarks widely used in both software development practice and in software engineering optimization research.Results: Among the results, we found that the energy behavior of Java I/O APIs is diverse. In particular, popular I/O APIs are not always the most energy efficient ones. Moreover, we were able to create 22 refactored versions of the studied benchmarks, eight of which were more energy efficient than the original version. The (statistically significant) energy savings of the refactored versions varied from 0.87% up to 17.19%. More importantly, these energy savings stem from very simple refactorings (often touching less than five lines of code).Conclusions: Our work indicates that there is ample room for studies targeting energy optimization of Java I/O APIs.

As mobile devices are supporting more and more of our daily activities, it is vital to widen their battery up-time as much as possible. In fact, according to the Wall Street Journal, 9/10 users suffer from low battery anxiety. The goal of our work is to understand how Android usage, apps, operating systems, hardware and user habits influence battery lifespan. Our strategy is to collect anonymous raw data from devices all over the world, through a mobile app, build and analyze a large-scale dataset containing real-world, day-to-day data, representative of user practices. So far, the dataset we collected includes 12 million+ (anonymous) data samples, across 900+ device brands and 5.000+ models. And, it keeps growing. The data we collect, which is publicly available and by different channels, is sufficiently heterogeneous for supporting studies with a wide range of focuses and research goals, thus opening the opportunity to inform and reshape user habits, and even influence the development of both hardware and software for mobile devices.

Over the last years, increasing attention has been given to creating energy-efficient software systems. However, developers still lack the knowledge and the tools to support them in that task. In this work, we explore our vision that energy consumption non-specialists can build software that consumes less energy by alternating, at development time, between third-party, readily available, diversely-designed pieces of software, without increasing the development complexity. To support our vision, we propose an approach for energy-aware development that combines the construction of application-independent energy profiles of Java collections and static analysis to produce an estimate of in which ways and how intensively a system employs these collections. By combining these two pieces of information, it is possible to produce energy-saving recommendations for alternative collection implementations to be used in different parts of the system. We implement this approach in a tool named CT+ that works with both desktop and mobile Java systems, and is capable of analyzing 40 different collection implementations of lists, maps, and sets. We applied CT+ to twelve software systems: two mobile-based, seven desktop-based, and three that can run in both environments. Our evaluation infrastructure involved a high-end server, a notebook, and three mobile devices. When applying the (mostly trivial) recommendations, we achieved up to 17.34% reduction in energy consumption just by replacing collection implementations. Even for a real world, mature, highly-optimized system such as Xalan, CT+ could achieve a 5.81% reduction in energy consumption. Our results indicate that some widely used collections, e.g., ArrayList, HashMap, and HashTable, are not energy-efficient and sometimes should be avoided when energy consumption is a major concern.

Code review is a software quality assurance practice widely employed in both open source and commercial software projects to detect defects, transfer knowledge and encourage adherence to coding standards. Notwithstanding, code reviews can also delay the incorporation of a code change into a code base, thus slowing down the overall development process. Part of this delay is often a consequence of reviewers not understanding, becoming confused by, or being uncertain about the intention, behavior, or effect of a code change.We investigate the reasons and impacts of confusion in code reviews, as well as the strategies developers adopt to cope with confusion. We employ a concurrent triangulation strategy to combine the analyses of survey responses and of the code review comments, and build a comprehensive confusion framework structured along the dimensions of the review process, the artifact being reviewed, the developers themselves and the relation between the developer and the artifact. The most frequent reasons for confusion are the missing rationale, discussion of non-functional requirements of the solution, and lack of familiarity with existing code. Developers report that confusion delays the merge decision, decreases review quality, and results in additional discussions. To cope with confusion developers request information, improve familiarity with existing code, and discuss off-line. Based on the results, we provide a series of implications for tool builders, as well as insights and suggestions for researchers. The results of our work offer empirical justification for the need to improve code review tools to support developers facing confusion.

This paper describes a set of practices, techniques and conceptual tools that were tailored by combining ideas from Challenge-based Learning, Design Thinking and Lean Startup. While the former focuses on intrinsically motivating the students to engage in the learning process by tackling real-world problems aligned with their sense of purpose, the latter emphasizes aspects of the system being built, such as feasibility, viability, and desirability. The framework conception is as an innovative approach for mobile application development in long-term courses (1- or 2-year) involving programming, design, and business. We applied the methodology in the context of a regular undergraduate course including students with a diverse computing and non-computing backgrounds and collected their feedback during the process.

During code review, developers request clarifications, suggest improvements, or ask for explanations about the rationale behind the implementation choices. We envision the emergence of tools to support developers during code review based on the automatic analysis of the argumentation structure and communicative intentions conveyed by developers' comments. As a preliminary step towards this goal, we conducted an exploratory case study by manually classifying 499 questions extracted from 399 Android code reviews to understand the real communicative intentions they convey. We observed that the majority of questions actually serve information seeking goals. Still, they represent less than half of the annotated sample, with other questions being used to serve a wider variety of developers' communication goals, including suggestions, request for action, and criticism. Based on our findings we formulate hypotheses on communicative intentions in code reviews that should be confirmed or rejected by follow-up studies.

Swift is a new programming language developed by Apple as a replacement to Objective-C. It features a sophisticated error handling (EH) mechanism that provides the kind of separation of concerns afforded by exception handling mechanisms in other languages, while also including constructs to improve safety and maintainability. However, Swift also inherits a software development culture stemming from Objective-C being the de-facto standard programming language for Apple platforms for the last 15 years. It is, therefore, a priori unclear whether Swift developers embrace the novel EH mechanisms of the programming language or still rely on the old EH culture of Objective-C even working in Swift.

Summary Continuous integration (CI) is a software development practice that has been gaining increasing popularity in the last few years. However, we still miss a collection of experiences regarding how software developers perceive the idea of CI, in terms of its fundamental concepts, the reasons that motivate the adoption of this practice, the reasons for build breakage, and the benefits and problems related to CI. To shed light on this direction, we conducted a user survey with 158 CI users. Through a mostly qualitative investigation, we produce a list of findings that are not always obvious. For instance, we observed that (1) developers are not sure whether a job failure represents a failure or not; (2) inadequate testing is the most common technical reason related to build breakage, whereas lack of time plays a role on the social reasons; and (3) although some respondents reported that CI systems increase the confidence that the code is in a known state, some respondents also reported that there is a false sense of confidence when blindly trusting tests. This empirical study is particularly relevant to those interested in better understanding and fostering CI practices either in an open‐source or industrial setting.

Code reviews are an important mechanism for assuring quality of source code changes. Reviewers can either add general comments pertaining to the entire change or pinpoint concerns or shortcomings about a specific part of the change using inline comments. Recent studies show that reviewers often do not understand the change being reviewed and its context.Our ultimate goal is to identify the factors that confuse code reviewers and understand how confusion impacts the efficiency and effectiveness of code review(er)s. As the first step towards this goal we focus on the identification of confusion in developers' comments. Based on an existing theoretical framework categorizing expressions of confusion, we manually classify 800 comments from code reviews of the Android project. We observe that confusion can be reasonably well-identified by humans: raters achieve moderate agreement (Fleiss' kappa 0.59 for the general comments and 0.49 for the inline ones). Then, for each kind of comment we build a series of automatic classifiers that, depending on the goals of the further analysis, can be trained to achieve high precision (0.875 for the general comments and 0.615 for the inline ones), high recall (0.944 for the general comments and 0.988 for the inline ones), or substantial precision and recall (0.696 and 0.542 for the general comments and 0.434 and 0.583 for the inline ones, respectively). These results motivate further research on the impact of confusion on the code review process. Moreover, other researchers can employ the proposed classifiers to analyze confusion in other contexts where software development-related discussions occur, such as mailing lists.

ForkJoin framework is a widely used parallel programming framework upon which both core concurrency libraries and real-world applications are built. Beneath its simple and user-friendly APIs, ForkJoin is a sophisticated managed parallel runtime unfamiliar to many application programmers: the framework core is a work-stealing scheduler, handles fine-grained tasks, and sustains the pressure from automatic memory management. ForkJoin poses a unique gap in the compute stack between high-level software engineering and low-level system optimization. Understanding and bridging this gap is crucial for the future of parallelism support in JVM-supported applications. This paper describes a comprehensive study on parallelism bottlenecks in ForkJoin applications, with a unique focus on how they interact with underlying system-level features, such as work stealing and memory management. We identify 6 bottlenecks, and found that refactoring them can significantly improve performance and energy efficiency. Our field study includes an in-depth analysis of Akka — a real-world actor framework — and 30 additional open-source ForkJoin projects. We sent our patches to the developers of 15 projects, and 7 out of the 9 projects that replied to our patches have accepted them.

Mobile devices have become ubiquitous in the recent years, but the complaints about energy consumption are almost universal. On Android, the developer can choose among several different approaches to develop an app. In this paper, we investigate the impact of some of the most popular development approaches on the energy consumption of Android apps. Our study uses a testbed of 33 different benchmarks and 3 applications on 5 different devices to compare the energy efficiency and performance of the most commonly used approaches to develop apps on Android: Java, JavaScript, and C/C++ (through the NDK tools). In our experiments, Javascript was more energy-efficient in 75% of all benchmarks, while their Java counterparts consume up to 36.27x more energy (median of 1.97x). On the other hand, both Java and C++ outperformed JavaScript in most of the benchmarks. Based on these results, four Java applications were re-engineered to use a combination of Java and either JavaScript or C/C++ functions. For one of the apps, the hybrid solution using Java and C++ spent 10x less time and almost 100x less energy than a pure Java solution. The results were not uniform, however. For another app, when we restructured its implementation so as to minimize cross-language method invocations, the hybrid solution using Java and C++ took 8% longer to execute and consumed 11% more energy than a hybrid solution using Java and JavaScript. Since most Android apps are written solely in Java, the results of this study indicate that leveraging a combination of approaches may lead to non-negligible improvements in energy-efficiency and performance.

The recent introduction of the pull-based development model promoted agile development practices such as Code Reviews and Continuous Integration (CI). CI, in particular, is currently a standard development practice in open-source software (OSS) projects. Although it is well-known that OSS contributors have different involvements (e.g., while some developers drive the project, there is a long tail of peripheral developers), little is known about how the contributor's degree of participation can influence the build status of an OSS project. Through TravisTorrent's dataset, we compare the success rates of builds made by casual and non-casual contributors and what factors on their contributions may influence the build result. Our results suggest that there is no representative difference between their build success (they are similar in 85% of the analyzed projects), meaning that being a casual contributor is not a strong indicator for creating failing builds. Also, factors like the size of their contributions and the number of project configurations (jobs) have the potential of impacting the build success.

Java programmers are served with numerous choices of collections, varying from simple sequential ordered lists to sophisticated hashtable implementations. These choices are well-known to have different characteristics in terms of performance, scalability, and thread-safety, and most of them are well studied. This paper analyzes an additional dimension, energy efficiency. We conducted an empirical investigation of 16 collection implementations (13 thread-safe, 3 non-thread-safe) grouped under 3 commonly used forms of collections (lists, sets, and mappings). Using micro-and real world-benchmarks (Tomcat and Xalan), we show that our results are meaningful and impactful. In general, we observed that simple design decisions can greatly impact energy consumption. In particular, we found that using a newer hashtable version can yield a 2.19x energy savings in the micro-benchmarks and up to 17% in the real world-benchmarks, when compared to the old associative implementation. Also, we observed that different implementations of the same thread-safe collection can have widely different energy consumption behaviors. This variation also applies to the different operations that each collection implements, e.g, a collection implementation that performs traversals very efficiently can be more than an order of magnitude less efficient than another implementation of the same collection when it comes to insertions.

Analyzing the energy consumption of application level software is an emerging direction. This artifact makes available all the toolset and raw data needed to reproduce the main findings of our research paper. The artifact consists of: ● The source code of the micro-benchmarks analyzed, ● The source code of the case study used, ● The jRAPL tool, ● The raw energy data generated by the jRAPL tool with the source code of the experiments, ● The plotting scripts used to create the figures of the paper, based on the raw energy data.

Recent work has studied the effect that factors such as code obfuscation, refactorings and data types have on energy efficiency. In this paper, we attempt to shed light on the energy behavior of programs written in a lazy purely functional language, Haskell. We have conducted two empirical studies to analyze the energy efficiency of Haskell programs from two different perspectives: strictness and concurrency. Our experimental space exploration comprises more than 2000 configurations and 20000 executions. We found out that small changes can make a big difference in terms of energy consumption. For example, in one of our benchmarks, under a specific configuration, choosing one data sharing primitive (MVar) over another (TMVar) can yield 60% energy savings. In another benchmark, the latter primitive can yield up to 30% energy savings over the former. Thus, tools that support developers in quickly refactoring a program to switch between different primitives can be of great help if energy is a concern. In addition, the relationship between energy consumption and performance is not always clear. In sequential benchmarks, high performance is an accurate proxy for low energy consumption. However, for one of our concurrent benchmarks, the variants with the best performance also exhibited the worst energy consumption. To support developers in better understanding this complex relationship, we have extended two existing performance analysis tools to also collect and present data about energy consumption.

Energy consumption has become an increasingly important topic in software development, especially due to the ubiquity of mobile devices, and the choice of programming language can directly impact battery life. This paper presents a study on the issue of energy efficiency on the Android platform, comparing the performance and energy consumption of 33 different benchmarks in the two main programming languages employed in Android development: Java and JavaScript. Preliminary results show that Java applications may consume up to 36.27x more energy, with a median of 2.28x, than their JavaScript counterparts, in benchmarks that are mostly CPU-intensive. In some scenarios, though, Java benchmarks exhibited better energy efficiency, with JavaScript consuming up to 2.27x more energy. Based on these results, two Java applications were re-engineered, and through the insertion of JavaScript functions, hybrid applications were produced. In both cases, improvements in energy efficiency were obtained. Considering that Android apps written in Java are the norm, results from this study indicate that using a combination of JavaScript and Java may lead to a non-negligible improvement in energy efficiency.

Recently, Apple released Swift, a modern programming language built to be the successor of Objective-C. In less than a year and a half after its first release, Swift became one of the most popular programming languages in the world, considering different popularity measures. A significant part of this success is due to Apple's strict control over its ecosystem, and the clear message that it will replace Objective-C in a near future. According to Apple, "Swift is a powerful and intuitive programming language[...]. Writing Swift code is interactive and fun, the syntax is concise yet expressive." However, little is known about how Swift developers perceive these benefits. In this paper, we conducted two studies aimed at uncovering the questions and strains that arise from this early adoption. First, we perform a thorough analysis on 59,156 questions asked about Swift on StackOverflow. Second, we interviewed 12 Swift developers to cross-validate the initial results. Our study reveals that developers do seem to find the language easy to understand and adopt, although 17.5% of the questions are about basic elements of the language. Still, there are many questions about problems in the toolset (compiler, Xcode, libraries). Some of our interviewees reinforced these problems.

Over the last years, energy consumption has become a first-class citizen in software development practice. While energy-efficient solutions on lower-level layers of the software stack are well-established, there is convincing evidence that even better results can be achieved by encouraging practitioners to participate in the process. For instance, previous work has shown that using a newer version of a concurrent data structure can yield a 2.19x energy savings when compared to the old associative implementation [75]. Nonetheless, little is known about how much software engineers are employing energy-efficient solutions in their applications and what solutions they employ for improving energy-efficiency. In this paper we present a qualitative study of "energy-aware commits". Using Github as our primary data source, we perform a thorough analysis on an initial sample of 2,189 commits and carefully curate a set of 371 energy-aware commits spread over 317 real-world non-trivial applications. Our study reveals that software developers heavily rely on low-level energy management approaches, such as frequency scaling and multiple levels of idleness. Also, our findings suggest that ill-chosen energy saving techniques can impact the correctness of an application. Yet, we found what we call "energy-aware interfaces", which are means for clients (e.g., Developers or end-users) to save energy in their applications just by using a function, abstracting away the low-level implementation details.

A growing number of software solutions have been proposed to address application-level energy consumption problems in the last few years. However, little is known about how much software developers are concerned about energy consumption, what aspects of energy consumption they consider important, and what solutions they have in mind for improving energy efficiency. In this paper we present the first empirical study on understanding the views of application programmers on software energy consumption problems. Using StackOverflow as our primary data source, we analyze a carefully curated sample of more than 300 questions and 550 answers from more than 800 users. With this data, we observed a number of interesting findings. Our study shows that practitioners are aware of the energy consumption problems: the questions they ask are not only diverse -- we found 5 main themes of questions -- but also often more interesting and challenging when compared to the control question set. Even though energy consumption-related questions are popular when considering a number of different popularity measures, the same cannot be said about the quality of their answers. In addition, we observed that some of these answers are often flawed or vague. We contrast the advice provided by these answers with the state-of-the-art research on energy consumption. Our summary of software energy consumption problems may help researchers focus on what matters the most to software developers and end users.

Java programmers are faced with numerous choices in managing concurrent execution on multicore platforms. These choices often have different trade-offs (e.g., performance, scalability, and correctness guarantees). This paper analyzes an additional dimension, energy consumption. It presents an empirical study aiming to illuminate the relationship between the choices and settings of thread management constructs and energy consumption. We consider three important thread management constructs in concurrent programming: explicit thread creation, fixed-size thread pooling, and work stealing. We further shed light on the energy/performance trade-off of three ``tuning knobs'' of these constructs: the number of threads, the task division strategy, and the characteristics of processed data. Through an extensive experimental space exploration over real-world Java programs, we produce a list of findings about the energy behaviors of concurrent programs, which are not always obvious. The study serves as a first step toward improving energy efficiency of concurrent programs on parallel architectures.

Context: Over recent years, software developers have been evaluating the benefits of both Service-Oriented Architecture and software fault tolerance techniques based on design diversity by creating fault-tolerant composite services that leverage functionally equivalent services, or variant services. Three major design issues need to be considered while building software fault-tolerant architectures based on design diversity namely, selection and execution of variants and selection of an adjudication algorithm to determine the correct or adjudicated result from the variants. Each design issue, in turn, can be realized by a set of alternative design solutions, which present different degrees of quality requirements (e.g. memory consumption and reliability). Objective: To investigate whether existing approaches for fault-tolerant composite services support the above mentioned design issues and to provide a detailed classification of the analysed approaches. Method: A systematic literature review of diversity-based approaches for fault-tolerant composite services, which compose our primary studies. Results: We found 17 primary studies providing direct evidence about the research question. Our findings reveal that the primary studies support a wide variety of design decisions. For example, (i) variant services may be chosen at different points during the software lifecycle; (ii) both parallel and sequential execution schemes have been addressed; and (iii) a variety of adjudication mechanisms were found amongst the target papers. Conclusion: We build up a broad picture of what design issues have been addressed by existing diversity-based approaches for fault-tolerant composite services. Finally, practical issues and difficulties are summarized and directions for future work are suggested.

Several studies argue that exception handling code usually has poor quality and that it is commonly neglected by developers. Moreover, it is said to be the least understood, documented, and tested part of the implementation of a system. However, there are very few studies that attempt to understand developers' perceptions about exception handling, in general, and exception handling bugs, in particular. In this paper, we present the results of a survey conducted with 154 developers that aims to fill in this gap. According to the respondents of the survey, exception handling code is in fact documented and tested infrequently. Also, many of the respondents have had to fix exception handling bugs, in particular those caused by empty catch blocks or exceptions caught unintentionally. The respondents believe that exception handling bugs are more easily fixed than other kinds of bugs. Also, we found out a significant difference in the opinion of the respondents pertaining to the quality of the exception handling code: more experienced developers tend to believe that it is worse. We present a comprehensive classification of exception handling bugs based on the study results.

A number of approaches based on design diversity moderate the communication between clients and functionally equivalent services, i.e., variant services, to tolerate software faults in service-oriented applications. Nevertheless, it is unclear whether variant services are actually diverse and fail on disjoint subsets of the input space. In a previous work, we proposed an experimental setup to assess design diversity of variant services that realize a requirements specification. In this work, we utilize the proposed experimental setup to assess the design diversity of a number of third-party Web services adhering to seven different requirements specifications. In this paper, we describe in detail the main findings and lessons learnt from this empirical study. Firstly, we investigate whether variant services are in fact diverse. Secondly, we investigate the effectiveness of service diversity for tolerating faults. The results suggest that there is diversity in the implementation of variant services. However, in some cases, this diversity might not be sufficient to improve system reliability. Our findings provide an important knowledge basis for engineering effective fault-tolerant service applications.

Background: A number of approaches leverage design diversity to tolerate software design faults in service-oriented applications. The use of design diversity depends on the assumption that functionally equivalent services, i.e., variant services, rarely fail on the same input case. However, there are no directives to assess whether variant services are actually diverse and fail on disjoint subsets of the input space. Aim: To provide proper assessment of service diversity in order to achieve a high level of reliability by employing either a diversity-based solution with the variant services or a single service that exhibits higher reliability than would be the case if design diversity was adopted. Method: We propose an experimental setup that encompasses (i) a set of directives to organize the preparation and execution of the experiment to investigate service diversity; (ii) investigation of whether variant services are actually diverse by using statistical tests; and (iii) an analysis of if and by how much the reliability of a diversity-based solution that leverages voters is an improvement over one that uses a single service. We evaluated the applicability and usefulness of the proposed experimental setup by employing it to assess diversity of variant services adhering to four different requirements specifications. For each specification, we analysed three different services. Results: We found that the proposed directives were effective for the purposes of this assessment. Assessment results demonstrated that services implementing the four requirements specifications are actually diverse at a 0.05 significance level. For two of the specifications, coincident failures of two or more services are infrequent enough to promote gains in overall system reliability. Conclusions: Our findings reveal threats to the effectiveness of service-oriented applications that relies on design diversity to tolerate faults. This in turn reinforces the usefulness of our experimental setup, which supports researchers' decision making when selecting variants for building fault-tolerant service applications.

Background: Despite the number of empirical studies that assess Aspect-Oriented Software Development (AOSD) techniques, more research is required to investigate, for example, how software maintainability is impacted when these techniques are employed. One way to minimize the effort and increase the reliability of results in further research is to systematize empirical studies in Aspect-Oriented Software Maintainability (AOSM). In this context, metrics are useful as indicators to quantify software quality attributes, such as maintenance. Currently, a high number of metrics have been used throughout the literature to measure software maintainability. However, there is no comprehensive catalogue showing which metrics can be used to measure AOSM. Aim: To identify an AOSM metrics suite to be used by researchers in AOSM research. Method: We performed a systematic mapping study based on Kitchenham and Charters' guidelines, which derived a research protocol, and used well known digital libraries engines to search the literature. Conclusions: A total of 138 primary studies were selected. They describe 67 aspect-oriented (AO) maintainability metrics. Also, out of the 575 object-oriented maintainability metrics that we analyzed, 469 can be adapted to AO software. This catalogue provides an objective guide to researchers looking for maintainability metrics to be used as indicators in their quantitative and qualitative assessments. We provide information such as authors, metrics description, and studies that used the metric. Researchers can use this information to decide which metrics are more suited for their studies.

Design by Contract (DbC) is a technique for developing and improving functional software correctness through definition of "contracts" between client classes and their suppliers. Such contracts are enforced during runtime and if any of them is violated a runtime error should occur. Runtime assertions checkers (RACs) are a well-known technique that enforces such contracts. Although they are largely used to implement the DbC technique in contemporary languages, like Java, studies have shown that characteristics of contemporary exception handling mechanisms can discard contract violations detected by RACs. As a result, a contract violation may not be reflected in a runtime error, breaking the supporting hypothesis of DbC. This paper presents an error recovery technique for RACs that tackles such limitations. This technique relies on aspect-oriented programming in order to extend the functionalities of existing RACs stopping contract violations from being discarded. We applied the recovery technique on top of five Java-based contemporary RACs (i.e., JML/jml, JML/ajml, JContractor, CEAP, and Jose). Preliminary results have shown that the proposed technique could actually prevent the contract violations from being discarded regardless of the characteristics of the exception handling code of the target application.

Abstract This paper presents an in‐depth study of the adequacy of the AspectJ language for modularizing and reusing exception‐handling code. The study consisted of refactoring existing applications so that the code responsible for implementing error‐handling strategies was moved to newly created exception handler aspects. We have performed quantitative assessments of five systems—four object‐oriented and one aspect‐oriented—based on four key quality attributes, namely separation of concerns, coupling, cohesion, and conciseness. Our investigation also included a multi‐perspective analysis of the refactored systems, including (i) the extent to which error‐handling aspects can be reused, (ii) the beneficial and harmful aspectization scenarios for exception handling, and (iii) the scalability of AOP to support the modularization of exception handling in the presence of other aspects. Copyright © 2009 John Wiley & Sons, Ltd.

Software product lines (SPLs) enable modular, large-scale reuse through a software architecture addressing multiple core and varying features. To reap the benefits of SPLs, their designs need to be stable. Design stability encompasses the sustenance of the product line’s modularity properties in the presence of changes to both the core and varying features. It is usually assumed that aspect-oriented programming promotes better modularity and changeability of product lines. However, there is no empirical evidence on its efficacy to prolong design stability of product lines in realistic development scenarios. This paper reports a quantitative study that evolves two SPLs to assess various facets of design stability of aspect-oriented implementations. Our investigation focused upon a multi-perspective analysis of the evolving product lines in terms of modularity, change propagation, and feature interaction.

It is usually assumed that exception handling code can be better modularized by the use of aspect-oriented programming (AOP) techniques. However, recent studies argue that the ad hoc use of AOP can be detrimental to the quality of a system. When refactoring exception handling code to aspects, developers and maintainers need to follow clear and simple principles to obtain a well-structured system design. Otherwise, typical problems that stem from poorly designed/implemented error handling code will arise, e.g. resource leaking and swallowed exceptions. In this paper, we propose a classification for error handling code based on the factors that we found out have more influence on its aspectization. Moreover, we present a scenario catalog comprising combinations of these factors and analyze how these scenarios positively or negatively affect the task of aspectizing exception handling. We evaluated the proposed catalog through a case study where we used it to guide the aspectization of exception handling in two real systems.

It is usually assumed that the implementation of exception handling can be better modularized by the use of aspect-oriented programming (AOP). However, the trade-offs involved in using AOP with this goal are not well-understood. This paper presents an in-depth study of the adequacy of the AspectJ language for modularizing exception handling code. The study consisted in refactoring existing applications so that the code responsible for implementing heterogeneous error handling strategies was moved to separate aspects. We have performed quantitative assessments of four systems - three object-oriented and one aspect-oriented - based on four quality attributes, namely separation of concerns, coupling, cohesion, and conciseness. Our investigation also included a multi-perspective analysis of the refactored systems, including (i) the reusability of the aspectized error handling code, (ii) the beneficial and harmful aspectization scenarios, and (iii) the scalability of AOP to aspectize exception handling in the presence of other crosscutting concerns.

Exception handling is a structuring technique that facilitates the design of software systems by encapsulating the process of error recovery. In this paper, we present a systematic approach for incorporating exceptional behaviour in the development of component-based software. The premise of our approach is that components alone do not provide the appropriate means to deal with exceptional behaviour in an effective manner, hence the need to consider the notion of collaborations for capturing the interactive behaviour between components when error recovery involves more than one component. The feasibility of the approach is demonstrated in terms of the mining control system case study. Copyright © 2004 John Wiley & Sons, Ltd.