Abstract Context Modern software systems change continuously. Larger changes like architectural redesigns, feature additions, or system-wide refactorings regularly impact the file structures within a software repository (i.e., developers adding, deleting, or moving files). Objective While a normal evolutionary phenomenon, file-structure changes in software repositories have received little attention in past research. An important question that arises is whether outliers (i.e., changes increasing or decreasing file structures more strongly) are potential signs of quality issues in repository management and tooling. Method In this article, we contribute the first large-scale study on outliers of file-structure changes. To this end, we investigated more than 12.2 million file-structure changes from 94,247 GitHub repositories that span ten programming languages. We first performed a quantitative analysis of all these changes to establish a baseline regarding the depths of file structures and changes to these. Using this baseline, we identified and manually inspected 3,049 outliers. Results Our quantitative data shows that file-structure changes are pervasive in the evolution of software repositories, representing 17.9% of all commits across the studied projects. Via our manual inspection, we found that outliers are often associated with programming errors, initial setups, and misconfigurations of package managers. Thus, they are an indicator of potential mistakes and quality problems. Conclusions Our findings demonstrate that current practices and tools for managing software repositories should take file-structure changes into account. This could help practitioners monitor for and mitigate erroneous or unintended structural changes. Researchers can use our methodology and findings to design follow-up studies and new techniques for more robust repository management.

Deforestation (i.e., the removal or destruction of forests by humans), particularly illegal, is a major cause of ecological and environmental problems. To combat illegal deforestation, the World Wildlife Fund (WWF) has developed an open-source R package to predict deforestation around the world using machine learning. The package has been used by and customized to various countries, providing immense value for environmental protection. However, the package was implemented by domain experts without software engineering background, resulting in an unstructured development process, a monolithic codebase, and a lack of documentation on processes and code. Aiming to build an open-source community to improve and maintain the package, the WWF team decided to focus on enhancing the accessibility and attractiveness of the codebase for newcomers. Supporting this goal, we conducted an action-research-like project using Scrum to improve the code quality, tooling, testing, processes, and documentation while also establishing practices to sustain and build upon these improvements. In this article, we describe this project and share our insights into opening an R package to make it more accessible for external open-source contributors. Our insights include guidance on communicating design decisions to domain experts without a software engineering background and on how to train them in software engineering practices. Further insights highlight the specific challenges of working with R packages. Lastly, our work showcases the contributions that software engineering can make to support environmental protection and can guide future projects in this direction.

Platform strategies, originating from pure hardware platforms, have proven effective in optimizing development processes in the automotive domain. Over time, software-driven innovations have emerged as the primary origin of novel features in automotive systems, as exemplified by functionalities like lane-keeping assistants, traffic-sign recognition, and the prospect of autonomous driving. To address the growing importance of software, automotive manufacturers progressively incorporate principles of software-platform engineering, for instance, by adopting software product lines. However, a notable gap still persists in thoroughly managing all facets of a cyber–physical automotive system, which involves hardware, software, electronics, variability, and their complex interactions. Efforts to address this gap and to achieve holistic platform management have resulted in electrics/electronics platforms to emerge in the automotive domain; but these platforms and transitioning towards them have not been fully worked out, yet. In this article, we contribute to addressing this gap by proposing a methodology for holistic electrics/electronics platform release management. We present detailed explanations of our methodology and its individual guidelines, which we derived from practical requirements and subsequently validated through expert assessments. Specifically, we conducted a series of workshops involving eight practitioners at a large international automotive manufacturer. Our methodology and insights can help other researchers and practitioners who work on adopting electrics/electronics platform management and emphasize possible directions for future research.

• Software has become a primary concern in the automotive industry. • There is a lack of decision-making support for managing automotive platforms. • Practitioners need efficient processes and tools for product-structuring concepts. • We prioritize requirements for decision making for electrics/electron platforms. With modern vehicles becoming increasingly software driven, long-standing automotive manufacturers face challenges in integrating software into their historically hardware-centered platforms. Even though hardware and software platforms leverage the same ideas, manufacturers must still unify hardware, software, and electrics/electron components; as well as their intricate interconnections and variability. To address this challenge, automotive manufacturers are adopting product-structuring and variant-management concepts from different areas to develop a holistic platform perspective. Here, particularly decision making has manifested as a key problem to inform the involved stakeholders and assess different options. Specifically, simply adding software to a static hardware platform is insufficient to address the current digital transformation in the automotive industry: the new platform dimensions must be understood, changes balanced against each other, and necessary rework defined to achieve holistic platform management. In this article, we contribute to tackling this problem by presenting a mixed-method study through which we derived a set of requirements to support decision-making for today’s automotive electrics/electron platforms. We used requirements and insights from our previous work, which we refined and validated via expert workshops and a questionnaire with 76 experts from the automotive industry. Besides confirming and validating our previous work, we contribute a new consolidated overview of requirements and criteria together with a prioritization by practitioners. The findings of our work offer valuable guidance for automotive practitioners, particularly for adopting product-structuring concepts like electrics/electron platforms and making decisions in the process. Our contributions further substantiate existing evidence and can guide research on developing decision-making techniques for complex electrics/electron platforms.

The rapidly growing number of scientific publications poses numerous challenges for researchers engaged in literature analyses. Structured methodologies like systematic literature reviews are becoming increasingly expensive, considering their attempt to cover all relevant publications. Despite the increasing efforts needed, the importance of literature reviews also leads to an increasing growth in their number. While there are support techniques (e.g., guidelines, tools, checklists) for conducting literature analyses, a concise and clear overview of such techniques for assessing the quality of the analysis itself is missing. Such an overview can help researchers identify techniques for their work, understand ambiguities between them, support peer reviews, and guide future research by highlighting open gaps. In this paper, we address this lack of an overview by identifying existing techniques for assessing the quality of systematic literature reviews, comparing their properties, and discussing their pros and cons. For this purpose, we elicited 14 techniques through a systematic literature search covering 15 years (2007–2021). Overall, our contributions can help researchers identify feasible techniques for assessing the quality of literature analyses and can guide the development of new techniques, thereby facilitating the conduct and improving the quality of literature analyses.

Outside of performance-focused domains, research software is typically designed with output in mind rather than runtime efficiency. So, the resulting software consumes more resources (time, hardware) and is less scalable, hindering larger or longitudinal studies without adaptations. In this paper, we report our experiences of iteratively identifying and optimizing performance bottlenecks to enable such analyses in an established research software. Specifically, we applied a top-down strategy to Arcan, an architecture-smell detection tool, to develop a tool (AsTdEA) for tracing architecture smells through software evolution. To identify performance bottlenecks and benchmark our improvements, we used the Qualitas Corpus and a custom dataset. We achieved a reduction in processing time of approx. 98 % and reduced the runtime complexity from almost quadratic to close-to-linear. By sharing our process and insights, we hope to guide researchers in optimizing their research software in the future.

In the past years, a growing number of highly-automated systems has build on Artificial-Intelligence (AI) capabilities, for example, self-driving vehicles or predictive health-state diagnoses. As for any software system, there is a risk that misbehavior occurs (e.g., system failure due to bugs) or that malicious actors aim to misuse the system (e.g., generating attack scripts), which can lead to safety and security incidents. While software safety and security incidents have been studied in the past, we are not aware of research focusing on the specifics of AI incidents. With this paper, we aim to shed light on this gap through a case survey of 240 incidents that we elicited from four datasets comprising safety and security incidents involving AI from 2014 to 2023. Using manual data analyses and automated topic modeling, we derived relevant topics as well as the major issues and contexts in which the incidents occurred. We find that the topic of AI incidents is, not surprisingly, becoming more and more relevant, particularly in the contexts of autonomous driving and process-automation robotics. Regarding security and its intersection with safety, most incidents connect to generative AI (i.e., large-language models, deep fakes) and computer-vision systems (i.e., facial recognition). This emphasizes the importance of security to also ensure safety in the context of AI systems, with our results further revealing a high number of serious consequences (system compromise, human injuries) and major violations of confidentiality, integrity, availability, as well as authorization. We hope to support practitioners and researchers in understanding major safety and security issues to support the development of more secure, safe, and trustworthy AI systems.

Forking is a common practice in open-source and industrial software development, leading to the emergence of complex fork ecosystems. Understanding the evolution and relationships within such ecosystems is crucial for developers and project managers to ensure that useful changes are merged back into the original project or synchronized between forks. However, understanding complex fork ecosystems with up to tens of thousands of forks in different states (e.g., abandoned, co-evolving) is challenging, with visualizations being a means to address this challenge. In this paper, we investigate six visualizations designed to provide key insights into the dynamics of fork ecosystems. We started our work by analyzing GitHub community feedback on the official Network Graph visualization for fork ecosystems and categorized the fork-related tasks mentioned by developers in 237 comments. Then, we designed our visualization prototype VisFork, which contains six different visualizations that serve the three most frequently mentioned tasks. These visualizations allow users to explore temporal patterns, commit classifications, and collaboration dynamics across a fork ecosystem. Through a user study involving 10 GitHub community participants and seven students, we evaluated the usefulness of the visualizations. The results demonstrate the potential of VisFork to provide valuable insights into fork ecosystems, with positive feedback on the visualizations, but also suggestions for further improvements. Our work contributes to the development of user-centered tools that help to understand the intricacies of fork-based development and promote collaborative software-development practices.

Abstract Context: Empirical studies with human participants (e.g., controlled experiments) are established methods in Software Engineering (SE) research to understand developers’ activities or the pros and cons of a technique, tool, or practice. Various guidelines and recommendations on designing and conducting different types of empirical studies in SE exist. However, the use of financial incentives (i.e., paying participants to compensate for their effort and improve the validity of a study) is rarely mentioned Objective: In this article, we analyze and discuss the use of financial incentives for human-oriented SE experimentation to derive corresponding guidelines and recommendations for researchers. Specifically, we propose how to extend the current state-of-the-art and provide a better understanding of when and how to incentivize. Method: We captured the state-of-the-art in SE by performing a Systematic Literature Review (SLR) involving 105 publications from six conferences and five journals published in 2020 and 2021. Then, we conducted an interdisciplinary analysis based on guidelines from experimental economics and behavioral psychology, two disciplines that research and use financial incentives. Results: Our results show that financial incentives are sparsely used in SE experimentation, mostly as completion fees. Especially performance-based and task-related financial incentives (i.e., payoff functions) are not used, even though we identified studies for which the validity may benefit from tailored payoff functions. To tackle this issue, we contribute an overview of how experiments in SE may benefit from financial incentivisation, a guideline for deciding on their use, and 11 recommendations on how to design them. Conclusions: We hope that our contributions get incorporated into standards (e.g., the ACM SIGSOFT Empirical Standards), helping researchers understand whether the use of financial incentives is useful for their experiments and how to define a suitable incentivisation strategy.

The COVID-19 pandemic has drastically changed daily life and required fast responses to new situations, such as restricted public life. A major means to limit infections have been contact-tracing apps that inform an individual about a potential infection, helping to initiate countermeasures faster. While different tracing apps have been compared technologically, we are not aware of studies providing insights into their development processes during the pandemic emergency situation. To address this gap, we report an exploratory case study on how the German open-source Corona-Warn-App has been developed at SAP SE—and how other organizations (e.g., Deutsche Telekom AG), researchers, and individual developers contributed. We elicited data on the process, practices, and challenges by interviewing six developers at SAP SE, analyzing documentation, and discussing our data with an expert on the app’s development. Overall, we provide insights into how the development process of the Corona-Warn-App differed from other projects at SAP SE (e.g., testing), discuss the causes (i.e., public interest causing researchers to perform tests), and study the consequences (i.e., emergency tickets by researchers). Our findings can guide organizations when developing software in similar emergency situations (e.g., pandemics) in which reliable software needs to be developed within a short period of time.

Architecture smells are a widely established concept to describe symptoms of software degradation by measuring perceived violations of software-design principles. As such, architecture smells can help developers assess and understand the architectural quality of their software system. However, research has rarely been concerned with how architecture smells evolve and whether they actually foster software degradation during a system’s evolution. Building on our previous work in this direction, we present extended techniques for measuring architecture smells, novel visualizations, as well as an empirical study of how architecture smells evolve and what typical patterns they exhibit in 485 releases of 14 open-source systems. Among others, the results of our study indicate that especially cyclic dependencies on the class-level are prone to becoming highly complex over time, with one of the reasons being the continued merging of smells, most often resulting in tangled multi-hubs. Moreover, we found unstable dependencies to mostly grow slowly over time, whereas hub-like dependencies remain rather stable during a system’s evolution. These findings are valuable for practitioners to identify and tackle system degeneration, as well as for researchers to scope new research on managing architecture smells and technical debt.

The COVID-19 virus has caused a global pandemic that has heavily impacted daily life. Rapid advances in testing and vaccinating led to an additional use case besides the well-known contact-tracing apps: certificate-verification systems. Verification systems are often commissioned by local authorities to enable more public life, and are often developed by smaller organizations or startups. So, the development of verification systems differs from other software projects, featuring interesting and unique properties. In this article, we present an experience report on the development of one verification system by a German startup, focusing on three properties: working in a pandemic, developing a product for handling a pandemic, and the startup context. To this end, we surveyed nine startup developers and analyzed the results with two experts from the startup. We found that the developers focused on fast delivery to cope with the time pressure of releasing the verification system, which is why some phases of typical development processes were hardly carried out. As a result, while the verification system is successful, we also identified negative effects of the properties (e.g., programming mistakes, well-being). We discuss our findings to guide researchers and practitioners in preparing for software engineering in future emergencies. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Maintaining a software system is a continuous and complex process, typically following a workflow defined by the responsible organization. However, in practice, developers often deviate from the defined process due to personal preferences, varying customer requirements, or urgent deadlines. Such deviations may cause problems later on, or they may indicate potential process improvements. To deal with deviations and improve processes, it is first necessary to fully understand the processes actually employed by developers. For this purpose, process-mining techniques have been proposed that primarily build on version-control data. In this paper, we present a complementary process-mining technique that uses Jira issues to recover process activities not visible in version-control data, particularly focusing on developers’ interactions with issues and each other. We conducted a case study with 74 repositories of 24 developer teams from one large international company to understand the technique’s merits. Our technique revealed process differences across teams and depending on the types of Jira issues, providing novel insights for the company that helped to better understand the employed processes.

Code is often reused to facilitate collaborative development, to create software variants, to experiment with new ideas, or to develop new features in isolation. Social-coding platforms, such as GitHub, enable enhanced code reuse with forking, pull requests, and cross-project traceability. With these concepts, forking has become a common strategy to reuse code by creating clones (i.e., forks) of projects. Thereby, forking establishes fork ecosystems of co-existing projects that are similar, but developed in parallel, often with rather sporadic code propagation and synchronization. Consequently, forked projects vary in quality and often involve redundant development efforts. Unfortunately, as we will show, many projects do not benefit from test cases created in other forks, even though those test cases could actually be reused to enhance the quality of other projects. We believe that reusing test cases—in addition to the implementation code—can improve software quality, software maintainability, and coding efficiency in fork ecosystems. While researchers have worked on test-case-reuse techniques, their potential to improve the quality of real fork ecosystems is unknown. To shed light on test-case reusability, we study to what extent test cases can be reused across forked projects. We mined a dataset of test cases from 305 fork ecosystems on GitHub—totaling 1,089 projects—and assessed the potential for reusing these test cases among the forked projects. By performing a manual inspection of the test cases' applicability, by transplanting the test cases, and by analyzing the causes of non-applicability, we contribute an understanding of the benefits (e.g., uncovering bugs) and of the challenges (e.g., automated code transplantation, deciding about applicability) of reusing test cases in fork ecosystems.

Architecture smells indicate violations of software-design principles. So, identifying and assessing architecture smells facilitates refactorings to reduce technical debt and ensure maintainability. Detecting architecture smells in only one version at a time provides a static and limited picture, since, for example, historical trends remain obfuscated. Both, for practitioners and researchers, obtaining information on how specific architecture smells evolved over time can yield valuable insights, be it for avoiding the growth of critical smells, grasping the code’s degradation, or getting a better understanding of development processes. To support such analyses, we developed our tool AsTdEA, which tracks architecture smells throughout a system’s evolution. AsTdEA runs a modified version of the architecture-smell detection tool Arcan and allows the automated batch-processing of multiple versions of one or multiple systems. First, AsTdEA generates data on the components that are involved in each smell on a version-to-version basis and how these intra-version smells are related with one another across the entire system history, forming inter-version smells. Second, for every intra-version smell, inter-version smell, and system version, AsTdEA outputs a multitude of properties, which we have already used for multiple empirical studies. In this paper, we show the implementation and use of AsTdEA, as well as the lessons that we learned during its development and how we want to improve it in the future.

Configurable software systems and software product lines build on features as first class entities for reasoning about commonalities and variability among system variants. While it is desirable to have modular features, this is not always achievable and research has shown that features interact frequently, which can come with negative effects like security vulnerabilities or bugs. Intensive research has been conducted regarding how and when features interact, focusing primarily on the implementation level and the variability mechanism therein. However, besides such structural, explicit feature dependencies represented in the code, there may also be more subtle, implicit feature dependencies. In this paper, we build on the idea that the co-evolution of features (i.e., co-changes between features) can reveal implicit dependencies, and thus point to poor design decisions that result in additional maintenance effort. We present a technique for analyzing feature co-changes based on repository mining and association rule mining to identify features that commonly change together and to reveal implicit dependencies. Moreover, we provide a large-scale multi-case study on five C systems (e.g., Linux kernel) to evaluate whether and how frequent such evolutionary dependencies occur. Our results reveal that a) feature co-changes occur quite frequently (25 to 70% of commits), b) a considerable amount of changes are supported by association rules (i.e, do not occur by chance), and c) several of these co-changes cannot be explained via explicit feature interactions. Overall, our technique and study complement existing research on feature dependencies and interactions by providing means for understanding implicit dependencies that are represented by feature co-evolution.

Intentions are fundamental in software engineering, but they are typically only implicitly considered through different abstractions, such as requirements, use cases, features, or issues. Specifically, software engineers develop and evolve (i.e., change) a software system based on such abstractions of a stakeholder’s intention—something a stakeholder wants the system to be able to do. Unfortunately, existing abstractions are (inherently) limited when it comes to representing stakeholder intentions and are mostly used for documenting only. So, whether a change in a system fulfills its underlying intention (and only this one) is an essential problem in practice that motivates many research areas (e.g., testing to ensure intended behavior, untangling intentions in commits). We argue that none of the existing abstractions is ideal for capturing intentions and controlling software evolution, which is why intentions are often vague and must be recovered, untangled, or understood in retrospect. In this paper, we reflect on the role of intentions (represented by changes) in software engineering and sketch how improving their management may support developers. Particularly, we argue that continuously managing and controlling intentions as well as their fulfillment has the potential to improve the reasoning about which stakeholder requests have been addressed, avoid misunderstandings, and prevent expensive retrospective analyses. To guide future research for achieving such benefits for researchers and practitioners, we discuss the relationships between different abstractions and intentions, and propose steps towards managing intentions.

Microservice architectures are an emergent service-oriented paradigm widely used in industry to develop and deploy scalable software systems. The underlying idea is to design highly independent services that implement small units of functionality and can interact with each other through lightweight interfaces. Even though microservices are often used with success, their design and maintenance pose novel challenges to software engineers. In particular, it is questionable whether the intended independence of microservices can actually be achieved in practice. So, it is important to understand how and why microservices evolve during a system’s life-cycle, for instance, to scope refactorings and improvements of a system’s architecture or to develop supporting tools. To provide insights into how microservices evolve, we report a large-scale empirical study on the (co-)evolution of microservices in 11 open-source systems, involving quantitative and qualitative analyses of 7,319 commits. Our quantitative results show that there are recurring patterns of (co-)evolution across all systems, for instance, “shotgun surgery” commits and microservices that are largely independent, evolve in tuples, or are evolved in almost all changes. We refine our results by analyzing service-evolving commits qualitatively to explore the (in-)dependence of microservices and the causes for their specific evolution. The contributions in this article provide an understanding for practitioners and researchers on how microservices evolve in what way, and how microservice-based systems may be improved.

Academic research, by its nature, is notorious for being a challenging and demanding field. However, these challenges may become more complicated for certain groups of researchers rather than others. For instance, junior researchers who make up a large group of the current scientific community, particularly in the computer science domain, may face various types of impediments. A notable hindrance to realizing the impediments is the difficulty of precisely delineating them. In this paper, we report an empirical investigation to measure the level of awareness of any kind of obstacles that might hinder junior researchers’ publishing ability and disturb their involvement. For this purpose, we conducted a survey targeting active researchers from the Software Engineering field with a total of 52 respondents. We mainly focus on two types of aspects: peer reviewing models and collaboration. Our findings indicate that junior researchers seem to be more comfortable with double-blind reviewing models with more than half (approximately 67.2%) of them voting in favor of this model. The results also show a significant agreement that a lack of experience especially in academic writing and supervision problems constitute the most influential barriers to publishing. Our findings can help understand the needs of junior researchers and provide insights into our research community and its specific groups.

Background. The constantly increasing number of scientific publications poses challenges for researchers to monitor, select, and assess the publications relevant for their own research. Several guidelines for assessing publications manually during a literature analysis exist, with researchers proposing (semi-)automated techniques to facilitate such assessments. Aims. Still, research indicates that current techniques require further improvements to facilitate the analysis of large sets of publications. In this paper, we propose a semi-automatic technique with which we aim to improve in this direction by facilitating the selection and assessment of publications. Method. Our technique uses publicly available data of a publication, namely citation counts, article-level metrics, venue metrics, and altmetrics, to guide an analyst in assessing its relevance and impact. To evaluate the feasibility of our technique and the included metrics, we performed an experimental analysis to automatically assign ratings to the retrieved publications. Results. The results indicate that our technique can help an analyst in assessing publications, and reduce manual effort. Through our technique, we achieve an average accuracy of 53 % with a recall of 71 %. While precision (14 %) and F1-score (21 %) are—not surprisingly, due to the high number of irrelevant results returned by automatic searches in digital libraries—low, we see an improvement of these values for more recent reviews for which we could collect more complete data. However, some manual effort is still required for the final selection of papers. Conclusions. While it is not possible to achieve full automation for selecting and quality assessing publications, we can see that our metrics-based technique can be a helpful means to provide an initial rating for the analyst. Also, incorporating altmetrics seems to be a promising addition to rate comparably recent publications, helping researchers to further facilitate the execution of literature analyses.

The rapidly expanding corpus of scientific publications poses various types of challenges for researchers, mostly concerning the selection and assessment of publications relevant to their research topic. Therefore, the scientific community is actively involved in proposing solutions for effectively retrieving promising publications. Traditional bibliometrics, such as citations, are most commonly used for evaluating the research impact of a publication, in spite of rightful criticism. More recently, the newly introduced altmetrics (e.g., Tweets) have gained popularity and are constantly being investigated to understand their usefulness and potential benefits for assessing the significance of publications. Researchers argue that altmetrics can be used to reflect the importance of a publication beyond the boundaries of traditional bibliometrics. However, it is important to be aware of the limitations and threats arising from altmetrics, too. In this paper, we present a survey analysis to understand the usefulness of altmetrics and determine their ability of being used as quality indicators for scientific research. Based on the findings, we discuss whether altmetrics can support the quality assessment during literature analyses to assist the analyst by reducing the required time and manual effort.

Abstract With the increasing demand for customized systems and rapidly evolving technology, software engineering faces many challenges. A particular challenge is the development and maintenance of systems that are highly variable both in space (concurrent variations of the system at one point in time) and time (sequential variations of the system, due to its evolution). Recent research aims to address this challenge by managing variability in space and time simultaneously. However, this research originates from two different areas, software product line engineering and software configuration management, resulting in non-uniform terminologies and a varying understanding of concepts. These problems hamper the communication and understanding of involved concepts, as well as the development of techniques that unify variability in space and time. To tackle these problems, we performed an iterative, expert-driven analysis of existing tools from both research areas to derive a conceptual model that integrates and unifies concepts of both dimensions of variability. In this article, we first explain the construction process and present the resulting conceptual model. We validate the model and discuss its coverage and granularity with respect to established concepts of variability in space and time. Furthermore, we perform a formal concept analysis to discuss the commonalities and differences among the tools we considered. Finally, we show illustrative applications to explain how the conceptual model can be used in practice to derive conforming tools. The conceptual model unifies concepts and relations used in software product line engineering and software configuration management, provides a unified terminology and common ground for researchers and developers for comparing their works, clarifies communication, and prevents redundant developments.

If software quality assurance is postponed or even abandoned for a software system, maintenance and evolution become harder or even impossible. One widely known symptom for the degradation of system quality are Architecture Smells (ASs), which violate fundamental principles of software design. In this paper, we present a study on the evolution of ASs as well as on how and when they foster system degradation. Thus, we provide valuable insights regarding what ASs are meaningful to assure system quality. To this end, we analyzed the evolution of three types of ASs in 14 open-source systems with a total of 485 versions. We adapted indicators used in previous studies to assess the severity of ASs (e.g., growth, lifetime), and relate ASs to technical debt as another established indicator. Our results indicate that 1) ASs remain mostly stable compared to the code size of a system, 2) certain types of ASs, such as cyclic dependencies, have a greater impact on system degradation, and 3) certain properties determine how much an AS contributes to software degradation. These findings are valuable for practitioners to identify and tackle system degeneration, as well as for researchers to scope new research on managing ASs and technical debt.

Many developers and organizations implement apps for Android, the most widely used operating system for mobile devices. Common problems developers face are the various hardware devices, customized Android variants, and frequent updates, forcing them to implement workarounds for the different versions and variants of Android APIs used in practice. In this paper, we contribute the Android Compatibility checkS dataSet (AndroidCompass) that comprises changes to compatibility checks developers use to enforce workarounds for specific Android versions in their apps. We extracted 80,324 changes to compatibility checks from 1,394 apps by analyzing the version histories of 2,399 projects from the F-Droid catalog. With AndroidCompass, we aim to provide data on when and how developers introduced or evolved workarounds to handle Android incompatibilities. We hope that AndroidCompass fosters research to deal with version incompatibilities, address potential design flaws, identify security concerns, and help derive solutions for other developers, among others-helping researchers to develop and evaluate novel techniques, and Android app as well as operating-system developers in engineering their software.

Modern software development heavily relies on the reuse of functionalities through Application Programming Interfaces (APIs). However, client developers can have issues identifying the correct usage of a certain API, causing misuses accompanied by software crashes or usability bugs. Therefore, researchers have aimed at identifying API misuses automatically by comparing client code usages to correct API usages. Some techniques rely on certain API-specific graph-based data structures to improve the abstract representation of API usages. Such techniques need to compare graphs, for instance, by computing distance metrics based on the minimal graph edit distance or the largest common subgraphs, whose computations are known to be NP-hard problems. Fortunately, there exist many abstractions for simplifying graph distance computation. However, their applicability for comparing graph representations of API usages has not been analyzed. In this paper, we provide a comparison of different distance algorithms of API-usage graphs regarding correctness and runtime. Particularly, correctness relates to the algorithms’ ability to identify similar correct API usages, but also to discriminate similar correct and false usages as well as non-similar usages. For this purpose, we systematically identified a set of eight graph-based distance algorithms and applied them on two datasets of real-world API usages and misuses. Interestingly, our results suggest that existing distance algorithms are not reliable for comparing API usage graphs. To improve on this situation, we identified and discuss the algorithms’ issues, based on which we formulate hypotheses to initiate research on overcoming them.

Software features are intuitive entities used to abstract and manage the functionalities of a software system, for instance, in product-line engineering and agile software development. Nonetheless, developers rarely make features explicit in code, which is why they have to perform costly program comprehension and particularly feature location to (re-)gain knowledge about the code. In a previous paper, we conducted an experiment on how explicit feature traces impact developers' program comprehension by facilitating feature location. We found that annotating features in code improved program comprehension, while decomposing them into classes had a negative impact. Additionally, but not reported in that paper, we were concerned with understanding whether the different traces would impact developers' memory regarding the code and its features. To this end, we repeatedly asked our participants questions about the code on different levels of detail within time periods of two weeks. Since developers' memory decays over time, we expected that our participants would provide fewer correct answers over time, with differences depending on the feature traces in their code. Unfortunately, the actual results were inconclusive and up for interpretation, particularly due to challenges in designing an experiment on developers' memory. In this paper, we discuss our experimental design, the null results, and challenges for improving the methodology of future studies in this direction.

Abstract Feature models are a helpful means to document, manage, maintain, and configure the variability of a software system, and thus are a core artifact in software product-line engineering. Due to the various purposes of feature models, they can be a cross-cutting concern in an organization, integrating technical and business aspects. For this reason, various stakeholders (e.g., developers and consultants) may get involved into modeling the features of a software product line. Currently, collaboration in such a scenario can only be done with face-to-face meetings or by combining single-user feature-model editors with additional communication and version-control systems. While face-to-face meetings are often costly and impractical, using version-control systems can cause merge conflicts and inconsistency within a model, due to the different intentions of the involved stakeholders. Advanced tools that solve these problems by enabling collaborative, real-time feature modeling, analogous to Google Docs or Overleaf for text editing, are missing. In this article, we build on a previous paper and describe (1) the extended formal foundations of collaborative, real-time feature modeling, (2) our conflict resolution algorithm in more detail, (3) proofs that our formalization converges and preserves causality as well as user intentions, (4) the implementation of our prototype, and (5) the results of an empirical evaluation to assess the prototype’s usability. Our contributions provide the basis for advancing existing feature-modeling tools and practices to support collaborative feature modeling. The results of our evaluation show that our prototype is considered helpful and valuable by 17 users, also indicating potential for extending our tool and opportunities for new research directions.

Abstract Software product-line engineering is arguably one of the most successful methods for establishing large portfolios of software variants in an application domain. However, despite the benefits, establishing a product line requires substantial upfront investments into a software platform with a proper product-line architecture, into new software-engineering processes (domain engineering and application engineering), into business strategies with commercially successful product-line visions and financial planning, as well as into re-organization of development teams. Moreover, establishing a full-fledged product line is not always possible or desired, and thus organizations often adopt product-line engineering only to an extent that deemed necessary or was possible. However, understanding the current state of adoption, namely, the maturity or performance of product-line engineering in an organization, is challenging, while being crucial to steer investments. To this end, several measurement methods have been proposed in the literature, with the most prominent one being the Family Evaluation Framework (FEF), introduced almost two decades ago. Unfortunately, applying it is not straightforward, and the benefits of using it have not been assessed so far. We present an experience report of applying the FEF to nine medium- to large-scale product lines in the avionics domain. We discuss how we tailored and executed the FEF, together with the relevant adaptations and extensions we needed to perform. Specifically, we elicited the data for the FEF assessment with 27 interviews over a period of 11 months. We discuss experiences and assess the benefits of using the FEF, aiming at helping other organizations assessing their practices for engineering their portfolios of software variants.

Developers collaboratively implement large-scale industrial and open-source projects. Such projects pose several challenges for developers, as they require considerable knowledge about the project and its development processes, for instance, to fix bugs or implement new features. Understanding what information developer communities codify on how to contribute to their project is crucial, for example, to onboard new developers or for researchers to scope analysis techniques. In this paper, we report the results of a qualitative analysis of 25 Unix-like distributions, focusing on what information the communities codify publicly on contributing. The results reveal no dedicated strategies to codify information on contribution or development practices. Still, non-technical contributions are easy to identify, while information on the development is hard to collect---and mostly concerned with versioning and bug reporting. Our insights help to understand information-provisioning strategies, identify information sources, and scope analyses.

Many organizations and open-source projects use the C preprocessor (CPP) to implement configurability in their software systems. Despite extensive research, existing studies on the effects of CPP use on program comprehension are still limited to experiences, opinions, and empirical studies with narrow scopes. So, it is unclear whether the CPP actually leads to what is sometimes referred to as "#ifdef hell." In this paper, we expand the existing evidence on program comprehension in the presence of CPP directives, but we also highlight a surprising dilemma. We conducted an empirical study, including an experiment and a questionnaire, on the impact of refactoring CPP directives with 521 experienced software developers. The results indicate that, in contrast to previous findings, comprehension performance slightly worsened in terms of correctness when our participants worked on code with refactored CPP directives. However, in alignment with previous findings, they preferred the refactored code, considering it more comprehensible and easier to work with. This dilemma of objective performance versus subjective preference is a surprising outcome that has not been found before. We argue that our work motivates the need for more studies to understand this dilemma-which may significantly impact common beliefs in research and practice.

Developers spend most of their time with program comprehension, obtaining (or recovering) the knowledge they need to perform a task. Researchers have investigated the information needs of developers to understand what knowledge is important and to scope techniques, for example, to facilitate program comprehension, support knowledge recovery, or identify experts. Similarly, researchers analyzed developers' memory to understand how they forget, which essentially causes the need to recover knowledge. However, we are not aware of studies linking these research directions to investigate what knowledge developers aim to keep in their memory, allowing them to ask less and different questions during knowledge recovery. To address this gap, we conducted an interview survey with 17 experienced developers, in which we investigated 1) what knowledge developers consider important to remember; 2) whether developers can correctly recall knowledge about their (smaller) systems; and 3) how their self-assessment relates to their actual knowledge. Our results indicate, among others, that developers consider architecture and abstract code knowledge (e.g., its intent) as most important to remember, that the perceived importance relates to their ability to recall knowledge correctly, and that their self-assessment decreases while reflecting about their system. Based on these findings, we discuss research directions and practical implications for managing and recovering developers' knowledge.

Software reuse lowers development costs and improves the quality of software systems. Two strategies are common: clone & own (copying and adapting a system) and platform-oriented reuse (building a configurable platform). The former is readily available, flexible, and initially cheap, but does not scale with the frequency of reuse, imposing high maintenance costs. The latter scales, but imposes high upfront investments for building the platform, and reduces flexibility. As such, each strategy has distinctive advantages and disadvantages, imposing different development activities and software architectures. Deciding for one strategy is a core decision with long-term impact on an organization’s software development. Unfortunately, the strategies’ costs are not well-understood - not surprisingly, given the lack of systematically elicited empirical data, which is difficult to collect. We present an empirical study of the development activities, costs, cost factors, and benefits associated with either reuse strategy. For this purpose, we combine quantitative and qualitative data that we triangulated from 26 interviews at a large organization and a systematic literature review covering 57 publications. Our study both confirms and refutes common hypotheses on software reuse. For instance, we confirm that developing for platform-oriented reuse is more expensive, but simultaneously reduces reuse costs; and that platform-orientation results in higher code quality compared to clone & own. Surprisingly, refuting common hypotheses, we find that change propagation can be more expensive in a platform, that platforms can facilitate the advancement into innovative markets, and that there is no strict distinction of clone & own and platform-oriented reuse in practice.

Developing software systems in large organizations requires the cooperation of various organizational units and stakeholders. As software-development processes are distributed among such organizational units; and are constantly transformed to fulfill new domain regulations, address changing customer requirements, or adopt new software-engineering methods; it is challenging to ensure, measure, and steer—essentially monitor—the quality of the resulting systems. One means to facilitate such monitoring throughout whole software-development processes are key performance indicators, which provide a consolidated analysis of an organizations’ performance. However, it is also challenging to introduce key performance indicators for the software development of a large organization, as they must be implemented at and accepted by all relevant organizational units. In this paper, we report our experiences of introducing new key performance indicators for software-development processes at Volkswagen Financial Services AG, a large organization in the financial sector. We describe i) our methodology; ii) how we customized and use key performance indicators; iii) benefits achieved, namely improved monitoring and comparability, which help to define quality-improving actions; iv) and six lessons learned. These insights are helpful for other practitioners, providing an overview of a methodology they can adopt to assess the feasibility of key performance indicators as well as their benefits. Moreover, we hope to motivate research to investigate methods for introducing and monitoring key performance indicators to facilitate their adoption.

Abstract Open-science initiatives have gained substantial momentum in computer science, and particularly in software-engineering research. A critical aspect of open-science is the public availability of artifacts (e.g., tools), which facilitates the replication, reproduction, extension, and verification of results. While we experienced that many artifacts are not publicly available, we are not aware of empirical evidence supporting this subjective claim. In this article, we report an empirical study on software artifact papers (SAPs) published at the International Conference on Software Engineering (ICSE), in which we investigated whether and how researchers have published their software artifacts, and whether this had scientific impact. Our dataset comprises 789 ICSE research track papers, including 604 SAPs (76.6 %), from the years 2007 to 2017. While showing a positive trend towards artifact availability, our results are still sobering. Even in 2017, only 58.5 % of the papers that stated to have developed a software artifact made that artifact publicly available. As we did find a small, but statistically significant, positive correlation between linking to artifacts in a paper and its scientific impact in terms of citations, we hope to motivate the research community to share more artifacts. With our insights, we aim to support the advancement of open science by discussing our results in the context of existing initiatives and guidelines. In particular, our findings advocate the need for clearly communicating artifacts and the use of non-commercial, persistent archives to provide replication packages.

Developers use a large amount of their time to understand the system they work on, an activity referred to as program comprehension. Especially software evolution and forgetting over time lead to developers becoming unfamiliar with a system. To support them during program comprehension, we can employ knowledge recovery to reverse engineer implicit information from the system and the platform (e.g., GitHub) it is hosted on. However, to recover useful knowledge and to provide it in a useful way, we first need to understand what knowledge developers forget to what extent, what sources are reliable to recover knowledge, and how to trace knowledge to the features in a system. We tackle these three issues, aiming to provide empirical insights and tooling to support developers during software evolution and maintenance. The results help practitioners, as we support the analysis and understanding of systems, as well as researchers, showing opportunities to automate, for example, reverse-engineering techniques.

Developers spend a substantial amount of their time with program comprehension. To improve their comprehension and refresh their memory, developers need to communicate with other developers, read the documentation, and analyze the source code. Many studies show that developers focus primarily on the source code and that small improvements can have a strong impact. As such, it is crucial to bring the code itself into a more comprehensible form. A particular technique for this purpose are explicit feature traces to easily identify a program’s functionalities. To improve our empirical understanding about the effects of feature traces, we report an online experiment with 49 professional software developers. We studied the impact of explicit feature traces, namely annotations and decomposition, on program comprehension and compared them to the same code without traces. Besides this experiment, we also asked our participants about their opinions in order to combine quantitative and qualitative data. Our results indicate that, as opposed to purely object-oriented code: (1) annotations can have positive effects on program comprehension; (2) decomposition can have a negative impact on bug localization; and (3) our participants perceive both techniques as beneficial. Moreover, none of the three code versions yields significant improvements on task completion time. Overall, our results indicate that lightweight traceability, such as using annotations, provides immediate benefits to developers during software development and maintenance without extensive training or tooling; and can improve current industrial practices that rely on heavyweight traceability tools (e.g., DOORS) and retroactive fulfillment of standards (e.g., ISO-26262, DO-178B).

Feature models are arguably one of the most intuitive and successful notations for modeling the features of a variant-rich software system. Feature models help developers to keep an overall understanding of the system, and also support scoping, planning, development, variant derivation, configuration, and maintenance activities that sustain the system's long-term success. Unfortunately, feature models are difficult to build and evolve. Features need to be identified, grouped, organized in a hierarchy, and mapped to software assets. Also, dependencies between features need to be declared. While feature models have been the subject of three decades of research, resulting in many feature-modeling notations together with automated analysis and configuration techniques, a generic set of principles for engineering feature models is still missing. It is not even clear whether feature models could be engineered using recurrent principles. Our work shows that such principles in fact exist. We analyzed feature-modeling practices elicited from ten interviews conducted with industrial practitioners and from 31 relevant papers. We synthesized a set of 34 principles covering eight different phases of feature modeling, from planning over model construction, to model maintenance and evolution. Grounded in empirical evidence, these principles provide practical, context-specific advice on how to perform feature modeling, describe what information sources to consider, and highlight common characteristics of feature models. We believe that our principles can support researchers and practitioners enhancing feature-modeling tooling, synthesis, and analyses techniques, as well as scope future research.

Summary Mutation testing is an approach to assess the quality of test cases. Mutants are modified versions of a system that ideally compose faulty behaviour. Test cases for a system are effective if they kill these mutants. For software product lines, several works have addressed mutation testing to inject variability faults, which may only exist in some variants. These works focus on variability models or specific implementation techniques. In contrast, feature‐oriented programming has been rarely investigated, wherefore, we (1) derive corresponding mutation operators, (2) investigate the feasibility of our proposed and conventional operators on 4 software product lines, and (3) discuss open challenges in mutation testing of software product lines. The results show that our proposed operators are suitable to cause variability faults and extend the capabilities of conventional operators. Nonetheless, mutation testing of software product lines is comparably expensive, due to a high number of variants and mutants—resulting in equivalence and redundancy.

Being familiar with the source code of a program comprises knowledge about its purpose, structure, and details. Consequently, familiarity is an important factor in many contexts of software development, especially for maintenance and program comprehension. As a result, familiarity is considered to some extent in many different approaches, for example, to model costs or to identify experts. Still, all approaches we are aware of require a manual assessment of familiarity and empirical analyses of forgetting in software development are missing. In this paper, we address this issue with an empirical study that we conducted with 60 open-source developers. We used a survey to receive information on the developers' familiarity and analyze the responses based on data we extract from their used version control systems. The results show that forgetting is an important factor when considering familiarity and program comprehension of developers. We find that a forgetting curve is partly applicable for software development, investigate three factors - the number of edits, ratio of owned code, and tracking behavior - that can impact familiarity with code, and derive a general memory strength for our participants. Our findings can be used to scope approaches that have to consider familiarity and they provide insights into forgetting in the context of software development.

In practice, organizations often rely on the clone-and-own approach to reuse and customize existing systems. While increasing maintenance costs encourage some organizations to adopt their development processes towards more systematic reuse, others still avoid migrating to a reusable platform. Based on our experiences, a barrier preventing the adoption of software product lines is the fear of introducing new and more problematic bugs---during the migration or later on. We are aware of several works that automate software-product-line adoption, but they neglect the migration and maintenance of test cases. Automating the refactoring of tests can help to facilitate the adoption barrier, compare the quality after migrations, and support maintenance. In this vision paper, we i) discuss open research challenges that are based on our experiences and ii) sketch a first framework to develop automated solutions. Overall, we aim to illustrate our idea and initiate further research to facilitate the adoption and maintenance of software product lines.

Software-product-line engineering is an approach to systematically manage reusable software features and has been widely adopted in practice. Still, in most cases, organizations start with a single product that they clone and modify when new customer requirements arise (a.k.a. clone-and-own). With an increasing number of variants, maintenance can become challenging and organizations may consider migrating towards a software product line, which is referred to as extractive approach. While this is the most common approach in practice, techniques to extract variability from cloned variants still fall short in several regards. In particular, this accounts for the low accuracy of automated analyses and refactoring, our limited understanding of the costs involved, and the high manual effort. A main reason for these limitations is the lack of realistic case studies. To tackle this problem, we provide a set of cloned variants. In this paper, we characterize these variants and challenge the research community to apply techniques for reverse engineering feature models, feature location, code smell analysis, architecture recovery, and the migration towards a software product line. By evaluating solutions with the developer of these variants, we aim to contribute to a larger body of knowledge on this real-world case study.

The source code of highly-configurable software is challenging to comprehend, analyze, and test. In particular, it is hard to identify all configurations that comprise a certain code location. We contribute PCLocator, a tool suite that solves this problem by utilizing static analysis tools for compile-time variability. Using BusyBox and the Variability Bugs Database (VBDb), we evaluate the correctness and performance of PCLocator. The results show that we are able to analyze files in a matter of seconds and derive correct configurations in 95% of all cases.

Due to its fast and simple applicability, clone-and-own is widely used in industry to develop software variants. In cooperation with different companies for thermoelectric products, we implemented multiple variants of a heat monitoring tool based on clone-and-own. After encountering redundancy-related problems during development and maintenance, we decided to migrate towards a software product line. Within this paper, we describe this case study of migrating cloned variants to a software product line based on the extractive approach. The resulting software product line encapsulates variability on several levels, including the underlying hardware systems, interfaces, and use cases. Currently, we support monitoring hardware from three different companies that use the same core system and provide a configurable front-end. We share our experiences and encountered problems with cloning and migration towards a software product line---focusing on feature extraction and modeling in particular. Furthermore, we provide a lightweight, web-based tool for modeling, configuring, and implementing software product lines, which we use to migrate and manage features. Besides this experience report, we contribute most of the created artifacts as open-source and freely available for the research community.

Summary Software product lines enable developers to derive similar products from a common code base. Existing implementation techniques can be categorized as composition‐based and annotation‐based approaches, with both approaches promising complementary benefits. However, annotation‐based approaches are commonly used in practice despite composition allowing physical separation of features and, thus, improving traceability and maintenance. A main hindrance to migrate annotated systems toward a composition‐based product line is the challenging and time‐consuming transformation task. For a company, it is difficult to predict the corresponding costs, and a successful outcome is uncertain. To overcome such problems, a solution proposed by the previous work is to use a hybrid approach, utilizing composition and annotation simultaneously. Based on this idea, we introduce a stepwise migration process from annotation‐based toward composition‐based approaches to lower the adoption barrier of composition. This process itself is independent of used implementation techniques and enables developers to incrementally migrate toward composition. We support our approach with detailed examples by partially migrating a real‐world system. In detail, we describe the following: (1) our migration process, (2) its application on a real‐world system, and (3) discuss practical challenges we face. We implemented the proposed approach and show that appropriate tool support helps to migrate toward composition‐based product lines. Based on the case study, we show that the hybrid product lines work correctly and can compete with the performance of the original annotated system. However, the results also illustrate open issues that have to be solved to apply such migrations in practice.

Follow-the-sun is an approach to develop software by handing off the progress to different time zones as the day passes. Hence, this approach allows companies to work on a project 24 hours a day, potentially reducing its time-to-market. However, several challenges, such as time zone differences or handing off work, are often reported. In this paper, we describe a case study on a follow-the-sun approach that was applied in a German company. During the approach, we did rarely face the aforementioned challenges but experienced different ones. For this reason, the company put its approach on hold but benefited from the learned lessons. Overall, we report on a partly successful follow-the-sun approach and identify five important practices.

Program comprehension is an important aspect of developing and maintaining software, as programmers spend most of their time comprehending source code. Thus, it is the focus of many studies and experiments to evaluate approaches and techniques that aim to improve program comprehension. As the amount of corresponding work increases, the question arises how researchers address program comprehension. To answer this question, we conducted a literature review of papers published at the International Conference on Program Comprehension, the major venue for research on program comprehension. In this article, we i) present preliminary results of the literature review and ii) derive further research directions. The results indicate the necessity for a more detailed analysis of program comprehension and empirical research.

Companies are often forced to customize their software products. Thus, a common practice is to clone and adapt existing systems to new customer requirements. With the extractive approach, those derived variants can be migrated into a software product line. However, changing to a new development process is risky and may result in unnecessary costs. Therefore, companies apply cost estimations to predict whether another development approach is beneficial. Existing cost models for software-product-line engineering focus on development from scratch. Contrarily, the extractive approach is more common in practice but specialized models are missing. Thus, in this work we focus on product-line extraction from a set of legacy systems. We i) describe according cost factors, ii) put them in context with the development process and cost curves, and iii) identify open challenges in product-line economics. This way, our work supports cost estimations for the extractive approach and provides a basis for further research.