Deep learning (DL) systems present unique challenges in software engineering, especially concerning quality attributes like correctness and resource efficiency. While DL models excel in specific tasks, engineering DL systems is still essential. The effort, cost, and potential diminishing returns of continual improvements must be carefully evaluated, as software engineers often face the critical decision of when to stop refining a system relative to its quality attributes. This experience paper explores the role of MLOps practices―such as monitoring and experiment tracking―in creating transparent and reproducible experimentation environments that enable teams to assess and justify the impact of design decisions on quality attributes. Furthermore, we report on experiences addressing the quality challenges by embedding domain knowledge into the design of a DL model and its integration within a larger system. The findings offer actionable insights into the benefits of domain knowledge and MLOps and the strategic consideration of when to limit further optimizations in DL projects to maximize overall system quality and reliability.

Background: As empirical software engineering evolves, more studies adopt data strategies-approaches that investigate digital artifacts such as models, source code, or system logs rather than relying on human subjects. Synthesizing results from such studies introduces new methodological challenges. Aims: This study assesses the effects of model quantization on correctness and resource efficiency in deep learning (DL) systems. Additionally, it explores the methodological implications of aggregating evidence from empirical studies that adopt data strategies. Method: We conducted a research synthesis of six primary studies that evaluate model quantization. We applied the Structured Synthesis Method (SSM) to aggregate the findings, which combines qualitative and quantitative evidence through diagrammatic modeling. A total of 19 evidence models were extracted and aggregated. Results: The aggregated evidence indicates that model quantization weakly negatively affects correctness metrics while consistently improving resource efficiency metrics, including storage size, inference latency, and GPU energy consumption-a manageable trade-off for many DL deployment contexts. Evidence across quantization techniques remains fragmented, underscoring the need for more focused empirical studies per technique. Conclusions: Model quantization offers substantial efficiency benefits with minor trade-offs in correctness, making it a suitable optimization strategy for resource-constrained environments. This study also demonstrates the feasibility of using SSM to synthesize findings from data strategy-based research.

The rapid growth of language models, particularly in code generation, requires substantial computational resources, raising concerns about energy consumption and environmental impact. Optimizing language models inference resource utilization is crucial, and Small Language Models (SLMs) offer a promising solution to reduce resource demands. Our goal is to analyze the impact of deep learning serving configurations, defined as combinations of runtime engines and execution providers, on resource utilization, in terms of energy consumption, execution time, and computing-resource utilization from the point of view of software engineers conducting inference in the context of code generation SLMs. We conducted a technology-oriented, multi-stage experimental pipeline using twelve code generation SLMs to investigate energy consumption, execution time, and computing-resource utilization across the configurations. Significant differences emerged across configurations. CUDA execution provider configurations outperformed CPU execution provider configurations in both energy consumption and execution time. Among the configurations, TORCH paired with CUDA demonstrated the greatest energy efficiency, achieving energy savings from 37.99% up to 89.16% compared to other serving configurations. Similarly, optimized runtime engines like ONNX with the CPU execution provider achieved from 8.98% up to 72.04% energy savings within CPU-based configurations. Also, TORCH paired with CUDA exhibited efficient computing-resource utilization. Serving configuration choice significantly impacts resource utilization. While further research is needed, we recommend the above configurations best suited to software engineers’ requirements for enhancing serving resource utilization efficiency. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board .

The latest advancements in machine learning, specifically in foundation models, are revolutionizing the frontiers of existing software engineering (SE) processes. This is a bi-directional phenomenon, where (1) software systems are now challenged to provide AI-enabled features to their users, and (2) AI is used to automate tasks within the software development lifecycle. In an era where sustainability is a pressing societal concern, our community needs to adopt a long-term plan enabling a conscious transformation that aligns with environmental sustainability values. In this article, we reflect on the impact of adopting environmentally friendly practices to create AI-enabled software systems and make considerations on the environmental impact of using foundation models for software development.

Automated program repair (APR) aims to automatize the process of repairing software bugs in order to reduce the cost of maintaining software programs. While APR accuracy has significantly improved in recent years, its energy impact remains unstudied. The field of green software research aims to measure the energy consumption required to develop, maintain, and use software products. Our main goal is to define the foundation for measuring the energy consumption of the APR activity. We state that an environmentally sustainable (or green) APR tool achieves a good balance between the ability to correctly repair bugs and the amount of energy consumed during such process. We measure the energy consumption of 10 traditional APR tools for Java and 11 fine-tuned large-language models (LLM) trying to repair real bugs from Defects4J. The results of this study show the existing tradeoff between energy consumption and repairability. In particular, APR tools such as TBar and RepairLlama repair more bugs than other approaches at the expense of a higher energy consumption. Other tools, such as SimFix and the LLM CodeT5-large, provide a good tradeoff between energy consumption and repairability. We also present guidelines consisting of a set of recommendations for developing greener APR.

The growing use of large machine learning models highlights concerns about their increasing computational demands. While the energy consumption of their training phase has received attention, fewer works have considered the inference phase. For ML inference, the binding of ML models to the ML system for user access, known as ML serving, is a critical yet understudied step for achieving efficiency in ML applications.

Hugging Face (HF) has established itself as a crucial platform for the development and sharing of machine learning (ML) models. This repository mining study, which delves into more than 380,000 models using data gathered via the HF Hub API, aims to explore the community engagement, evolution, and maintenance around models hosted on HF - aspects that have yet to be comprehensively explored in the literature. We first examine the overall growth and popularity of HF, uncovering trends in ML domains, framework usage, authors grouping and the evolution of tags and datasets used. Through text analysis of model card descriptions, we also seek to identify prevalent themes and insights within the developer community. Our investigation further extends to the maintenance aspects of models, where we evaluate the maintenance status of ML models, classify commit messages into various categories (corrective, perfective, and adaptive), analyze the evolution across development stages of commits metrics and introduce a new classification system that estimates the maintenance status of models based on multiple attributes. This study aims to provide valuable insights about ML model maintenance and evolution that could inform future model development strategies on platforms like HF.

Background: The rise of machine learning (ML) systems has exacerbated their carbon footprint due to increased capabilities and model sizes. However, there is scarce knowledge on how the carbon footprint of ML models is actually measured, reported, and evaluated. Aims: This paper analyzes the measurement of the carbon footprint of 1,417 ML models and associated datasets on Hugging Face. Hugging Face is the most popular repository for pretrained ML models. We aim to provide insights and recommendations on how to report and optimize the carbon efficiency of ML models. Method: We conduct the first repository mining study on the Hugging Face Hub API on carbon emissions and answer two research questions: (1) how do ML model creators measure and report carbon emissions on Hugging Face Hub?, and (2) what aspects impact the carbon emissions of training ML models? Results: Key findings from the study include a stalled proportion of carbon emissions-reporting models, a slight decrease in reported carbon footprint on Hugging Face over the past 2 years, and a continued dominance of NLP as the main application domain reporting emissions. The study also uncovers correlations between carbon emissions and various attributes, such as model size, dataset size, ML application domains and performance metrics. Conclusions: The results emphasize the need for software measurements to improve energy reporting practices and the promotion of carbon-efficient model development within the Hugging Face community. To address this issue, we propose two classifications: one for categorizing models based on their carbon emission reporting practices and another for their carbon efficiency. With these classification proposals, we aim to encourage transparency and sustainable model development within the ML community.

Nowadays, AI-based systems have achieved outstanding results and have outperformed humans in different domains. However, the processes of training AI models and inferring from them require high computational resources, which pose a significant challenge in the current energy efficiency societal demand. To cope with this challenge, this research project paper describes the main vision, goals, and expected outcomes of the GAISSA project. The GAISSA project aims at providing data scientists and software engineers tool-supported, architecture-centric methods for the modelling and development of green AI-based systems. Although the project is in an initial stage, we describe the current research results, which illustrate the potential to achieve GAISSA objectives.

Current research in the computer vision field mainly focuses on improving Deep Learning (DL) correctness and inference time performance. However, there is still little work on the huge carbon footprint that has training DL models. This study aims to analyze the impact of the model architecture and training environment when training greener computer vision models. We divide this goal into two research questions. First, we analyze the effects of model architecture on achieving greener models while keeping correctness at optimal levels. Second, we study the influence of the training environment on producing greener models. To investigate these relationships, we collect multiple metrics related to energy efficiency and model correctness during the models’ training. Then, we outline the trade-offs between the measured energy efficiency and the models’ correctness regarding model architecture, and their relationship with the training environment. We conduct this research in the context of a computer vision system for image classification. In conclusion, we show that selecting the proper model architecture and training environment can reduce energy consumption dramatically (up to 98.83%) at the cost of negligible decreases in correctness. Also, we find evidence that GPUs should scale with the models’ computational complexity for better energy efficiency.

Abstract Software organisations aim to develop and maintain high‐quality software systems. Due to large amounts of behaviour data available, software organisations can conduct data‐driven software maintenance. Indeed, software quality assurance and improvement programs have attracted many researchers' attention. Bayesian Networks (BNs) are proposed as a log analysis technique to discover poor performance indicators in a system and to explore usage patterns that usually require temporal analysis. For this, an action research study is designed and conducted to improve the software quality and the user experience of a web application using BNs as a technique to analyse software logs. To this aim, three models with BNs are created. As a result, multiple enhancement points have been identified within the application ranging from performance issues and errors to recurring user usage patterns. These enhancement points enable the creation of cards in the Scrum process of the web application, contributing to its data‐driven software maintenance. Finally, the authors consider that BNs within quality‐aware and data‐driven software maintenance have great potential as a software log analysis technique and encourage the community to deepen its possible applications. For this, the applied methodology and a replication package are shared.

In despite of agile and rapid software development (ARSD) being researched and applied extensively, managing quality requirements (QRs) are still challenging. As ARSD processes produce a large amount of data, measurement has become a strategy to facilitate QR management. This study aims to survey the literature related to QR management through metrics in ARSD, focusing on: bibliometrics, QR metrics, and quality-related indicators used in quality management. The study design includes the definition of research questions, selection criteria, and snowballing as search strategy. We selected 61 primary studies (2001–2019). Despite a large body of knowledge and standards, there is no consensus regarding QR measurement. Terminology is varying as are the measuring models. However, seemingly different measurement models do contain similarities. The industrial relevance of the primary studies shows that practitioners have a need to improve quality measurement. Our collection of measures and data sources can serve as a starting point for practitioners to include quality measurement into their decision-making processes. Researchers could benefit from the identified similarities to start building a common framework for quality measurement. In addition, this could help researchers identify what quality aspects need more focus, e.g., security and usability that have surprisingly few metrics reported.

AI-based systems are software systems with functionalities enabled by at least one AI component (e.g., for image- and speech-recognition, and autonomous driving). AI-based systems are becoming pervasive in society due to advances in AI. However, there is limited synthesized knowledge on Software Engineering (SE) approaches for building, operating, and maintaining AI-based systems. To collect and analyze state-of-the-art knowledge about SE for AI-based systems, we conducted a systematic mapping study. We considered 248 studies published between January 2010 and March 2020. SE for AI-based systems is an emerging research area, where more than 2/3 of the studies have been published since 2018. The most studied properties of AI-based systems are dependability and safety. We identified multiple SE approaches for AI-based systems, which we classified according to the SWEBOK areas. Studies related to software testing and software quality are very prevalent, while areas like software maintenance seem neglected. Data-related issues are the most recurrent challenges. Our results are valuable for: researchers, to quickly understand the state of the art and learn which topics need more research; practitioners, to learn about the approaches and challenges that SE entails for AI-based systems; and, educators, to bridge the gap among SE and AI in their curricula.

Developers work on different tasks in different conditions based on individual technical skills and personal habits. Identifying developer groups by mining their repositories is key for various tasks ranging from understanding developers types in open source projects, to help project managers concerned with the team allocation and coordination of human resources in companies. We aimed at identifying distinct groups of developer profiles based on well defined characteristics and at characterizing the most common quality issue types introduced by each profile in their code. We considered 77,932 commits of 33 open source Java projects, clustering their 2460 developers using dimensionality reduction techniques and applying the k-means algorithm. We identified five profiles among 2460 developers based on project experience, developer productivity and the common quality issues they introduce in the code. Results can be used by developer teams to detect and cope with harmful practices, in order to be more efficient by reducing the number of bugs they produce, looking for adequate training options, and balancing their teams.

Context: Managing Non-Functional Requirements (NFRs) in software projects is challenging, and projects that adopt Model-Driven Development (MDD) are no exception. Although several methods and techniques have been proposed to face this challenge, there is still little evidence on how NFRs are handled in MDD by practitioners. Knowing more about the state of the practice may help researchers to steer their research and practitioners to improve their daily work. Objective: In this paper, we present our findings from an interview-based survey conducted with practitioners working in 18 different companies from 6 European countries. From a practitioner's point of view, the paper shows what barriers and benefits the management of NFRs as part of the MDD process can bring to companies, how NFRs are supported by MDD approaches, and which strategies are followed when (some) types of NFRs are not supported by MDD approaches. Results: Our study shows that practitioners perceive MDD adoption as a complex process with little to no tool support for NFRs, reporting productivity and maintainability as the types of NFRs expected to be supported when MDD is adopted. But in general, companies adapt MDD to deal with NFRs. When NFRs are not supported, the generated code is sometimes changed manually, thus compromising the maintainability of the software developed. However, the interviewed practitioners claim that the benefits of using MDD outweight the extra effort required by these manual adaptations. Conclusion: Overall, the results indicate that it is important for practitioners to handle `NFRs in MDD, but further research is necessary in order to lower the barrier for supporting a broad spectrum of NFRs with MDD. Still, much conceptual and tool implementation work seems to be necessary to lower the barrier of integrating the broad spectrum of NFRs in practice.

Background: Practitioners would like to take action based on software metrics, as long as they find them reliable. Existing literature explores how metrics can be made reliable, but remains unclear if there are other conditions necessary for a metric to be actionable. Context & Method: In the context of a European H2020 Project, we conducted a multiple case study to study metrics' use in four companies, and identified instances where these metrics influenced actions. We used an online questionnaire to enquire about the project participants' views on actionable metrics. Next, we invited one participant from each company to elaborate on the identified metrics' use for taking actions and the questionnaire responses (N=17). Result: We learned that a metric that is practical, contextual, and exhibits high data quality characteristics is actionable. Even a non-actionable metric can be useful, but an actionable metric mostly requires interpretation. However, the more these metrics are simple and reflect the software development context accurately, the less interpretation required to infer actionable information from the metric. Company size and project characteristics can also influence the type of metric that can be actionable. Conclusion: This exploration of industry's views on actionable metrics help characterize actionable metrics in practical terms. This awareness of what characteristics constitute an actionable metric can facilitate their definition and development right from the start of a software metrics program.

As the development progresses, software projects tend to accumulate Technical Debt and become harder to maintain. Multiple tools exist with the mission to help practitioners to better manage Technical Debt. Despite this progress, there is a lack of tools providing actionable and self-learned suggestions to practitioners aimed at mitigating the impact of Technical Debt in real projects. We aim to create a data-driven, lightweight, and self-learning tool positioning highly impactful refactoring proposals on a Jira backlog. Bearing this goal in mind, the first two authors have founded a startup, called Skuld.ai, with the vision of becoming the go-to software renovation company. In this tool paper, we present the software architecture and demonstrate the main functionalities of our tool. It has been showcased to practitioners, receiving positive feedback. Currently, its release to the market is underway thanks to an industry-research institute collaboration with Fraunhofer IESE to incorporate self-learning technical debt capabilities.

Background: Despite the growth in the use of software analytics platforms in industry, little empirical evidence is available about the challenges that practitioners face and the value that these platforms provide. Aim: The goal of this research is to explore the benefits of using a software analytics platform for practitioners managing quality. Method: In a technology transfer project, a software analytics platform was incrementally developed between academic and industrial partners to address their software quality problems. This paper focuses on exploring the value provided by this software analytics platform in two pilot projects. Results: Practitioners emphasized major benefits including the improvement of product quality and process performance and an increased awareness of product readiness. They especially perceived the semi-automated functionality of generating quality requirements by the software analytics platform as the benefit with the highest impact and most novel value for them. Conclusions: Practitioners can benefit from modern software analytics platforms, especially if they have time to adopt such a platform carefully and integrate it into their quality assurance activities.

Existing literature proposes success factors for establishing metrics programs. However, very few studies focus on factors that could ensure long-term use of metrics, and even fewer studies investigate such factors in the context of Agile Software Development (ASD). Motivated by this knowledge gap, we aim to identify success factors for operationalizing metrics in ASD, particularly, factors that could help in the long-term use of metrics. We conducted a multiple case study, where we operationalized process metrics at two software-intensive companies using ASD. We learned that data availability and development process are the two fundamental success factors for process metrics operationalization, albeit less prominent in literature. Companies prefer iterative and incremental operationalization of stable and functional process metrics, which is analogous to the agile way of working. Metrics trustworthiness plays a key role in successful operationalization of process metrics, and is potentially vital to ensuring their long-term use. By comparing the identified success factors with the existing literature, we conclude that success factors concerning data availability, development process, and metrics trustworthiness warrant greater attention, especially to maximize the chances of long-term use of process metrics.

Background: Accessing relevant data on the product, process, and usage perspectives of software as well as integrating and analyzing such data is crucial for getting reliable and timely actionable insights aimed at continuously managing software quality in Rapid Software Development (RSD). In this context, several software analytics tools have been developed in recent years. However, there is a lack of explainable software analytics that software practitioners trust. Aims: We aimed at creating a quality model (called Q-Rapids quality model) for actionable analytics in RSD, implementing it, and evaluating its understandability and relevance. Method: We performed workshops at four companies in order to determine relevant metrics as well as product and process factors. We also elicited how these metrics and factors are used and interpreted by practitioners when making decisions in RSD. We specified the Q-Rapids quality model by comparing and integrating the results of the four workshops. Then we implemented the Q-Rapids tool to support the usage of the Q-Rapids quality model as well as the gathering, integration, and analysis of the required data. Afterwards we installed the Q-Rapids tool in the four companies and performed semi-structured interviews with eight product owners to evaluate the understandability and relevance of the Q-Rapids quality model. Results: The participants of the evaluation perceived the metrics as well as the product and process factors of the Q-Rapids quality model as understandable. Also, they considered the Q-Rapids quality model relevant for identifying product and process deficiencies (e.g., blocking code situations). Conclusions: By means of heterogeneous data sources, the Q-Rapids quality model enables detecting problems that take more time to find manually and adds transparency among the perspectives of system, process, and usage.

Mashup user interfaces provides their functionality through the combination of different services. The integration of such services can be solved by using reusable and third-party components. Furthermore, these interfaces must be adapted to user preferences, context changes, user interactions and c omponent availability. Model transformation is a useful mechanism to address this adaptation but normally these operations only focus on the functional requirements. In this sense, quality attributes should be included in the adaptation process to obtain the best adapted mashup user interface. This paper proposes a generic quality-aware transformation process to support the adaptation of software architectures. The transformation process has been applied in ENIA, a geographic information system, by constructing a specific quality model for the adaptation of mashup user interfaces. This model is taken into account for evaluating the different transformation alternatives and choosing the one that maximizes the quality assessments. The approach has been validated by a set of adaptation scenarios that are intended to maximize different quality factors and therefore apply distinct combinations of metrics.

Software reference architectures provide templates and guidelines for designing systems in a particular domain. Companies use them to achieve interoperability of (parts of) their software, standardization, and faster development. In contrast to system-specific software architectures that "emerge" during development, reference architectures dictate significant parts of the software design early on. Agile software development frameworks (such as Scrum) acknowledge changing software requirements and the need to adapt the software design accordingly. In this paper, we present lessons learned about how reference architectures interact with Scrum (the most frequently used agile process framework). These lessons are based on observing software development projects in five companies. We found that reference architectures can support good practice in Scrum: They provide enough design upfront without too much effort, reduce documentation activities, facilitate knowledge sharing, and contribute to "architectural thinking" of developers. However, reference architectures can impose risks or even threats to the success of Scrum (e.g., to self-organizing and motivated teams).

Context: Several empirical studies investigated the benefits and drawbacks of acquiring a Software Reference Architecture (SRA) to construct a family of software systems with similar architectural needs. However, these empirical results have not been synthesized by any study yet. Such synthesized evidence is essential to make informed decisions whether or not to adopt an SRA in an organization. Goal: To aggregate existing empirically- grounded evidence about the benefits and drawbacks of SRAs, aiming at supporting organizations' decision making on their adoption. Method: To identify primary studies in the technical literature through a systematic literature review, and then, use the Structured Synthesis Method (SSM) to aggregate qualitative and quantitative evidence through the use of diagrammatic models. Results: From the five identified primary studies, five SRA benefits have considerably increased their belief value after aggregation: interoperability of software systems, reduced development costs, improved communication among stakeholders, reduced risk, and reduced time- to-market. Also, one drawback of SRAs has increased its belief value: the required learning curve for developers. Conclusions: The aggregated results consolidate knowledge and confidence on some of the studied SRA effects. The commonly reported effects showed a clear increment of their belief and pointed out to broader generalization. The effects that did not show any belief increment are important to detect areas requiring further evidence to reach a higher degree of consolidation. Practitioners might benefit from these results to support the decision of adopting an SRA in practice.

Context: Software reference architectures (SRA) have emerged as an approach to systematically reuse architectural knowledge and software elements in the development of software systems. Over the last years, research has been conducted to uncover the artifacts that SRAs provide in order to build software systems. However, empirical studies have not focused on providing industrial evidence about such artifacts. Aim: This paper investigates which artifacts constitute an SRA, how SRAs are designed, the potential reuse of SRA's artifacts, and how they are used in practice. Method: The study consists of a case study made in collaboration with a multinational consulting company that designs SRAs for diverse client organizations. A total of nine European client organizations that use an SRA participated in the study. We analyzed available documentation and contacted 28 practitioners. Results: In the nine analyzed projects, we observed that the artifacts that constitute an SRA are mainly software elements, guidelines and documentation. The design and implementation of SRAs are influenced by the reuse of artifacts from previous software system development and experiences, and the reuse of an SRA across different business domains may be possible when they are platform-oriented. Regarding SRAs usage, we observed that conformance checking is seldom performed. Conclusions: This study reports artifacts of SRAs as stated by practitioners in order to help software architects and scientists in the inception, design, and application of SRAs.