Open Source Software (OSS) quality evaluation is essential for ensuring the adoption and effective use of OSS projects across various domains.Most works on OSS quality evaluation have focused on the development of models or frameworks rather than providing practical implementations.These proposals can be challenging to use for inexperienced users, which highlights the need for user-friendly tools.This paper presents preliminary findings from a Systematic Literature Review (SLR) that investigates the characteristics, limitations, and gaps of current Quality Evaluation Tools (QETs) for OSS.Our analysis reveals the diversity of quality models underlying these tools and the absence of standardization, which impedes the comparability and reliability of evaluation results.This work also informs the next steps in improving OSS quality evaluation practices.

ABSTRACT Context With the fast advancement of techniques in artificial intelligence (AI) and of the target infrastructures in the last decades, AI software is becoming an undeniable part of software system projects. As in most cases in history, however, development methods and guides follow the advancements in technology with phase differences. Purpose With an aim to elicit and integrate available evidence from AI software development practices into a process model, this study synthesizes the contributions of the validation studies reported in scientific literature. Method We applied a systematic literature review to retrieve, select, and analyze the primary studies. After a comprehensive and rigorous search and scoping review, we identified 82 studies that make various contributions in relation to AI software development practices. To increase the effectiveness of the synthesis and the usefulness of the outcome, for detailed analysis, we selected 14 primary studies (out of 82) that empirically validated their contributions. Results We carefully reviewed the selected studies that validate proposals on approaches/models, methods/techniques, tasks/phases, lessons learned/best practices, or workflows. We mapped the steps/activities in these proposals with the knowledge areas in SWEBOK, and using the evidence in this mapping and the primary studies, we synthesized a process model that integrates activities, artifacts, and roles for AI‐enabled software system development. Conclusion To the best of our knowledge, this is the first study that proposes such a process model by eliciting and gathering the contributions of the validation studies in a bottom‐up manner. We expect that the output of this synthesis will be input for further research to validate or improve the process model.

ABSTRACT Software engineering plays a critical role in improving the quality of software systems, because identifying and correcting defects is one of the most expensive tasks in software development life cycle. For instance, determining whether a software product still has defects before distributing it is crucial. The customer's confidence in the software product will decline if the defects are discovered after it has been deployed. Machine learning‐based techniques for predicting software defects have lately started to yield encouraging results. The software defect prediction system's prediction results are raised by machine learning models. More accurate models tend to be more complicated, which makes them harder to interpret. As the rationale behind machine learning models' decisions are obscure, it is challenging to employ them in actual production. In this study, we employ five different machine learning models which are random forest (RF), gradient boosting (GB), naive Bayes (NB), multilayer perceptron (MLP), and neural network (NN) to predict software defects and also provide an explainable artificial intelligence (XAI) framework to both locally and globally increase openness throughout the machine learning pipeline. While global explanations identify general trends and feature importance, local explanations provide insights into individual instances, and their combination allows for a holistic understanding of the model. This is accomplished through the utilization of Explainable AI algorithms, which aim to reduce the “black‐boxiness” of ML models by explaining the reasoning behind a prediction. The explanations provide quantifiable information about the characteristics that affect defect prediction. These justifications are produced using six XAI methods, namely, SHAP, anchor, ELI5, LIME, partial dependence plot (PDP), and ProtoDash. We use the KC2 dataset to apply these methods to the software defect prediction (SDP) system, and provide and discuss the results.

ABSTRACT The use of embedded systems has increased significantly over the last decade with the proliferation of Internet of Things technology, automotive and healthcare innovations and the use of smart home appliances and consumer electronics. With this increase, the need for higher quality embedded systems has increased. There are various guidelines and standards, such as ISO/IEC 9126 and ISO/IEC 25010, for product quality evaluation. However, these guidelines cannot be directly applied to embedded systems due to the nature of these systems. Applying traditional quality standards or guidelines on these systems without modification may degrade the performance of the system, increase memory usage or energy consumption, or affect other critical physical metrics adversely. Consequently, several models and approaches have either been introduced or have adopted existing guidelines to produce high‐quality embedded systems. With this motivation, to understand the state of the art, and to identify the research directions in the field, we conducted a systematic literature review (SLR). In our research, we have investigated studies published from 1980 to 2024 and provided a comprehensive review of the scientific literature on quality models, quality attributes, employed practices, and the challenges, gaps, and pitfalls in the field.

Abstract Software architecture plays a fundamental role in overcoming the challenges of the development process of large‐scale and complex software systems. The software architecture of a system is the result of an extensive process in which several stakeholders negotiate issues and solutions, and as a result of this negotiation, a series of architectural decisions are made. This survey study aims to determine the experiences of the software industry experts with respect to architectural decision‐making, the factors that are effective in decision‐making, and the technical and social problems they encounter. An online questionnaire‐based survey was conducted with 101 practitioners. The responses were analyzed qualitatively and quantitatively. Analysis of responses revealed that the majority of the participants prefer to document some or all of the architectural decisions taken and to store these documents in web‐based collaboration software. Decisions are usually made by teams of two or three, and discussion‐based approaches (brainstorming and consensus) are adopted. In the software architecture decision‐making process, “major business impact” is the most challenging situation. Information sharing and keeping track of decisions and decision rationale are areas in need of improvement as identified by most participants. From the participants' feedback and their answers to open‐ended questions, we concluded that the software architecture decision‐making process has an important role in the industry. Our key findings are that decisions made in the architectural decision‐making process are taken by teams and generally all decisions are documented. In projects where decisions are made by a single person, peer pressure is found to be significantly different from pressure in projects where decisions are made by the group. This is an indication that as the number of people in the decision‐making process increases, the disagreements also increase.

Abstract Microservice architecture (MSA) is an architectural style that is designed to support the modular development of software systems within a particular domain. It is characterized by the use of small, independently deployable services, which can be developed and deployed autonomously. The benefits of MSA include improved scalability, fault‐tolerance, and ease of deployment and maintenance. However, developing MSA for a specific domain can be challenging and requires a thorough consideration of various concerns such as service boundaries, communication protocols, and security. To support the easy development and guidance of an MSA for practitioners, we present a reference architecture design for MSA. The reference architecture has been designed after a comprehensive domain analysis of MSA in the literature and the MSAs of key vendors. This analysis has been used to identify best practices and common patterns that can be applied to the development of MSA. Additionally, relevant architecture viewpoints have been selected to model the corresponding architecture views, providing a clear and comprehensive understanding of the architecture. To validate the proposed reference architecture, a multi‐case study research approach has been used. In this approach, two industrial case studies have been used to demonstrate the practical applicability of the proposed reference architecture. The results of these case studies have shown that the reference architecture can be used to effectively guide the development of MSA in real‐world scenarios.

Process mining mainly focuses on discovering control flow models, conformance checking and analyzing bottlenecks. It extends the scope by looking at the other perspectives like time, data and resources by connecting events in the event logs to this process model. These perspectives are not isolated and are all related to each other. For each perspective, there is a different technique, which is dedicated to the relevant perspective, applied and these techniques may need to consume the results of one another in a sequence of process mining analyses. As a result, a holistic process model is created by attaching and binding related attributes of the event logs to the backbone (control flow) of the model. Therefore, representing the holistic model and keeping what is produced from each perspective in a secure and immutable way while applying the multiple perspectives become important. In this study, a BPMN-extended Data Model is proposed to put together the models from the multi-perspective process mining and a tool is developed to keep this data model as an asset into a private blockchain developed by using Hyperledger Fabric. The practical relevance and validity of the approach are shown in the case studies that use real-life data from two different domains.

Abstract As open project repositories have become widespread, evaluating the quality of open source software (OSS) has gained attention in the software community. Despite the variety in quality evaluation models or frameworks (QEMoF) proposed for OSS, their adoption in practice remains limited. To further investigate this argument, a systematic literature review (SLR) has been conducted to characterize the existing QEMoF for OSS, based on particular properties, and to examine comprehensively their content and structure for identifying the gap between theory and practice. In this context, a total of 36 primary studies in the period of 2003–2020 have been analyzed. It has been identified that the majority of QEMoF takes ISO/IEC 9126 as a reference, evaluates community‐based aspects, mostly evaluates maintainability and usability as quality attributes, supports subjective and quantitative evaluations, and faces challenges due to diverse and dynamic nature of OSS products. For a better overview of the state of the art in the field, evaluation criteria for comparing the overall quality of the QEMoF are identified and applied. Finally, overall findings are discussed, and suggestions are provided for practitioners and researchers to formulate more robust studies in the future, which can advance the current situation for the benefit of the software community.

Information Technology (IT) is a complex domain.In order to properly manage IT related processes, several frameworks including ITIL (Information Technologies Infrastructure Library), COBIT (Control OBjectives for Information and related Technologies), IT Service CMMI (IT Service Capability Maturity Model) and many others have emerged in recent decades.Meanwhile, the prevalence of Agile methods has increased, posing the coexistence of Agile approach with different IT frameworks already adopted in organizations.More specifically, the pursuit of being agile in the area of digitalization pushes organizations to go for agile transformation while preserving full compliance to IT frameworks for the sake of their survival.The necessity for this coexistence, however, brings its own challenges and solutions for harmonizing the requirements of both parties.In this paper, we focus on harmonizing the requirements of COBIT and Scrum in a same organization, which is especially challenging when a full compliance to COBIT is expected.Therefore, this study aims to identifying the challenges of and possible solutions for the coexistence of Scrum and COBIT (version 4.1 in this case) in an organization, by considering two case studies: one from the literature and the case of Akbank delivered in this study.Thus, it extends the corresponding previous case study from two points: adds one more case study to enrich the results from the previous case study and provides more opportunity to make generalization by considering two independent cases.

As the number of available Open Source Software (OSS) and the interest they attract are increasing, numerous product attributes are provided to developers and users for evaluating the quality and success of an OSS. Accordingly, various articles in the literature assess the quality and success of OSS, by using different quality attributes and metrics and different approaches. Though this variety can be considered as a positive indicator of research interest and maturation on one side, it creates a kind of jungle in defining and understanding the terms 'quality' and 'success' on the other side. Based on this challenge, in this study, we targeted a systematic mapping (SM) of the articles on quality and success of OSS. More than 474 articles have appeared in this area between the years 2002 and 2017, and the final pool of 128 articles is obtained by defining and applying inclusion and exclusion criteria. SM was employed to develop a classification scheme and categorized the existing body of articles with respect to five research questions (RQs) on: contribution and research types, quality criteria and metrics, success criteria and metrics, the relation of quality and success, and demographics. We observed that the majority of the articles assess the concept of quality as 'code quality', whereas the concept of success is mostly perceived as 'market success' and/or 'developer activity'. Moreover, the metrics of 'contributing developers/users', and the quality attribute of 'functionality' are the quality criteria most employed in the assessment of success.

One of the most important quality indicators of a software product is its defect rates and accordingly, Software Defect Prediction (SDP) has become a widely-accepted practice in both academia and industry to reduce the cost of defects. Despite being reported less frequently, applying SDP in the early phases of software development is even more beneficial in reducing the cost of quality. In this regard, we expect that the existence of a standard procedure or approach for evaluating and selecting a convenient method for building a prediction model in the early phases could make SDP less challenging, especially for the practitioners. Therefore, in this paper, we identify criteria to evaluate SDP methods and select the best-fit one with respect to stakeholder needs which are elicited via a questionnaire in the early phases of the development lifecycle. We consider stakeholder needs in SDP method selection which is based on multi-criteria decision analysis. We investigate the feasibility of our evaluation approach over an open dataset suitable for early SDP, which initially confirm the feasibility of the proposed approach.

Software engineering researchers have, over the years, proposed different critical success factors (CSFs) which are believed to be critically correlated with the success of software projects. To conduct an empirical investigation into the correlation of CSFs with success of software projects, we adapt and extend in this work an existing contingency fit model of CSFs. To archive the above objective, we designed an online survey and gathered CSF-related data for 101 software projects in the Turkish software industry. Among our findings is that the top three CSFs having the most significant associations with project success were: (1) team experience with the software development methodologies, (2) team’s expertise with the task, and (3) project monitoring and controlling. A comprehensive correlation analysis between the CSFs and project success indicates positive associations between the majority of the factors and variables, however, in most of the cases at non-significant levels. By adding to the body of evidence in this field, the results of the study will be useful for a wide audience. Software managers can use the results to prioritize the improvement opportunities in their organizations w.r.t. the discussed CSFs. Software engineers might use the results to improve their skills in different dimensions, and researchers might use the results to prioritize and conduct follow-up in-depth studies on those factors.

Software process assessment (SPA) is the foundation step for software process improvement. ISO/IEC 15504 defines the term process assessment as "the systematic evaluation of an organization's processes against a process reference model (PRM)". In process assessment, there is a need to set and maintain a mapping between an organization's processes and a PRM, where process experts transform the gap between the two into opportunities for process improvement. To maintain such a mapping requires a continuous tracking and alignment between the organization's processes and the PRM(s). The use of ontologies might be a suitable solution to provide computerized tool support for SPA that becomes erroneous and time-consuming if done manually. With an aim to understand the use and usefulness of ontologies in SPA, in this study, we have performed a systematic literature review (SLR). We have searched the most known digital libraries and selected 14 studies out of 54 initially selected and 571 initially retrieved. We analyzed the selected studies with respect to a number of research questions that address; contribution facet, targeted software processes, research facet, process improvement model used, process assessment model used, ontology representation language, purpose of ontology use, qualitative and quantitative benefits reported, and challenges faced. As a result, we synthesized a conceptual model of ontology-based support in SPA. We hope the results of our work will be useful for researchers and practitioners to direct their future studies on the use of ontologies for SPA.

Process mining (PM) as applied in different industrial settings is considered as an effective technique for business process analytics. In healthcare domain, applications of this technique are yet limited but emerging. Process conformance verification, which is applied to measure differences between process model specification and process enactments, is one of the basic reasons to use the PM technique. The differences highlight the gaps to improve the process model specification by using the information gained from the process enactments. This paper first provides an overview of the studies that use the PM technique for conformance verification of healthcare processes as elicited from a systematic mapping study. It then describes the features of a tool that will use the PM technique to carry out process analytics in an intensive care unit of a university hospital.

Automation of functional size measurement (FSM) process has increasingly gained importance since manual measurement is time-consuming, costly, and sometimes errorprone. There exist studies that automate measurement from different software artifacts such as requirements specifications, design models, and software code. In this study we review and compare four studies that we have carried out in recent years to automate COSMIC FSM from software code. Based on the comparison and the lessons learned, we derive an operational scenario for automated FSM from software code and propose a set of requirements that need to be considered in automation. We think the proposal will be helpful not only for our future studies and also for future work of interested researchers.

One of the biggest problems that software organizations encounter is specifying the resources required and the duration of projects. Organizations that record the number of defects and the effort spent on fixing these defects are able to correctly predict the latent defects in the product and the effort required to remove these latent defects. The use of reliability models reported in the literature is typical to achieve this prediction, but the number of studies that report defect prediction models for iterative software development is scarce. In this article we present a case study which predicts the defectiveness of new releases in an iterative, civil project where defect arrival phase data is not recorded. We investigated Linear Regression Model and Rayleigh Model one of the statistical reliability model that contain time information, to predict the module level and project level defectiveness of the new releases of an iterative project through the iterations. The models were created by using 29 successive releases for the project level and 15 successive releases for the module level defect density data. This article explains the procedures that were applied to generate the defectiveness models and the lessons learned from the studies.

Functional Size Measurement (FSM) methods aim to measure the size of software by quantifying its functional user requirements. Although FSM is greatly beneficial in software project management, due to involving judgment on the part of the measurer, differences for the same application may occur. In addition, when an organization wants to build its own historical repository, FSM of the previously developed software might be costly. In this paper, therefore, we examine the possibility of measuring the functional size from source or binary code automatically. Our method utilizes UML Sequence Diagrams derived from functional execution traces of the software at runtime, by which we apply COSMIC FSM rules to measure size. We have aimed to develop a tool called 'Cosmic Solver' to support our method and used its initial prototype for functional size measurement of a sample, three-tier Java business application. This paper explains the basics of the method and the prototype, and the results from its application on the sample. It has been found that the size extracted by the prototype convergent to the one obtained by manual calculation.

The issue of functional size measurement is crucial for software project management, and the instant measurement of functional size from source code might be beneficial for progress tracking. Since it is time-consuming and costly when functional size measurement from source code is done manually, automating the process of measurement came to the fore. In this study, runtime measurement of COSMIC functional size is aimed by the discovery of functional processes, which are triggered via user interface of a three tier Java business application. A Measurement Library has been developed to monitor the data movements occurring in the functional processes. The measurement method which requires code addition into the source code of the application for the utilization of the library is semi-automatic. In a recent study, we reported that the utilization of the library from a simple student registration system led to 92% approximate results in functional sizes measured automatically and calculated manually. Subsequently in this study, three case studies have been carried out to compare the costs of semi-automatic and manual measurements to verify if the method is worth the cost. The results have shown that the method can decrease costs up to %280 compared to the manual measurement process when it is integrated early in the coding phase. This study explains our semi-automatic functional size measurement method, details the implementation of the case studies, and overviews the results.

Defects occur in many software development projects. It is important to extract semantic information from defects to investigate their root causes and improve the process. In this study, enactment data of a software development process in which defects originated was used to support Orthogonal Defect Classification (ODC). In a sample project of a CMMI ML3 organization ODC was applied to the defects and the utilization of the process enactment data was found to be effective and efficient in providing information about the root causes of the defects and deriving improvement actions. The defect attributes were analyzed and compared before and after applying suggested improvement actions. The comparison between the initial and the improved defect trigger and origin distributions showed that there was a positive change in the software development process of the project.

The issue of "Functional Size Measurement" is increasing in importance for software project management. It is considerably time-consuming and costly when functional size measurement is manually done. In this study, automatic measurement of COSMIC functional size for three-tier Java business applications at rum-time is aimed. Functional size measurement is performed by using a self-developed "Measurement Library" for functional processes that are triggered via graphical user interface of the application. The library has been utilized by being imported from a simple student registration system, and the application size has been measured automatically. It has been found that the COSMIC functional sizes measured automatically and calculated manually are 92% approximate.

Background: The number of studies that quantitatively reveal the effect of using Agile models on development performance is scarce. Aim: In this paper, we explain the analysis of a plan-driven. Incremental Process within Siemens EC. carried out to understand its performance as a base for the comparison with the Agile Process. Method: We introduce a method to systematically analyze a software development process to understand its performance quantitatively, and explain the application of the method on the System Test phase of the Incremental Process. Results: The utilization of the analysis method practically contributed to the implementation which was completed in 8 person-days. An analysis of derived measure values indicated that there was no outlier value in the interval of mean ± 3 standard deviation for the measures. Conclusions: We expect that the method will be more beneficial in capturing the context of the Agile Process and its measures.