The integration of Large Language Models (LLMs), such as ChatGPT and GitHub Copilot, into software engineering workflows has shown potential to enhance productivity, particularly in software testing. This paper investigates whether LLM support improves defect detection effectiveness during unit testing. Building on prior studies comparing manual and tool-supported testing, we replicated and extended an experiment where participants wrote unit tests for a Java-based system with seeded defects within a time-boxed session, supported by LLMs. Comparing LLM supported and manual testing, results show that LLM support significantly increases the number of unit tests generated, defect detection rates, and overall testing efficiency. These findings highlight the potential of LLMs to improve testing and defect detection outcomes, providing empirical insights into their practical application in software testing.

In recent years, computer science education has increasingly focused on the development and application of automated code assessment methods. Test-based assessments of student submissions generate large quantities of data that could be used for learning analytics. Existing literature highlights a predominant use of unit tests for grading rather than for extracting insights into students' challenges in learning programming concepts. Moreover, competence models, which systematically outline required learning outcomes, are increasingly important during curriculum design. We introduce a novel approach for systematically developing programming assignments aligned with competence models. This method also aids educators in creating unit tests that support competence-based learning analytics. As our method maps competences to individual test cases and thereby quantifies a student's proficiency in an assignment, this helps educators to evaluate, whether students have the ability to understand and master the required competences. Experimental application of our method demonstrated enhanced clarity in understanding student assignments and a considerable improvement in the quality of learning analytics data.

Both physical and software products have a functional use for which they were designed. Putting a new project idea into action necessitates a clear, holistic vision of the expected gain and costs. However, gain (or value) can have many different manifestations that often go far beyond pure functionality. For one thing, it can be about purely monetary value, but it can also be about expanding the company’s reach, improving customer loyalty, displacing competitors, or generating other additional benefits for the company or the user. Cost and Value Engineering is a promising approach for addressing such a value-centered perspective without losing sight of expenses. In typical Software Engineering courses Software Engineering students learn how to deal with requirements in general, as well as user stories, epics, and other requirements artifacts in the context of agile software development projects. However, aside from rudimentary planning activities for user stories or epics, they rarely learn to apply good value prioritization techniques. The primary goal of this paper is to present a validation experiment for two cost and value engineering methods in order to determine whether these methods are appropriate and valuable in a student context. The results are promising, indicating that these methods should be used in software engineering courses on a regular basis. In order for these methodologies to be properly incorporated into Software Engineering courses, an integrated tool support is required.

Abstract When following the principles of value‐based software engineering, business, customer satisfaction, and engineering considerations need to be balanced to develop and operate the software so that it satisfies the different stakeholders' expectations. This, however, requires knowing the relevant quality metrics covering these value‐oriented expectations and potential sources for their measurement. In this work, a categorisation of value‐oriented quality metrics that are practically relevant is presented. Therefore, the authors conducted an online survey with practitioners who assessed the relevance of 61 value‐oriented metrics, gathered from a preceding systematic mapping study. The authors grouped these metrics into 10 categories, based on financial, customer satisfaction, value proposition, and creation perspectives. Also, the authors examined the frequency of particular steps at which these measures accrue and identified their most relevant data sources. The participants rated metrics for feature reliability, performance, as well as test and development efficiency as most relevant for value orientation. According to the participants, the authors' collection covers all relevant metrics for addressing financial and market and feature usability aspects. The authors' categorisation and the metrics' relevance assessments shall support software engineers in selecting relevant metrics and their sources for software product development.

Abstract Agile software development methods have been increasingly adopted by many organizations at different organizational levels. Whether named agile adoption, agile transition, agile transformation, digital transformation or new ways of working, the success of embracing this change process mostly remains uncertain. This is primarily because there are many ways of evaluating success. Based on the existing agile assessment models, we developed a model of principles with associated practice clusters that serves as a core for a new agile assessment model that is capable of assessing agile organizations at different scale. Towards our ultimate goal to establish a lightweight, context-sensitive agile maturity model, we validated our initial findings in an expert interview study to identify improvement points, and ensure the at hand model’s applicability, coherence and relevance. The results of the interview study show that the structure as well as the content of our assessment model fits with the experts’ expectations and experience.

Non-functional requirements of software address a multitude of objectives, expectations, and even liabilities that must be considered during its development and operation. Typi-cally, these non-functional requirements originate from different domains and their concrete scope, notion, and demarcation to functional requirements is often ambiguous. We conducted a systematic mapping study, including 142 selected primary studies, extracted the focus areas, and synthesized the themes and objectives of the described NFRs. In order to also examine non-engineering-focused aspects related to non-functional requirements in DevOps, we added a backward snowballing step and included 17 primary studies. Our analysis revealed 7 recurrent focus areas and 41 themes that characterize NFRs in DevOps, along with typical objectives for these themes. Overall, the focus areas and themes of NFRs in DevOps are very diverse and reflect the different perspectives required to align software engineering with technical quality, business, compliance, and organizational considerations. The lack of methodological support for specifying, measuring, and evaluating fulfillment of these NFRs in DevOps-driven projects offers ample opportunities for future research in this field. Particularly, there is a need for empirically validated approaches for operationalizing non-engineering-focused objectives of software.

Features in a software system usually must satisfy different quality expectations, arising e.g., from their usage context or the long-term strategy of the manufacturer. As an example, the maintainability of the source code will likely be more important if the associated feature is frequently used by customers or if it has strategic value for the software manufacturer. Accordingly, features that process user-generated near real-time data will likely impose higher requirements towards performance efficiency than other features of the same application for maintaining the user profile. In order to practically approach these qualitative subtleties particularly in a DevOps context, we need an operational means to specify and automatically evaluate the fulfillment of these feature-dependent non-functional requirements, e.g., through quantitative constraints. However, the multitude of systems involved in DevOps and the heterogeneous data types of measures accruing on these systems hinder their effortless acquisition and automated evaluation.In this paper we present an operational constraint language for specifying and evaluating feature-dependent non-functional requirements quantitatively. Our language provides a compact set of time series operations, time filters, and comparison operators and allows to define metrical and ordinal threshold values. A comprehensive evaluation based on a large-scale software project with measures spanning the period over one year shows the performance and suitability of the approach for evaluating feature- dependent non-functional requirements specially in DevOps.

Agile software development is considered as a game changer by some [1]. There are others who approach the domain more skeptical [2]. Clearly, there is a gap in terms of how Agile is perceived. This gap could be reduced, if proper measures would have been timely employed. Agile maturity assessment models have been helpful in providing means and guidance for reducing this gap. Yet, a detailed look into existing models, comparative studies and literature reviews in the domain suggests that the agile maturity assessment models themselves are far from being mature. Further, the gap between what the proposed models offer and what industry really needs is frequently discussed by those comparative studies. In this study, we reflect on the existing agile maturity assessment models, compare them against a comprehensive set of criteria derived from the literature and from our background in assessing the capabilities of software development organizations. We conclude that none of the analyzed agile maturity assessment models are sound enough to be used in a practical context. Nevertheless, some of the models have interesting elements that can be reused for the development of a new agile assessment model.

Features in a software system usually must satisfy different quality expectations originating from the various stakeholders of a software system, ranging from direct users to the software manufacturer. As an example, the design quality of the source code will likely be more important if the associated feature is frequently used by customers or if it has strategic value for the software manufacturer. In order to effectively approach these qualitative subtleties, we need a means to specify non-functional requirements on the level of individual software features. Fine-grained specification of non-functional requirements on feature level respects their individual relevance for a feature and facilitates deriving suitable constraints for them. Particularly in DevOps-driven software projects with operational data the fulfillment of non-functional requirements for individual features can then continuously be evaluated. This guides software engineers in meeting these quality expectations on a much finer level than if treated uniformly across the software system. In this paper we present an extension of the QUAMOCO meta quality model for specifying and evaluating non-functional requirements on feature level. While the existing meta model focused on static quality measures, the extension of the meta model now also captures dynamic measures which accumulate during feature execution. An exploratory case study based on the results from an interview study shows the completeness of the approach for specifying and evaluating feature-dependent non-functional requirements.

Technical debt (TD) in a software system is a metaphor that tries to illustrate the remediation effort of the already introduced quality deficit and the impact thereof to the business value of the system. To address TD, various management activities are proposed, each addressing a particular purpose. Whereas the activities of debt identification and measurement are broadly considered in literature, the activities of debt prioritization and communication lack appropriate approaches with an economic perspective. This work proposes a TD prioritization approach. Therefore, it narrows down the focus of TD to design debt and relies on the quantification of design best practices. Further, the non-conformance of these practices is assessed by applying a benchmarking technique. As a result, the gained information is transferred into a portfolio-matrix to support the prioritization and communication of design remediation actions. The applicability and suitability of the approach are demonstrated by using the source code of the open source project GeoGebra.

The design debt metaphor tries to illustrate quality deficits in the design of a software and the impact thereof to the business value of the system. To pay off the debt, the literature offers various approaches for identifying and prioritizing these design flaws, but without proper support in aligning strategic improvement actions to the identified issues. This work addresses this challenge and examines the suitability of our proposed portfolio-based design assessment approach. Therefore, this investigation is conducted based on three case studies where the product source code was analyzed and assessed using our portfolio-based approach. As a result, the approach has proven to be able to recommend concrete and valuable design improvement actions that can be adapted to project constraints.

To measure object-oriented design quality, metric-based approaches have been established. These have then been enhanced by identifying design smells in code. While these approaches are useful for identifying hot spots that should be refactored, they are still too vague to sufficiently guide software developers to implement improvements. This is why our previous work focuses on measuring the compliance of source code with object-oriented design best practices. These design best practices were systematically derived from the literature and can be mapped to design principles, which can help reveal fundamental object-oriented design issues in a software product. Despite the successful applications of this approach in industrial and open source projects, there is little accepted knowledge about the importance of various design best practices. Consequently, this paper shows the result of an online survey aimed at identifying the importance of 49 design best practices on design quality. In total, 214 people participated in the survey, resulting in an average of 138 opinions for each practice. Based on these opinions, five very important, 21 important, 12 moderately important and 11 unimportant design best practices could be derived. This information about importance helps managing design improvements in a focused way.

Current software development trends have shortened release cycles and forced developers to implement short-term solutions that cannot cope with increasing product complexity. This phenomenon of introducing hasty design choices or applying bad design practices becomes something known as technical debt, in particular design debt. To pay off this debt, the literature offers approaches for identifying these design flaws; however, few methods for properly prioritizing investment efforts are available. In this paper, we propose an approach that supports the decision-making process regarding design improvements. It identifies violations of design best practices that are then arranged within a two-dimensional portfolio matrix. This matrix combines the importance of practices of design quality with actual achievement relative to a benchmark suite. To show the application of the approach in a quality-improvement process, we performed a feasibility study on three open-source projects and a benchmark suite containing 50 projects. This study clearly shows that the importance of the design best practices greatly impacts the improvement decisions and must be aligned with the strategic quality goals of the product.

The measurement of software quality, including the preparation and management of the necessary resources and libraries, is a major challenge in continuous software quality measurement and assessment. When applying code analysis tools to a large number of projects, the preparation of the source code and its dependencies, focusing on the completeness of these elements, is the basis for correct analysis results. In order to make this preparation process efficient and effective, there is a need to automate this process. Therefore, we built a tool infrastructure, which automates this preparation and analysis process. As part of the code preparation process, we developed the tool LibLoader, which automatically resolves missing dependencies in open source Java projects. This enables the analysis of complete projects in due time and with more accurate results from static code analysis tools.

IT-CISQ (Consortium for IT Software Quality) standard that was initiated and is driven by an OMG (Object Management Group) consortium claims to provide operational measures for a number of quality criteria of the ISO / IEC 25010 quality standard. In order to show the applicability of the IT-CISQ standard, we built a tool that provides automatic measures for the maintainability related measures of the IT-CISQ standard. We evaluated five Java open source projects with our IT-CISQ quality model and compared these results with the result of an expert-centered evaluation of the maintainability of these projects. The study shows that the IT-CISQ quality model for maintainability is well suited as a predictor for the quality criterion maintainability. Compared to our more comprehensive EMISQ (Expert Centered Method for Internal Software Quality) quality model (165 EMISQ measures vs. 18 IT-CISQ measures), the IT-CISQ quality model yields a similar order of projects. However, it is of course less suitable to give good guidance about improving aspects, as our more comprehensive EMISQ quality model includes considerably more maintainability related aspects of software. Furthermore, the tooling provided by us is innovative and could directly be used in practice by itself.

Industrial engineering is inherently of mechatronic character. The integration of all involved disciplines across the entire engineering life cycle is considered as a key element for improvements but requires appropriate conceptual support. Mechatronic objects provide such an approach. A criteria catalog for an assessment-based qualitative evaluation of respective engineering artifacts was developed, which aims to support characterizing, comparing, and improving mechatronic objects. In addition, it allows identifying directions for engineering improvements. In order to align future research efforts with industry needs, we conducted a survey among practitioners and researchers on their estimations regarding the potential of the mechatronic object concept and the implementation of selected criteria. This paper provides the results and insights from this survey and suggests a roadmap in order to improve mechatronic engineering.

Published software quality models either provide abstract quality attributes or concrete quality assessments. There are no models that seamlessly integrate both aspects. In the project Quamoco, we built a comprehensive approach with the aim to close this gap. For this, we developed in several iterations a meta quality model specifying general concepts, a quality base model covering the most important quality factors and a quality assessment approach. The meta model introduces the new concept of a product factor, which bridges the gap between concrete measurements and abstract quality aspects. Product factors have measures and instruments to operationalise quality by measurements from manual inspection and tool analysis. The base model uses the ISO 25010 quality attributes, which we refine by 200 factors and 600 measures for Java and C# systems. We found in several empirical validations that the assessment results fit to the expectations of experts for the corresponding systems. The empirical analyses also showed that several of the correlations are statistically significant and that the maintainability part of the base model has the highest correlation, which fits to the fact that this part is the most comprehensive. Although we still see room for extending and improving the base model, it shows a high correspondence with expert opinions and hence is able to form the basis for repeatable and understandable quality assessments in practice.

Existing software quality models typically focus on common quality characteristics such as the ISO 25010 software quality characteristics. However, most of them provide insufficient operationalization for quality assessments of source code. Moreover, they usually focus on software in general or on information systems and do not sufficiently cover the particularities of embedded systems. We have developed a quality model that covers quality requirements for source code that are specific for embedded systems software. It provides comprehensive operationalization (with 336 measures) for C and C++ systems, which allows for largely automated quality assessments. The empirical evaluations performed acknowledge moderate completeness of the requirements and the associated measures. Therefore, we still see room for improvements to allow covering even more aspects of embedded systems software quality. Nevertheless, the empirical validation (based on three industrial products) shows good concordance between the results gained by the automatic model-based assessment and independent expert judgment on code quality.

Industrial engineering as a specialization of systems engineering inherently has mechatronic character. The seamless integration of engineering activities across engineering disciplines and throughout the overall engineering life cycle is a key element for achieving optimal technical as well as economic results and has to be supported by adequate engineering concepts and models. In this paper we present a criteria catalog for the qualitative evaluation of engineering artifacts created in or for engineering projects or contained in mechatronic product catalogs. The criteria catalog forms the basis for an assessment-based approach for evaluating mechatronic objects with the main purposes of providing the basis for characterizing, comparing, and improving them and furthermore for identifying directions for engineering improvements.

Industrial engineering can be regarded as a specialization of systems engineering. It deals with the engineering of industrial solutions, e.g. power plants or airport logistics systems. In today's competitive environment, the industrial solutions business faces high cost and productivity pressure, increasing complexity, and highly demanding customers. Transforming this business from the project-focused development of highly customer-specific solutions to the customization of a 'solution product' and fostering reuse in general are judged highly rewarding approaches to tackle these challenges. As a consequence, an ISO/IEC 15504 conformant process reference model for the industrial solutions business has been proposed, based on the integration of a process reference model for reuse in industrial engineering and a product-oriented process model for the engineering of industrial solutions. In this paper we present the results of analyzing this reference model against the international standard ISO/IEC 15288 on systems life cycle processes. The results of the mapping of these models are analyzed to identify further enhancements of the reference model for the industrial solutions business, and for identifying potential directions for the further evolution of the international standard.

Abstract Improving reuse in industrial engineering is more and more recognized as a key to economic success by providers of industrial solutions. It generally increases the quality of the engineered systems, shortens engineering time, and decreases engineering costs for the development of customer‐specific industrial solutions. We therefore developed an integrated set of methods for assessing an organization's reuse practices, identifying its reuse potential and guiding the selection and implementation of improvement actions. While the assessment of reuse practices is performed according to the principles of ISO/IEC 15504 for process assessment, identification of improvements for reuse in industrial engineering is much more complex compared to ‘traditional’ process capability‐oriented improvement. Orthogonal to improving along the process capability dimension it also involves strategic decisions on the overall design of the engineering process, the pursued engineering and reuse paradigms, the desired organizational reuse maturity stages, etc. To tackle this problem, we developed a potentials analysis method that assures compliance of the improvements with the strategic goals of the organization. It bridges the gap between the process assessment and the improvement action planning parts of the methodology by supporting the selection of reuse practices considering strategic issues and business goals. Copyright © 2009 John Wiley & Sons, Ltd.

There is empirical evidence that the code quality of software has an important impact on the external, i.e., user perceptible, software quality. Currently a large number of source code metrics exist that seem to ease the evaluation of code quality. Nevertheless, studies show that the theoretical foundations are weak and promising approaches for the automatic assessment of code quality are to be considered with great caution. We therefore came to the conclusion that the metric values and other findings provided by various static code analysis tools can only be used in the context of an expert-centred assessment of internal software quality. In order to be able to carry out code quality assessments in a timely and efficient manner it is inevitable to have additional tool support. For that purpose we developed the eclipsed based tool software product quality reporter (SPQR) that supports expert-centred evaluation of source code - from the formulation of project-specific quality models up to the generation of preliminary code quality reports. The application of SPQR already proved its usefulness in various code assessment projects around the world.

Only a few studies exist that try to investigate whether there is a significant correlation between external software quality and the data provided by static code analysis tools. A clarification on this issue could pave the way for more precise prediction models on the probability of defects based on the violation of programming rules. We therefore initiated a study where the defect data of selected versions of the open source development environment ldquoEclipse SDKrdquo is correlated with the data provided by the static code analysis tools PMD and FindBugs applied the source code of Eclipse. The results from this study are promising as especially some PMD rulesets show a good correlation with the defect data and could therefore serve as basis for measurement, control and prediction of software quality.

The idea of design by contract (DEC), realized in the statically typed object-oriented programming language Eiffel, can be viewed as a systematic approach to specifying and implementing object-oriented software systems. We believe that a statically typed programming language is not suitable in the analysis and design phase of a prototyping-oriented software life cycle. For this purpose, dynamically typed interpreted programming languages are better suited. Unfortunately, dynamically typed programming languages usually do not support the concept of DEC. Therefore we integrated DEC into the programming language Python by using a metaprogramming approach, i.e., without changing the language or the run-time system. We adopted the DEC concept by adding mechanisms for dynamic type checking for method parameters and instance variables. The proposed combination of a more formal approach with a slim programming language provides a good basis for elicitation and documentation tasks in the analysis and design phase, especially in cases of a prototyping-oriented software development approach. Although the approach presented provides basic tool support for the analysis and design phase, further tool support especially for browsing assertions, is desirable.