Abstract When following the principles of value‐based software engineering, business, customer satisfaction, and engineering considerations need to be balanced to develop and operate the software so that it satisfies the different stakeholders' expectations. This, however, requires knowing the relevant quality metrics covering these value‐oriented expectations and potential sources for their measurement. In this work, a categorisation of value‐oriented quality metrics that are practically relevant is presented. Therefore, the authors conducted an online survey with practitioners who assessed the relevance of 61 value‐oriented metrics, gathered from a preceding systematic mapping study. The authors grouped these metrics into 10 categories, based on financial, customer satisfaction, value proposition, and creation perspectives. Also, the authors examined the frequency of particular steps at which these measures accrue and identified their most relevant data sources. The participants rated metrics for feature reliability, performance, as well as test and development efficiency as most relevant for value orientation. According to the participants, the authors' collection covers all relevant metrics for addressing financial and market and feature usability aspects. The authors' categorisation and the metrics' relevance assessments shall support software engineers in selecting relevant metrics and their sources for software product development.

Non-functional requirements of software address a multitude of objectives, expectations, and even liabilities that must be considered during its development and operation. Typi-cally, these non-functional requirements originate from different domains and their concrete scope, notion, and demarcation to functional requirements is often ambiguous. We conducted a systematic mapping study, including 142 selected primary studies, extracted the focus areas, and synthesized the themes and objectives of the described NFRs. In order to also examine non-engineering-focused aspects related to non-functional requirements in DevOps, we added a backward snowballing step and included 17 primary studies. Our analysis revealed 7 recurrent focus areas and 41 themes that characterize NFRs in DevOps, along with typical objectives for these themes. Overall, the focus areas and themes of NFRs in DevOps are very diverse and reflect the different perspectives required to align software engineering with technical quality, business, compliance, and organizational considerations. The lack of methodological support for specifying, measuring, and evaluating fulfillment of these NFRs in DevOps-driven projects offers ample opportunities for future research in this field. Particularly, there is a need for empirically validated approaches for operationalizing non-engineering-focused objectives of software.

Features in a software system usually must satisfy different quality expectations, arising e.g., from their usage context or the long-term strategy of the manufacturer. As an example, the maintainability of the source code will likely be more important if the associated feature is frequently used by customers or if it has strategic value for the software manufacturer. Accordingly, features that process user-generated near real-time data will likely impose higher requirements towards performance efficiency than other features of the same application for maintaining the user profile. In order to practically approach these qualitative subtleties particularly in a DevOps context, we need an operational means to specify and automatically evaluate the fulfillment of these feature-dependent non-functional requirements, e.g., through quantitative constraints. However, the multitude of systems involved in DevOps and the heterogeneous data types of measures accruing on these systems hinder their effortless acquisition and automated evaluation.In this paper we present an operational constraint language for specifying and evaluating feature-dependent non-functional requirements quantitatively. Our language provides a compact set of time series operations, time filters, and comparison operators and allows to define metrical and ordinal threshold values. A comprehensive evaluation based on a large-scale software project with measures spanning the period over one year shows the performance and suitability of the approach for evaluating feature- dependent non-functional requirements specially in DevOps.

Features in a software system usually must satisfy different quality expectations originating from the various stakeholders of a software system, ranging from direct users to the software manufacturer. As an example, the design quality of the source code will likely be more important if the associated feature is frequently used by customers or if it has strategic value for the software manufacturer. In order to effectively approach these qualitative subtleties, we need a means to specify non-functional requirements on the level of individual software features. Fine-grained specification of non-functional requirements on feature level respects their individual relevance for a feature and facilitates deriving suitable constraints for them. Particularly in DevOps-driven software projects with operational data the fulfillment of non-functional requirements for individual features can then continuously be evaluated. This guides software engineers in meeting these quality expectations on a much finer level than if treated uniformly across the software system. In this paper we present an extension of the QUAMOCO meta quality model for specifying and evaluating non-functional requirements on feature level. While the existing meta model focused on static quality measures, the extension of the meta model now also captures dynamic measures which accumulate during feature execution. An exploratory case study based on the results from an interview study shows the completeness of the approach for specifying and evaluating feature-dependent non-functional requirements.

While formal task models allow definition and time-based assessment of user-interaction, they have not yet been used as a baseline for assessing the variability patterns of user interaction with software. Consequently, operationalizing these variability patterns enables us to evaluate how suitable the defined task-execution paths are for the user to achieve a predefined goal. Improvements of the software could thereby be evidence-based on knowledge of changes' effects on functional suitability for the user rather than on prospective or opinion-based usage scenarios. Following a design thinking approach tailored to software engineering, understanding and observing these variability patterns are mandatory steps for continuous user-centered improvement of software. In practice however, due to the absence of an operational quality model for functional suitability it is hardly possible to effectively put these operational measures into context and to derive concrete software improvement actions. Having operational suitability metrics is even more important for software engineers as it allows to increasingly focus development activities especially on functionality which has business value for the user.