ABSTRACT The global adoption of artificial intelligence (AI) has increased dramatically in recent years, becoming commonplace in many fields. Such a pervasiveness has led to changes in how AI is perceived, strengthening discussions on its societal consequences. Thus, a new class of requirements for AI‐based solutions emerged. Broadly speaking, those on “explainability” aim to provide a transparent representation of the (often opaque) reasoning method that an AI‐based solution uses when prompted. This work presents a methodology for validating a class of explainable AI (XAI) models, called deterministic rule‐based models, which are used for expressing an explainable approximation of classifiers based on machine learning. The validation methodology combines logical deduction with constraint‐based reasoning in numerical domains, and it either succeeds or returns quantitative estimations of the invalid deviations found. This information allows us to assess the correctness of an XAI model, or in the case of deviations, to evaluate if it still can be deemed acceptable. The validation methodology has been applied to a simulation‐based study where the decision‐making process copes with the spread of SARS‐COV‐2 inside a railway station. The considered case study is a controlled but nontrivial example that shows the overall applicability of the methodology.

Blockchain technology is increasingly adopted in scenarios requiring trust and data integrity. On the Ethereum blockchain, the proxy pattern has become increasingly popular because it allows smart contract code to evolve while preserving stored data. However, a key challenge remains ensuring that such upgrades do not introduce breaking changes or cause disruptions to other contracts and off-chain systems. In this article, we introduce Catana , a framework that leverages historical transactions for Capture-Replay testing of proxy-based Upgradeable Smart Contracts (USCs). Catana assesses the potential impact of an upgrade by comparing the outcomes of replayed transactions with those from the previous version deployed on the main network. Additionally, it extracts and decodes contract state variables, providing deeper insights into how code changes affect the contract state, and helping developers mitigate issues before deployment. Experiments demonstrate that analyzing storage data accounts for the majority (about 86.5%) of detected disruptive upgrades. We also evaluate different policies for building replay test suites from historical transactions. Results identify a strategy that maximizes effectiveness while requiring a small number of replay test executions. Even a test suite containing just one transaction per each invoked method achieved good effectiveness (about 60%) in detecting disruptive upgrades.

Test flakiness is a topmost concern in software test automation. While conducting pre-deployment testing, those tests that are flagged as flaky are put aside for being either repaired or discarded. We hypothesise that some flaky tests could provide useful insights if run in the field, i.e., they could help identify failures that manifest themselves sporadically during In House testing, but are later experienced in operation. We present the first simulation study to investigate the behaviour of flaky tests when moved to the field. The work compares the behaviour of known flaky tests from an open-source library when executed in the development environment vs. when executed in a simulation of the field. Our experimentation over 52 test methods labelled as flaky provides a first confirmation that moving from the development environment to the field, the behaviour of tests changes. In particular, the failure frequency of intermittently failing tests can increase, and we could also identify few cases of field failures that would have been hardly detected during In House testing due to the numerous combinations of inputs and states. In most cases, such flakiness was rooted in the design of the test method itself, however we could also identify an actual bug. The results of our study suggest that the identification of an intermittently failing behaviour could be a valuable hint for a test engineer, and hence flaky tests should not be dismissed right away.

Cross-coverage of a program P refers to the test coverage measured over a different program Q that is functionally equivalent to P. The novel concept of cross-coverage can find useful applications in the test of redundant software. We apply here cross-coverage for test suite augmentation and show that additional test cases generated from the coverage of an equivalent program, referred to as cross tests, can increase the coverage of a program in more effective way than a random baseline. We also observe that -contrary to traditional coverage testing-cross coverage could help finding (artificially created) missing functionality faults.

The emergence of microservices architecture calls for novel methodologies and technological frameworks that support the design, development, and maintenance of applications structured according to this new architectural style. In this paper, we consider the issue of designing suitable strategies for the governance of testing activities within the microservices paradigm. We focus on the problem of discovering implicit relations between test programs that help to avoid re-running all the available test suites each time one of its constituents evolves. We propose a dynamic analysis technique and its supporting framework that collects information about the invocations of local and remote APIs. Information on test program execution is obtained in two ways: instrumenting the test program code or running a symbolic execution engine. The extracted information is processed by a rule-based automated reasoning engine, which infers implicit similarities among test programs. We show that our analysis technique can be used to support the reduction of test suites, and therefore has good application potential in the context of regression test optimisation. The proposed approach has been validated against two real-world microservices applications.

Abstract To enter the production stage, in DevOps practices candidate software releases have to pass quality gates, where they are assessed to meet established target values for key indicators of interest. We believe software reliability should be an important such indicator, as it greatly contributes to the end‐user satisfaction. We propose DevOpRET , an approach for reliability testing as part of the acceptance testing stage in DevOps. DevOpRET relies on operational‐profile–based testing, a common reliability assessment technique. DevOpRET leverages usage and failure data monitored in operations to continuously refine its estimate. We evaluate accuracy and efficiency of DevOpRET through controlled experiments with a real‐world open source platform and with a microservice architectures benchmark. The results show that DevOpRET provides accurate and efficient estimates of the true reliability over subsequent DevOps cycles.

Summary Modern software systems accommodate complex configurations and execution conditions that depend on the environment where the software is run. While in house testing can exercise only a fraction of such execution contexts, in vivo testing can take advantage of the execution state observed in the field to conduct further testing activities. In this paper, we present the Groucho approach to in vivo testing. Groucho can suspend the execution, run some in vivo tests, rollback the side effects introduced by such tests, and eventually resume normal execution. The approach can be transparently applied to the original application, even if only available as compiled code, and it is fully automated. Our empirical studies of the performance overhead introduced by Groucho under various configurations showed that this may be kept to a negligible level by activating in vivo testing with low probability. Our empirical studies about the effectiveness of the approach confirm previous findings on the existence of faults that are unlikely exposed in house and become easy to expose in the field. Moreover, we include the first study to quantify the coverage increase gained when in vivo testing is added to complement in house testing.

Abstract The volume at hand presents the Special Issue on “Distributed Complex Systems: Governance, Engineering, and Maintenance”. The Special Issue has been originally conceived within the context of the 2nd International Workshop on Governing Adaptive and Unplanned Systems of Systems (GAUSS 2020), one of the co‐located events of the 31st International Symposium on Software Reliability Engineering (ISSRE 2020). The authors of the best papers at GAUSS 2020 have been invited to submit an extended version of their previous work. In addition, the editors opened the submission also to all the other researches working on technical and managerial solution for governing Distributed Complex Systems. Ultimate objective of the editors with this Special Issue is to promote discussions focusing on anticipating, mitigating, or reacting to scenarios that were unplanned or under‐specified at design time.

There is an ongoing interest in the Software Engineering field for multivocal literature reviews including grey literature. However, at the same time, the role of the grey literature is still controversial, and the benefits of its inclusion in systematic reviews are object of discussion. Some of these arguments concern the quality assessment methods for grey literature entries, which is often considered a challenging and critical task. On the one hand, apart from a few proposals, there is a lack of an acknowledged methodological support for the inclusion of Software Engineering grey literature in systematic surveys. On the other hand, the unstructured shape of the grey literature contents could lead to bias in the evaluation process impacting on the quality of the surveys. This work leverages an approach on fuzzy Likert scales, and it proposes a methodology for managing the explicit uncertainties emerging during the assessment of entries from the grey literature. The methodology also strengthens the adoption of consensus policies that take into account the individual confidence level expressed for each of the collected scores.

The outcome of test case execution depends on the state of the object under test. While testers can carefully choose meaningful and representative object states for test execution, it is unaffordable to cover the combinatorial space of possible object states exhaustively. An appealing option is to delegate part of the testing activities to the runtime and to execute test cases in the field whenever a new or uncommon state is observed. We have designed and developed Groucho, a framework for in-vivo testing of Java applications. Among the challenges that we faced, the most important ones are isolation of the test session from the user session and minimal performance overhead. Experimental results show that if the activation probability is kept reasonably small (e.g., $10 ^{- {4}}$), the impact of the framework is imperceptible(i.e., either statistically insignificant or with a negligible effect size).

Quality of Experience (QoE) refers to the end users level of satisfaction with a real-time service, in particular in relation to its audio and video quality. Advances in WebRTC technology have favored the spread of multimedia services through use of any browser. Provision of adequate QoE in such services is of paramount importance. The assessment of QoE is costly and can be done only late in the service lifecycle. In this work we propose a simple approach for QoE-driven non-functional testing of WebRTC services that relies on the ElasTest open-source platform for end-to-end testing of large complex systems. We describe the ElasTest platform, the proposed approach and an experimental study. In this study, we compared qualitatively and quantitatively the effort required in the ElasTest supported scenario with respect to a "traditional" solution, showing great savings in terms of effort and time.

Summary European Union (EU) projects are means of the European Commission for funding research activities. Such projects address challenging research objectives by involving both academic and industrial partners, from several countries. Information and communication technologies–related projects often undertake to deliver a software system prototype. In such a context, most of the typical issues of global requirements engineering may emerge. Partners can have different background and expertise, needs are not sharply defined, and communication is hampered by linguistic and cultural differences. If these issues are not carefully taken into account from the beginning, problems frequently emerge during project execution. This paper presents the experience of applying a customized elicitation and refinement approach in the context of the Learn PAd EU project, which involved about 50 people. The approach combines collaborative elicitation and wiki‐based refinement sessions to come to a set of consolidated requirements. Lessons learnt are discussed as a guidance for researchers dealing with analogous issues in similar contexts. Some of the major observations refer to the importance of initial face‐to‐face meetings when combined with asynchronous remote interactions; the role of moderators that have to encourage collaboration and foster a shared understanding; and the definition of guidelines to select wiki‐based platforms.

Model-driven engineering (MDE) promotes automated model transformations along the entire development process. Guaranteeing the quality of early models is essential for a successful application of MDE techniques and related tool-supported model refinements. Do these models properly reflect the requirements elicited from the owners of the problem domain? Ultimately, this question needs to be asked to the domain experts. The problem is that a gap exists between the respective backgrounds of modeling experts and domain experts. MDE developers cannot show a model to the domain experts and simply ask them whether it is correct with respect to the requirements they had in mind. To facilitate their interaction and make such validation more systematic, we propose a methodology and a tool that derive a set of customizable questionnaires expressed in natural language from each model to be validated. Unexpected answers by domain experts help to identify those portions of the models requiring deeper attention. We illustrate the methodology and the current status of the developed tool MOTHIA, which can handle UML Use Case, Class, and Activity diagrams. We assess MOTHIA effectiveness in reducing the gap between domain and modeling experts, and in detecting modeling faults on the European Project CHOReOS.

We present a research roadmap that defines an enhanced model-driven software engineering approach focused on non-functional properties models. Currently, we have implemented two sub-processes of this roadmap: Property Modeling and Monitoring. We provide a property-driven approach to runtime monitoring based on a comprehensive Property Meta- Model (PMM) and on a generic configurable event-based monitoring infrastructure.

Service oriented computing is based on a typical combination of features such as very late binding, run-time integration of software elements owned and managed by third parties, run-time changes. These characteristics generally make difficult both static and dynamic verification capabilities of service-centric systems. In this domain verification and testing research communities have to face new issues and revise existing solutions; possibly profiting of the new opportunities that the new paradigm makes available. In this paper, focusing on service orchestrations, we propose an approach to automatic test case generation aiming in particular at checking the behaviour of services participating in a given orchestration. The approach exploits the availability of a runnable model (the BPEL specification) and uses model checking techniques to derive test cases suitable to detect possible integration problems. The approach has been implemented in a plug-in for the Eclipse platform already released for public usage. In this way BPEL developers can easily derive, using a single environment, test suites for each participant service they would like to compose.

This paper proposes a novel approach to performance testing, called virtual capture and replay (VCR), that couples capture and replay techniques with the checkpointing capabilities provided by the latest virtualization technologies. VCR enables software performance testers to automatically take a snapshot of a running system when certain critical conditions are verified, and to later replay the scenario that led to those conditions. Several in-depth analyses can be separately carried out in the laboratory just by rewinding the captured scenario and replaying it using different probes and analysis tools.