As REST APIs become an increasingly significant part of software systems, their validation is becoming more critical. Hence, testing and uncovering underlying issues are of utmost importance for improving software quality. However, testing REST APIs is challenging mainly due to the difficulty of assessing whether the output of an API call is correct, i.e., the test oracle problem. Metamorphic testing is a specification-based testing approach for situations where correct outputs are unknown or not specified explicitly. To check the correctness of a system, relations between the different outputs are specified. We present ARMeta, a tool-supported approach that uses an LLM-based multi-agent workflow to support metamorphic testing of REST APIs documented with OpenAPI. The agentic workflow is used to identify metamorphic test scenarios and specify them in the Given-When-Then format. These scenarios are automatically implemented as executable tests and executed against the system under test. We evaluate ARMeta on two publicly available web applications that expose REST interfaces and compare its performance with a scenario-based testing baseline. The results show that ARMeta explores behaviors that serve as a complement to existing scenario-based testing approaches.

In many industrial domains, the Functional Mock-up Interface (FMI) is used to exchange simulation models as Functional Mock-up Units (FMUs) across different partners using various modelling tools. This opens up the possibilities for simulation-based verification and validation using FMUs for ensuring reliable system behaviour. However, deriving effective test oracles for these simulation models remains challenging due to the absence of explicit expected outputs. This limits the applicability of conventional testing approaches, which require access to the internal workings of the systems. Metamorphic testing (MT) addresses this limitation by leveraging metamorphic relations (MRs), but extracting such relations from specifications remains largely a manual and error-prone process. To address this challenge, we propose an LLM-powered multi-agent workflow for specification-based metamorphic testing of FMU-based simulation models. The approach takes functional and interface specifications as input and orchestrates multiple agents to extract requirements and derive MRs. These MRs are expressed using Given-When-Then patterns to structure input conditions (Given), transformations (When), and expected output behaviours (Then). These relations are then used to generate metamorphic test cases, execute simulations, and evaluate output consistency across multiple sessions. We evaluate the approach on a Lube Oil Cooling system FMU, demonstrating its ability to automatically generate meaningful MRs and corresponding test cases. Preliminary results indicate that the proposed workflow can effectively support the systematic verification and validation of dynamic simulation models by reducing manual effort and improving test generation.

Graphical user interface (GUI) tests are used to validate end-to-end business flows. In microservice-based systems, a single user action may trigger a chain of inter-service calls, which require verification. In practice, end-to-end test suite design is largely manual and hard to maintain. We propose a multi-agent workflow that takes as input use cases, microservice interface specification, architecture documentation, and a GUI description, and produces executable Selenium tests with test oracles. We evaluate the approach on open-source microservice systems by comparing the generated suites with manually created benchmarks. During test execution, we collect execution traces to evaluate the coverage of microservice endpoints. In addition, we evaluate the fault detection capability of the generated test suites by seeding faults into the systems under test.

VeriDevOps offers a methodology and a set of integrated mechanisms that significantly improve automation in DevOps to protect systems at operations time and prevent security issues at development time by (1) specifying security requirements, (2) generating trace monitors, (3) locating root causes of vulnerabilities, and (4) identifying security flaws in code and designs. This paper presents a methodology that enhances productivity and enables the continuous integration/delivery of trustworthy systems. We outline the methodology, its application to relevant scenarios, and offer recommendations for engineers and managers adopting the VeriDevOps approach. Practitioners applying the VeriDevOps methodology should include security modeling in the DevOps process, integrate security verification throughout all stages, utilize automated test generation tools for security requirements, and implement a comprehensive security monitoring system, with regular review and update procedures to maintain relevance and effectiveness.

With the increasing popularity of the cloud continuum, the security of different layers and nodes involved has become more relevant than ever. Intrusion detection systems, are one of the main tools to identify and intercept intrusion attacks. Furthermore, identifying the attacks in time, before they are completed, is necessary in order to deploy countermeasures in time and to limit the losses. In this work, we evaluate the use of transformer models for implementing early-detection signature-based detection systems targeted at Cloud Continuum. We implement the approach in the context of our tool for early detection of network intrusions and we evaluate it using the CICIDS2017 dataset and MQTT-IDS-2020. The results show that transformer models are a viable alternative for early-detection systems and this will pave the road for further research on the topic.

We elaborate on a metamorphic approach for testing industrial control systems. The proposed approach consists of two phases: an exploration phase in which we learn about fault patterns of the system under test and an exploitation phase where the observed fault patterns are used for targeted testing. Our method extracts metamorphic relations and input space of the system from its requirements. The seed input used for testing is extracted from the execution logs of the system and used to generate source tests and follow-up tests automatically. The morphed input is constructed based on the seed input and refined using a set of constraints. The approach is exemplified on a position control system and the results show that it is effective in discovering faults with an increased level of automation.

Current software development practices are increasingly based on using both COTS and legacy components which make such systems prone to security vulnerabilities. The modern practice addressing ever changing conditions, DevOps, promotes frequent software deliveries, however, verification methods artifacts should be updated in a timely fashion to cope with the pace of the process. VeriDevOps, Horizon 2020 project, aims at providing a faster feedback loop for verifying the security requirements and other quality attributes of large scale cyber-physical systems. VeriDevOps focuses on optimizing the security verification activities, by automatically creating verifiable models directly from security requirements formulated in natural language, using these models to check security properties on design models and then generating artefacts such as, tests or monitors that can be used later in the DevOps process. The main drivers for these advances are: Natural Language Processing, a combined formal verification and model-based testing approach, and machine-learning-based security monitors. VeriDevOps is in its initial stage - the project started on 1.10.2020 and it will run for three years. In this paper we will present the major conceptual ideas behind the project approach as well as the organizational settings.

In this paper, we report on our 5-year’s practical experience of designing, developing and then deploying a Model-based Requirements Engineering (MBRE) approach and language in the context of three different large European collaborative projects providing complex software solutions. Based on data collected both during projects execution and via a survey realized afterwards, we intend to show that such an approach can bring interesting benefits in terms of scalability (e.g., large number of handled requirements), heterogeneity (e.g., partners with different types of RE background), traceability (e.g. from the requirements to the software components), automation (e.g., requirement documentation generation), usefulness or usability. To illustrate our contribution, we exemplify the application of our MBRE approach and language with concrete elements coming from one of these European research projects. We also discuss further the general benefits and current limitations of using this MBRE approach and corresponding language.

In this paper, we present the MegaM@Rt2 ECSEL project and discuss in details our approach for fostering collaboration in this project. We choose to use an internal hackathon approach that focuses on technical collaboration between case study owners and tool/method providers. The novelty of the approach is that we organize the technical workshop at our regular project progress meetings as a challenge-based contest involving all partners in the project. Case study partners submit their challenges related to the project goals and their use cases in advance. These challenges are concise enough to be experimented within approximately 4 hours. Teams are then formed to address those challenges. The teams include tool/method providers, case study owners and researchers/developers from other consortium members. On the hackathon day, partners work together to come with results addressing the challenges that are both interesting to encourage collaboration and convincing to continue further deeper investigations. Obtained results demonstrate that the hackathon approach stimulated knowledge exchanges among project partners and triggered new collaborations, notably between tool providers and use case owners.

Performance bottlenecks resulting in high response times and low throughput of software systems can ruin the reputation of the companies that rely on them. Almost two-thirds of performance bottlenecks are triggered on specific input values. However, finding the input values for performance test cases that can identify performance bottlenecks in a large-scale complex system within a reasonable amount of time is a cumbersome, cost-intensive, and time-consuming task. The reason is that there can be numerous combinations of test input values to explore in a limited amount of time. This paper presents PerfXRL, a novel approach for finding those combinations of input values that can reveal performance bottlenecks in the system under test. Our approach uses reinforcement learning to explore a large input space comprising combinations of input values and to learn to focus on those areas of the input space which trigger performance bottlenecks. The experimental results show that PerfxRL can detect 72% more performance bottlenecks than random testing by only exploring the 25% of the input space.

The Foundational Subset for Executable UML Models (fUML) and the Action language for fUML (Alf) can be used for creating executable models in the Eclipse-based UML editing tool called Papyrus. An fUML execution engine in Papyrus, such as Moka, allows to simulate or execute fUML models along with their associated Alf code. However, for exhaustive simulation of such models, one must provide input data required to reach and cover all important elements not only in the graphical fUML models, but also in the textual Alf code. In this paper, we present MATERA2-AlfTester, an Eclipse-plugin for exhaustive simulation and test generation for fUML models. MATERA2-AlfTester integrates with Papyrus and Moka tools and extends their functionally by allowing one to automatically generate test data, test suite with test oracle, and partial Java code at design time. We also present the simulation and testing process of MATERA2-AlfTester with the help of an example and demonstrate how exhaustive simulation and test generation with MATERA2-AlfTester can help designers in assessing and improving the quality of fUML models.

We present a model-based mutation testing approach, for evaluating the authentication and authorization of web services in a multi-user context. Model of a web service and its security requirements are designed using UPPAAL Timed Automata. The model is mutated to create invalid behavior which is used for test generation to reveal faults in the system under test. The approach is supported by a model-based mutation testing tool, µUTA, that automatically generates mutants, selects a collection of suitable mutants for testing and generates test cases from them. We modify a previously defined mutation operator and introduce three new operators for additional mutants. We define criteria for the mutation-selection and demonstrate the approach on a blog web service. Results show that the approach can discover authorization faults that were not detected by traditional methods.

The poor performance of web-based systems can negatively impact the profitability and reputation of the companies that rely on them. Finding those user scenarios which can significantly degrade the performance of a web application is very important in order to take necessary countermeasures, for instance, allocating additional resources. Furthermore, one would like to understand how the system under test performs under increased workload triggered by the worst-case user scenarios. In our previous work, we have formalized the expected behavior of the users of web applications by using probabilistic workload models and we have shown how to use such models to generate load against the system under test. As an extension, in this article, we suggest a performance space exploration approach for inferring the worst-case user scenario in a given workload model which has the potential to create the highest resource utilization on the system under test with respect to a given resource. We propose two alternative methods: one which identifies the exact worst-case user scenario of the given workload model, but it does not scale up for models with a large number of loops, and one which provides an approximate solution which, in turn, is more suitable for models with a large number of loops. We conduct several experiments to show that the identified user scenarios do provide in practice an increased resource utilization on the system under test when compared to the original models.

A major challenge for the European electronic industry is to enhance productivity while reducing costs and ensuring quality in development, integration and maintenance. Model-Driven Engineering (MDE) principles and techniques have already shown promising capabilities but still need to scale to support real-world scenarios implied by the full deployment and use of complex electronic components and systems. Moreover, maintaining efficient traceability, integration and communication between two fundamental system life-time phases (design time and runtime) is another challenge facing scalability of MDE. This paper presents an overview of the ECSEL project entitled "MegaModelling at runtime -- Scalable model-based framework for continuous development and runtime validation of complex systems" (MegaM@Rt2), whose aim is to address the above mentioned challenges facing MDE. Driven by both large and small industrial enterprises, with the support of research partners and technology providers, MegaM@Rt2 aims to deliver a framework of tools and methods for: 1) system engineering/design & continuous development, 2) related runtime analysis and 3) global model & traceability management, respectively. The diverse industrial use cases (covering domains such as aeronautics, railway, construction and telecommunications) will integrate and apply such a framework that shall demonstrate the validation of the MegaM@Rt2 solution.

In this paper, we describe a systematic literature review (SLR) on the use of environment models in model-based testing (MBT). By applying selection criteria, we narrowed down the identified studies from two hundred ninety seven papers to sixty one papers which are used in this analysis. The results show that environment models are especially useful in testing systems with high complexity and nondeterministic behaviors in terms of facilitating automatic test generation. However, building environment models is not a trivial task due to the lack of a systematic methodology and of supporting tools for automation.

We present an integrated approach to design and validate RESTful composite web services. We use the Unified Modeling Language (UML) to specify the requirements, behavior and published resources of each web service. In our approach, a service can invoke other services and exhibit complex and timed behavior while still complying with the REST architectural style. We show how to transform service specifications into UPPAAL timed automata for verification and test generation. The service requirements are propagated to the UPPAAL timed automata during the transformation. Their reachability is verified in UPPAAL and they are used for computing coverage level during test generation. We validate our approach with a case study of a holiday booking web service.

In this paper, we present an approach for performance testing of software services. In our approach, we use Probabilistic Timed Automata (PTA) to model the workload of the system, by describing how different user types interact with the system. We use these models to generate load in real-time and we measure different performance indicators.

We discuss a design methodology for SegBus, a multicore segmented bus platform. The methodology supports the modeling of the platform at several abstraction levels, enabling the designer to focus only on the relevant aspects of the architecture at a given development stage. We employ the unified modeling language (UML) as a specification language for theSegBusplatform and we customize its elements to serve our specific purposes via the profiling mechanism. The approach enables us to take advantage of graphical models of the platform and of automated refinements of these models towards implementation.

We present a methodology for synthesizing TTA protocol processors onto CMOS and FPGA from application specifications with reduced designer intervention and a short turn-around time. The methodology builds up on our earlier work in generating synthesizable processor models from system level specifications for 0.18 /spl mu/m CMOS technology. We test the application level methodology by comparing results obtained from a generated FPGA synthesis model to results obtained from a generated CMOS synthesis model. We synthesized an architecture for processing the IPv6 protocol, which resulted in an implementation that achieved the clock speed of 45 MHz. Due to the scalable parallelism of TTA architectures, this corresponds to an approximate throughput of 500 Mbps for IPv6 routing. From the results we were able to conclude that the critical delay in our generated FPGA implementations is formed inside our protocol processing functional units.

In this paper we present a design case study in configuring our protocol processor architecture to meet the performance requirements of IPv6 routing at gigabit speeds. Our methodology makes it possible to make fast reliable analyses of the problem on a high level and to find its key bottlenecks and design constraints. Based on the analyses we suggest architectural configurations for the target application. The best configurations can then be further analyzed in more detailed system-level simulations and physical estimations.