The use of natural language (NL) test cases for validating graphical user interface (GUI) applications is emerging as a promising direction to manually written executable test scripts, which are costly to develop and difficult to maintain. Recent advances in large language models (LLMs) have opened the possibility of the direct execution of NL test cases by LLM agents. This paper investigates this direction, focusing on the impact on NL test case unsoundness and on test case execution consistency. NL test cases are inherently unsound, as they may yield false failures due to ambiguous instructions or unpredictable agent behaviour. Furthermore, repeated executions of the same NL test case may lead to inconsistent outcomes, undermining test reliability. To address these challenges, we propose an algorithm for executing NL test cases with guardrail mechanisms and specialised agents that dynamically verify the correct execution of each test step. We introduce measures to evaluate the capabilities of LLMs in test execution and one measure to quantify execution consistency. We propose a definition of weak unsoundness to characterise contexts in which NL test case execution remains acceptable, with respect to the industrial quality levels Six Sigma. Our experimental evaluation with eight publicly available LLMs, ranging from 3B to 70B parameters, demonstrates both the potential and current limitations of current LLM agents for GUI testing. Our experiments show that Meta Llama 3.1 70B demonstrates acceptable capabilities in NL test case execution with high execution consistency (above the level 3-sigma). We provide prototype tools, test suites, and results.

International audience

International audience

International audience

International audience

International audience

International audience

This paper addresses the problem of recovering behavioural models from IoT devices in order to help engineers understand how they are functioning and audit them.We present a model learning approach called ASSESS, which takes as inputs execution traces collected from IoT devices and generates models called systems of Labelled Transition Systems (LTSs).ASSESS generates as many LTSs as components integrated and identified into a device.The approach is specialised to IoT devices as it takes into account two architectures often used to integrate components with this kind of system (cyclic functioning, loosely-coupled or decoupled architectures).We experimented the approach on two IoT devices and an IoT gateway to evaluate the model conciseness and the approach efficiency.

Design Patterns are now widely accepted and used in software engineering; they represent generic and reusable solutions to common problems in software design. Security patterns are specialised patterns whose purpose is to help design applications that should meet security requirements. The enthusiasm surrounding security patterns has made emerge several catalogues listing up to 180 different patterns at the moment. This growing number brings an increased difficulty in choosing the most appropriate patterns for a given design problem. We propose a security pattern classification to facilitate the security pattern choice and a classification method based on data integration. The classification exposes relationships among software attacks, security principles and security patterns. It expresses the pattern combinations that are countermeasures to a given attack. This classification is semi-automatically inferred by means of a data-store integrating disparate publicly available security data. The data-store is also used to generate Attack Defense Trees. In our context, these illustrate, for a given attack, its sub-attacks, steps, techniques and the related defenses given under the form of security pattern combinations. Such trees make the pattern classification more readable even for beginners in security patterns. Finally, we evaluate on human subjects the benefits of using a pattern classification established for Web applications, which covers 215 attacks, 66 security principles and 26 security patterns.

This paper addresses the problem of learning models of component-based systems. We focus on model learning approaches that generate state diagram models of software or systems. We present COnfECt, a method that supplements passive model learning approaches to generate models of component-based systems seen as black-boxes. We define the behaviours of components that call each other with Callable Extended FSMs (CEFSM). COnfECt tries to detect component behaviours from execution traces and generates systems of CEFSMs. To reach that purpose, COnfECt is based on the notions of trace analysis, event correlation, model similarity and data clustering. We describe the two main steps of COnfECt in the paper and show an example of integration with the passive model learning approach Gk-tail.

Security at the design stage of the software life cycle can be performed by means of security patterns, which are viable and reusable solutions to regular security problems. Their generic nature and growing number make their choice difficult though, even for experts in system design. To guide them through the appropriate choice of patterns, we present a methodology of security pattern classification and the classification itself, which exposes relationships among CAPEC attacks, CWE weaknesses and security patterns. Given a CAPEC attack, the classification expresses the security pattern combinations that overcome the attack. The methodology, which generates the classification is composed of five steps, which decompose patterns and attacks into sets of more precise sub-properties that are associated. These steps provide the justifications of the classification and can be followed again to upgrade it. From the classification, we also generate Attack-Defense Trees (ADTtrees), which depict an attack, its sub-attacks and the related defenses in the form of security pattern combinations. Without loss of generality, this classification has been established for Web applications and covers 215 attacks, 136 software weaknesses and 26 security patterns.

Security patterns are generic solutions that can be applied since early stages of software life to overcome recurrent security weaknesses. Their generic nature and growing number make their choice difficult, even for experts in system design. To help them on the pattern choice, this paper proposes a semi-automatic methodology of classification and the classification itself, which exposes relationships among software weaknesses, security principles and security patterns. It expresses which patterns remove a given weakness with respect to the security principles that have to be addressed to fix the weakness. The methodology is based on seven steps, which anatomize patterns and weaknesses into set of more precise sub-properties that are associated through a hierarchical organization of security principles. These steps provide the detailed justifications of the resulting classification and allow its upgrade. Without loss of generality, this classification has been established for Web applications and covers 185 software weaknesses, 26 security patterns and 66 security principles. Research supported by the industrial chair on Digital Confidence (http://confiance-numerique.clermont-universite.fr/index-en.html).

This paper tackles the problem of testing production systems, i.e. systems that run in industrial environments, and that are distributed over several devices and sensors. Usually, such systems lack of models, or are expressed with models that are not up to date. Without any model, the testing process is often done by hand, and tends to be an heavy and tedious task. This paper contributes to this issue by proposing a framework called Autofunk, which combines different fields such as model inference, expert systems, and machine learning. This framework, designed with the collaboration of our industrial partner Michelin, infers formal models that can be used as specifications to perform offline passive testing. Given a large set of production messages, it infers exact models that only capture the functional behaviours of a system under analysis. Thereafter, inferred models are used as input by a passive tester, which checks whether a system under test conforms to these models. Since inferred models do not express all the possible behaviours that should happen, we define conformance with two implementation relations. We evaluate our framework on real production systems and show that it can be used in practice.

Web Services are more and more used in designing and building systems in open and dynamic distributed environments. The security of these transactions is becoming a critical issue. This paper proposes a security testing method for stateful Web Services. We define some specific security rules with the Nomad language. Then, we construct test cases from a symbolic specification and test purposes derived from the previous rules. We present some experimentation results based on roughly 100 Web Services and we show that 11 percent have vulnerabilities, using the rules introduce in the article.

This paper presents a methodology and a set of tools for the modelling, validation and testing of Web service composition, conceived and developed within the French national project WebMov. This methodology includes several modelling techniques, based mainly on some variations of Timed Extended Finite State Machines (TEFSM) formalism, which provide a formal model of the BPEL description of Web services composition. These models are used as a reference for the application of different test generation and passive testing techniques for conformance and robustness checking. The whole WebMov methodology is integrated within a dedicated framework, composed by a set of tools that implement the model representation, the test generation and passive testing algorithms. This framework also permits the interaction of these tools to achieve specific modelling and testing activities in a complementary way. A case study based on a real service, a Travel Reservation Web Service, is presented as well as the results of the application of the proposed WebMov methodology and tools.

WS-BPEL processes are usually overlapped in large Business applications composed of several Web Services. Such applications are more and more developed with respect of quality processes. Testability is an important quality degree, which evaluates the fault detection coverage during the testing process and the testing cost. In this paper, we focus on a well-known testability criterion called observability, which evaluates if enough distinguishable events can be observed while testing. We study the observability of ABPEL (Abstract-BPEL) specifications and we describe some ABPEL observability degradation properties. From these, we propose some observability enhancement methods which detect observability issues in ABPEL specifications and semi-automatically update the ABPEL code.

Asynchronous javascript and XML (AJAX) is a recent group of technologies used to develop dynamic web pages. Ajax applications are wisely used nowadays and need to be tested to ensure their reliability. This paper introduces a method and an architecture for automatic AJAX application testing. We use STS automata for describing the application and for generating test cases. We perform an improved random testing using some predefined values and also test purpose based testing for verifying specific properties. The testing framework is composed of several testers which control and monitor the test execution to give a test verdict. The Google map search application is used as an example to illustrate the method.

This paper deals with communicating system robustness testing by proposing a method for checking the correct behaviour of the Implementation Under Test (IUT) in unexpected situations. This formal method takes specifications written with IOLTS and generates robustness test cases by using the ioco theory. IOLTS are used to model many systems like distributed ones or Web service compositions. We present an algorithm permitting to generate sound test cases especially focusing on robustness aspects.

This study is devoted to test quality of timed systems. In order to produce reliable systems, they need to be tested before industrial development, but system testing costs a lot of money and time. We suggest to make some preliminary analysis of the system in order to evaluate the cost before starting the tests. This test quality is evaluated by means of four factors. We detail each factor and show how to measure them in all different cases of system specification.