In industry, software testing is the primary method to verify and validate the functionality, performance, security, usability, and so on, of software-based systems. Test automation has gained increasing attention in industry over the last decade, following decades of intense research into test automation and model-based testing. However, designing, developing, maintaining and evolving test automation is a considerable effort. Meanwhile, AI’s breakthroughs in many engineering fields are opening up new perspectives for software testing, for both manual and automated testing. This paper reviews recent research on AI augmentation in software test automation, from no automation to full automation. It also discusses new forms of testing made possible by AI. Based on this, the newly developed taxonomy, ai4st , is presented and used to classify recent research and identify open research questions.

Firewalls are a critical part in any security frame-work. Most firewalls consist of a large amount of sequential rules that are unstructured and confusing. Unfortunately, because a lot of rules configuration work is done manually by the network administrators, misconfigurations are very common and can affect the reliability of the firewall. Identifying such anomalies is a challenging task. In this paper, we propose a tree based simulation and verification model to verify if the implemented firewall of a system is in compliance with the belonging firewall requirements. The proposed methodology was developed in relation with the H2020 FORTIKA project and was evaluated in the scope of case studies with industrial partners. The case studies in question related to large scale telecom infrastructures involving critical scenarios in the scope of Smart Cities in general and SME cyber-security protection. Thereby, the executed case studies demonstrate how our approach can lead to improved structuring of firewalls and belonging rules, to the comfortable visualization of firewall structures and decision patterns, and finally to the verification of system and context requirements imposed by the firewall operation environment.

The 10th International Conference on Software Testing, Verification, and Validation (ICST 2017) was held on March 13 to 17, 2017, in Tokyo, Japan. The aim of the ICST conference is to bring together researchers and practitioners who study the theory, techniques, technologies, and applications that concern all aspects of software testing, verification, and validation of software systems. In the main research program, the ICST 2017 program chairs, Ina Schieferdecker and Hironori Washizaki selected 36 full papers and 8 short papers for inclusion in the proceedings from among 135 submissions based on the recommendation of the program committee. All papers were refereed by at least three program committee members. Of the 36 full papers accepted, we selected seven papers for consideration for this special issue of STVR. These papers were extended from their conference version by the authors and were reviewed according to the standard STVR reviewing process. We thank all the ICST and STVR reviewers for their hard work. Three papers successfully completed the review process and are contained in this special issue. The rest of this editorial provides a brief overview of these three papers. The first paper - “Choosing The Fitness Function for the Job: Automated Generation of Test Suites that Detect Real Faults” by Alireza Salahirad, Hussein Almulla, and Gregory Gay - studies the effectiveness of different fitness functions in search-based unit test generation for detecting faults. Experiment results on real faults from the Defects4J database reveal that the branch coverage fitness function is the most effective. The study also reveals that the most important factor related to the likelihood of detection is the satisfaction of the chosen criterion's test obligations. The second paper - “Complexity Vulnerability Analysis using Symbolic Execution” - Kasper Luckow, Rody Kersten, and Corina Pasareanu - presents a symbolic execution technique for analyzing the worst-case complexity of programs. The technique uses path policies to guide the symbolic execution towards worst-case paths. The evaluation shows that the technique can detect complexity vulnerabilities in realistic software as well as standard implementations of classic algorithms. The third paper - “Model-based Testing of Apache ZooKeeper: Fundamental API Usage and Watchers” by Cyrille Artho, Kazuaki Banzai, Quentin Gros, Guillaume Rousset, Lei Ma, Takashi Kitamura, Masami Hagiya, Yoshinori Tanabe, and Mitsuharu Yamamoto - presents a model-based testing technique to generate test cases for concurrent client sessions executing against ZooKeeper. The technique defines the semantics of watchers in ZooKeeper and the test oracle to handle the chain of events that eventually leads to the watcher being triggered. The evaluation shows that the technique can detect known defects as well as seeded mutations that implement possible flaws.

It is the aim of IoT-Testware to supply a rich set of TTCN-3 test suites and test cases for IoT technologies to enable developers in setting up a comprehensive test environment of their own, if needed from the beginning of a project. Initially, IoT-Testware will focus on protocols like CoAP and MQTT. To ensure test and implementation technology independence, the test suites will be realized in TTCN-3 and implemented with Titan. TTCN-3 has been defined and standardized by the European Telecommunication Standards Institute in ETSI ES 201873 and related extension packages. It is implemented and supported in Eclipse IoT by the Titan project. The test suites will contain tests for conformance, interoperability, robustness, and security aspects.

Modern IP-based wide-area surveillance systems often build on networks of multi-modal, intelligent and mobile sensor units. Detection of complex events is performed on intelligent sensors and fusing input in the sensor units or centralized control room components. The domain of surveillance and public safety creates requirement for robustness and fault-tolerance. This article will present an automated intelligence architecture for mobile surveillance, which provides capabilities for combining on-board event detection in sensor units, centralized decision making on the server side, and automated exploitation of mobile surveillance unit positioning data. This architecture must be very reliable to provide services in the face of challenges such as natural disasters and fire, potentially damaging the infrastructure of the surveillance system. To increase its reliability and robustness, we study the introduction of a self-healing system into the architecture and examine the combined system’s operation in three case studies.

A key challenge for open data portals is the aggregation of metadata from various data catalogs (on different administrative level or from different application fields) also known as metadata harvesting. This paper describes harvesting at the pilot of the German open government portal GovData.de, which is scheduled to become the data portal for all German public administration levels.

A compelling direction of improving trustworthiness of software-based systems is to open their ingredients: Open source software, open data sets, open system interfaces like open technical standards allow constructing major or up to all elements of a software-based systems. That gives means to use the wisdom of the crowd to assess and evaluate the quality, security and trustworthiness of software components. In addition, the software components can mature along a continuous feedback and revision loop with the crowd.

Requirements on the quality of open data depend on the exact context of a software and has to be set specifically for the different kinds of data in software-based systems. Quality of open data characteristics according to ISTQB include aspects of : currency; relevance; consistency; reliability; correctness; and completeness.

The European ITEA2 project DIAMONDS (Development and Industrial Application of Multi-Domain Security Testing Technologies) develops under the direction of Fraunhofer FOKUS, Berlin efficient and automated security test methods for security-critical, networked systems in various industrial domains such as industrial automation, banking and telecommunications. DIAMONDS develops methods to design objective, transparent, repeatable, and automated security tests that focus on system specifications and related risks. The project goals include the development of a security test pattern catalogue and the development of model-based security testing techniques such as risk-based testing and model-based fuzz testing. The project results are made available through publications and contributions to the standardization at ETSI and other standardization bodies. The presentation focusses on model-based fuzz testing, reviews the state of the art, compare it to similar approaches such as mutation testing, and presents first results on behaviour fuzzing for security testing.

The problems of testing software based systems that, like automobiles, steadily increase in complexity are still not solved. To cope with the requirements and complexities of today's systems, adequate test solutions are needed, which at least feature a minimum of flexibility, reusability and abstraction. The Testing and Test Control Notation TTCN-3 is a test specification language, which was originally developed to meet the requirements of testing telecommunication systems. The language is modular, well-structured, standardized and supports testing of communicating systems. However, the correctness of a large number of embedded systems can not be assessed by checking functional requirements only. In addition to that non-functional requirements, especially time related input-output behavior, have to be considered. The current version of TTCN-3 has only limited capabilities for testing such non-functional properties. To overcome these limitations we will extend TTCN-3 with a small set of specific language means that are dedicated to check real-time properties of embedded system. We will explain the syntax and semantics of the new constructs, compare our solution with the expressiveness of standard TTCN-3 and, as a proof of concept, provide a small example from the automotive domain that particularly motivate the use of TTCN-3 in the context of AUTOSAR.

The use of the standardised testing notation, testing and test control notation (TTCN-3) language has increased continuously over the last years. Many test suites of large sizes covering different domains exist. Therefore, it becomes important to provide the TTCN-3 community with methods and tools to evaluate the quality of tests. This paper presents the idea of evaluating the quality of the test data stimuli by using a data clustering method and measuring the coverage related to data clusters. A cluster contains stimuli which are considered similar for the system under test (SUT) behaviour; that means that each stimuli within a cluster should provide similar results from the test point of view.

Software development and testing of enterprise resource planning (ERP) systems demands dedicated methods to tackle its special features. As manual testing is not able to systematically test ERP systems due to the involved complexity, an effective testing approach should be automated, also requiring that the appropriate test data has to be provided alongside. In this paper we identify four main challenges regarding the provision of test data for automatic testing of ERP software: system test data supply, system test data stability, input test data constraints and test data correlation. Several possible solutions to these challenges are discussed. We conclude with an outlook to possible research activities.

Performance benchmarking is a method to assess performance characteristics of various systems across different chip/system architectures under realistic conditions, to collect measurements such as success/fail rate, response times or round-trip delays and to identify problems for scalability or usability aspects under heavy load. In this paper, a benchmarking workload model for IP multimedia subsystem (IMS) is introduced. Our performance methodology is based on simulation of real world traffic conditions by defining a set of requirements which best characterize a population of UEs (user equipment).

A major field in research and development are open, distributed, communication-intense systems, which are currently characterized by increasing heterogeneity in structure and technologies and by growing complexity and behavior dynamics. Autonomous systems are being investigated in order to return to manageable and stable systems whose operation and maintenance is affordable. By resolving the obstacles of current systems via enlarged system autonomy, the quality assurance of such dynamic, reconfigurable, adjustable systems becomes problematic. This keynote reviews requirements, methods and approaches for model-based quality assurance of autonomous systems

Automating the generation of object-oriented unit tests is a challenging task. This is mainly due to the complexity and peculiarities that the principles of object-orientation imply. One of these principles is the encapsulation of class members which prevents non-public methods and attributes of the class under test from being freely accessed.

Management of services over large distributed systems together with the growing complexity of underlying technologies has become a challenge in the present time. Faults can occur anywhere in a distributed system. Online validation addresses the activity of evaluating the behavior of services and service environments in the production phase in order to determine functional faults immediately. In this paper, we present our work on the design and implementation of an online validation platform. We describe a generic architecture, which can be instantiated for implementing validation systems for any type of distributed environments. We also present our experiences in using TTCN-3 to implement an online validation system for active network environments.

Deploying a distributed application on the nodes of a network has undoubtedly been a daunting task, which encompasses the distribution as well as the configuration of its components. Deployment and configuration concerns the placement of software components onto the nodes of a target environment, their installation, configuration, and instantiation, as well as the distribution and configuration tasks during the service usage phase. However, they are not fully addressed by the current CORBA component model (CCM). This paper elaborates on the platform architecture for deployment support and proposes the necessary extensions to current CCM.

This paper presents a new approach to interoperability testing of CORBA implementations. The ORB Under Test (OUT) is embedded in the test system in such a way, that the behavior of either the client side or the server side of the OUT is controllable and observable. Components of the test system emulate the peer entities of the OUT and the applications based on the OUT in order to get direct access both to the GIOP interface and to the CORBA APIs. A specification framework for the test system is proposed. It separates the concerns of test data and test behavior to facilitate modularity and dynamic of the test system. The description languages CORBA IDL, ODMG ODL and TTCN-3 are used in the framework.

This paper shows that testing open object-based distributed systems is, in principle, feasible by applying the ISO/ITU-T conformance testing methodology framework. In this context, a gap in the (semi-) automatic test development and execution process has been identified and bridged. We have developed an approach to the application of the standardized test notation TTCN for testing ODL-based implementations. The equivalents of TTCN features informal TINA specifications have been identified. Further a TTCN-based test system implementation has been integrated in the CORBA environment. Test specification and execution are discussed w.r.t. the TINA retailer reference point.

This paper introduces our ongoing development of a network node architecture, which meets the requirements of dynamic run-time installation and configuration of network services. The objective is to provide a network service run-time system, which is independent from installed services and suitable for effective deployment of existing, new and future services. By developing network service specific resource abstractions, the architecture will be more flexible than existing approaches.

Abstract. This paper discusses validation techniques for communication protocols and analyzes the practical use of selected validation techniques in an automated manner. A case study on validating the ATM (Asynchronous Transfer Mode) Signalling Protocol as specified by ITU-T in Q.2931 is used for this analysis. Different error classes are identified and validated. An assessment of the different validation techniques in terms of effort and quality of results is given. As a result of the case study, an evaluation of the Q.2931 SDL specification completes the paper.