This paper presents a validation approach to feature selection in software product lines (SPL). SPLs consist of similar products tailored to different needs, while SPLs sharing a common platform where feature configurations define product families. Validating feature configurations is critical to avoid defective shipments, recalls, and disposal. Exhaustive, pairwise, and combinatorial testing, among others, aim at ensuring configuration correctness. This paper introduces a novel method for improving feature selection and validation in SPLs by minimizing redundancy while ensuring configurations align with customer needs. The method emphasizes uncovering the differences in feature structures through "complex" and "simple" models, which helps identify and helps identify and tolerate potential errors arising from incorrect feature configurations. This ensures broader coverage while effectively managing dependencies. A case study using the Access Point (AP) SPL model, which is a networking device designed to enhance the strength of an existing wireless signal and expand its coverage area. The AP can enable or disable specific features on AP SPL depending on the characteristics of the third-party gateway with which it is integrated. AP SPL model with 66 features lead to 266configurations, generated by Exhaustive Testing. Pairwise testing achieves 87% coverage with 132 test cases, while combinatorial testing reaches 94% with 45,760 cases. Our method ensures 100% feature coverage with just 3 test configurations. Thus, the approach introduced in this paper enhances product quality while reducing costs by avoiding redundant tests, making the approaches valuable for large-scale SPLs.

Existing software product line (SPL) engineering testing approaches generally provide positive testing that validates the SPL’s functionality. Negative testing is commonly neglected. This research aims to unify behavioral and feature models of an SPL, enable testing before and after variability binding for domain-centric and product-centric testing, and combine positive and negative testing for a holistic testing view. This study suggests behavioral modeling with event sequence graphs (ESGs). This heterogeneous modeling strategy supports bottom-up domain testing and top-down product testing with the feature model. This new feature-oriented ESG test creation method generates shorter test sequences than the original ESG optimum test sequences. Statechart and original ESG test-generating methods are compared. Positive testing findings are similar. The Statechart technique generated 12 test cases with 59 events, whereas the ESG technique created six test cases with 60 events. The ESG technique generated 205 negative test cases with 858 events with the Test Suite Designer tool. However, the Conformiq Designer tool for the Statechart technique does not have a negative test case generation capability. It is shown that the proposed ESG-based holistic approach confirms not only the desirable (positive) properties but also the undesirable (negative) ones. As an additional research, the traditional ESG test-generating approach is compared to the new feature-oriented method on six SPLs of different sizes and features. Our case study results show that the traditional ESG test generation approach demonstrated higher positive test generation scores compare to the proposed feature-oriented test generation approach. However, our proposed feature-oriented test generation approach is capable of generating shorter test sequences, which could be beneficial for reducing the execution time of test cases compared to traditional ESG approach. Finally, our case study has also shown that regardless of the test generation approach, there has been found no significant difference between the Bottom-up and Top-down test strategies with respect to their positive test generation scores.

Boolean expressions are highly involved in control flows of programs and software specifications. Coverage criteria for Boolean expressions aim at producing minimal test suites to detect software faults. There exist various testing criteria, efficiency of which is usually evaluated through mutation analysis. This paper proposes an integer programming-based minimal test suite generation technique relying on mutation analysis. The proposed technique also takes into account the cost of fault detection. The technique is optimal such that the resulting test suite guarantees to detect all the mutants under given fault assumptions, while maximizing the average percentage of fault detection of a test suite. Therefore, the approach presented can also be considered as a reference method to check the efficiency of any common technique. The method is evaluated using four well-known real benchmark sets of Boolean expressions and is also exemplary compared with MCDC criterion. The results show that the test suites generated by the proposed method provide better fault coverage values and faster fault detection.

Model-based GUI testing has achieved widespread recognition in academy thanks to its advantages compared to code-based testing due to its potentials to automate testing and the ability to cover bigger parts more efficiently.In this study design paper, we address the scalability part of the model-based GUI testing by using community detection algorithms.A case study is presented as an example of possible improvements to make a model-based testing approach more efficient.We demonstrate layered ESG models as an example of our approach to consider the scalability problem.We present rough calculations with expected results, which show 9 times smaller time and space units for 100 events in the ESG model when a community detection algorithm is applied.

Presents the introductory welcome message from the conference proceedings. May include the conference officers' congratulations to all involved with the conference event and publication of the proceedings record.

User inputs are critical for the security, safety, and reliability of software systems. This paper proposes a new concept called user input contracts, which is an integral part of a design-by-contract supplemented development process, and a model-based testing approach to detect violations of user input contracts. The approach generates test cases from an input contract integrated with graph-based model of user interface specification and applies them to the system under consideration. The paper presents a proof-of-concept tool that has been developed and used to validate the approach by experiments. The experiments are conducted on a web-based system for marketing tourist services to analyze input robustness of system under consideration with respect to user input contracts.

This paper rigorously introduces the concept of model-based mutation testing (MBMT) and positions it in the landscape of mutation testing. Two elementary mutation operators, insertion and omission , are exemplarily applied to a hierarchy of graph-based models of increasing expressive power including directed graphs, event sequence graphs, finite-state machines and statecharts. Test cases generated based on the mutated models ( mutants ) are used to determine not only whether each mutant can be killed but also whether there are any faults in the corresponding system under consideration (SUC) developed based on the original model. Novelties of our approach are: (1) evaluation of the fault detection capability (in terms of revealing faults in the SUC) of test sets generated based on the mutated models, and (2) superseding of the great variety of existing mutation operators by iterations and combinations of the two proposed elementary operators. Three case studies were conducted on industrial and commercial real-life systems to demonstrate the feasibility of using the proposed MBMT approach in detecting faults in SUC, and to analyze its characteristic features. Our experimental data suggest that test sets generated based on the mutated models created by insertion operators are more effective in revealing faults in SUC than those generated by omission operators. Worth noting is that test sets following the MBMT approach were able to detect faults in the systems that were tested by manufacturers and independent testing organizations before they were released.

Model-based testing employs models for testing. Model-based mutation testing (MBMT) additionally involves fault models, called mutants, by applying mutation operators to the original model. A problem encountered with MBMT is the elimination of equivalent mutants and multiple mutants modeling the same faults. Another problem is the need to compare a mutant to the original model for test generation. This paper proposes an event-based approach to MBMT that is not fixed on single events and a single model but rather operates on sequences of events of length k ≥ 1 and invokes a sequence of models that are derived from the original one by varying its morphology based on k. The approach employs formal grammars, related mutation operators, and algorithms to generate test cases, enabling the following: (1) the exclusion of equivalent mutants and multiple mutants; (2) the generation of a test case in linear time to kill a selected mutant without comparing it to the original model; (3) the analysis of morphologically different models enabling the systematic generation of mutants, thereby extending the set of fault models studied in related literature. Three case studies validate the approach and analyze its characteristics in comparison to random testing and another MBMT approach.

SUMMARY The behavior of composed Web services depends on the results of the invoked services; unexpected behavior of one of the invoked services can threat the correct execution of an entire composition. This paper proposes an event‐based approach to black‐box testing of Web service compositions based on event sequence graphs, which are extended by facilities to deal not only with service behavior under regular circumstances (i.e., where cooperating services are working as expected) but also with their behavior in undesirable situations (i.e., where cooperating services arenotworking as expected). Furthermore, the approach can be used independently of artifacts (e.g., Business Process Execution Language) or type of composition (orchestration/choreography). A large case study, based on a commercial Web application, demonstrates the feasibility of the approach and analyzes its characteristics. Test generation and execution are supported by dedicated tools. Especially, the use of an enterprise service bus for test execution is noteworthy and differs from other approaches. The results of the case study encourage to suggest that the new approach has the power to detect faults systematically, performing properly even with complex and large compositions. Copyright © 2012 John Wiley & Sons, Ltd.

Model-based testing (MBT) focuses on relevant, mostly user-centric features of the system under consideration (SUC) and enables test case generation without requiring source code. Depending on these features and the preferences of the tester, modeling can be event-based or state-based. This paper compares both techniques using mutation testing, which is originally code-based, but has recently been extended to enable also MBT. For the comparison, the paper introduces frameworks that are composed of a set of models, a set of mutation operators, a set of coverage criteria, and a set of test generation algorithms. The introduced concepts and notions are demonstrated over a case study based on a large web-based commercial portal. Analysis of the experimental data yields results on the discussed frameworks reviewing benefits and drawbacks of event-based and state-based testing.

Model-based testing attempts to generate test cases from a model focusing on relevant aspects of a given system under consideration (SUC). When SUC becomes too large to be modeled in a single step, existing design techniques usually require a modularization of the modeling process. Thereby, the refinement process results in a decomposition of the model into several hierarchical layers. Conventional testing requires the refined components be completely replaced by these subcomponents for test case generation. Mostly, this resolution of components leads to an oversized, large model where test case generation becomes very costly, and the generated test case set is very large leading to infeasible long test execution time. To solve these problems, we present a new strategy to reduce (i) the number of test cases, and (ii) the costs of test case generation and test execution. For determining the trade-off due to this cost reduction, the reliability achieved by the new approach is compared with the reliability of the conventional approach. A case study based on a large web-based commercial system validates the approach and discusses its characteristics. We found out that the new approach could detect about 80% of the faults for about 20% of the test effort compared with the conventional approach.

A go-back (GB) function for canceling recent user or system operations and going back to and resuming of previous state(s) is very often used regardless of the application domain. Therefore, faulty handling of them can cause severe damages in those applications. This paper proposes a mutation-based approach to testing GB functions modeled by pushdown automata. Novel mutation operators, recent coverage criteria, and a new algorithm for test case generation are introduced. A case study validates the approach and discusses its characteristics.

Testing of graphical user interfaces is important due to its potential to reveal faults in operation and performance of the system under consideration. Most existing test approaches generate test cases as sequences of events of different length. The cost of the test process depends on the number and total length of those test sequences. One of the problems to be encountered is the determination of the test sequence length. Widely accepted hypothesis is that the longer the test sequences, the higher the chances to detect faults. However, there is no evidence that an increase of the test sequence length really affect the fault detection. This paper introduces a reliability theoretical approach to analyze the problem in the light of real-life case studies. Based on a reliability growth model the expected number of additional faults is predicted that will be detected when increasing the length of test sequences.

Boundary overflows are caused by violation of constraints, mostly limiting the range of internal values of program, and can be provoked by an intruder to gain control of or access to stored data. In order to countermeasure this well-known vulnerability issue, this paper focuses on input validation of graphical user interfaces (GUI). The approach proposed generates test cases for numerical inputs based on GUI specification through decision tables. If boundary overflow error(s) are detected, the source code will be analyzed to localize and correct the encountered error(s) automatically.

This paper proposes an approach for validation of numerical inputs based on graphical user interfaces (GUI) that are modeled and specified by event sequence graphs (ESG). For considering complex structures of input data, ESGs are augmented by decision tables and patterns of design by contract (DbC). The approach is evaluated by experiments on boundary overflows, which occur when input values violate the range of specified values. Furthermore, a tool is presented that implements our approach enabling a semi-automatically detection of boundary overflow errors and suggesting correction steps based on DbC.

A service-oriented architecture (SOA) for Web applications is often implemented using Web service (WS) standards and consists of different functions the executions of which are perceived as events. The order and time-appropriateness of occurrences of these events play a vital role for the proper working of a real-time SOA. This paper presents an event-based approach for modeling and testing of functional behavior of WS in SOA by event sequence graphs (ESG). Nodes of ESG represent events, e.g., "request" or "response", and arcs give the sequence of these events. For representing parameter values, e.g., for time-out of function calls, ESG are augmented by decision tables (DT). A case study carried out on a commercial Web system with SOA validates the approach and analyzes its characteristic issues. The novelty of the approach stems from (i) its simplicity and lucidity in representing complex real-time Web applications based on WS in SOA, and (ii) its modeling that considers a comfortable fault management.

In the fall of 1968, NATO hosted in Garmisch-Partenkirchen, close to Munich, a conference devoted to the problems of the computer industry that was having a great deal of trouble in producing large and complex programs. The term Software Engineering SE) was not in general use at that time, its adoption for the title of this conference was deliberately provocative. As a result, the conference and its report have played a major role in gaining general acceptance of the term SE.

Presents the introductory welcome message from the conference proceedings.

Provides a listing of current committee members and society officers.

This paper proposes a graph-model-based approach to prioritizing the test process. Tests are ranked according to their preference degrees which are determined indirectly, i.e., through classifying the events. To construct the groups of events, unsupervised neural network is trained by adaptive competitive learning algorithm. A case study demonstrates and validates the approach.

With the growing complexity of computer-based systems, their graphical user interfaces have also become more complex. Accordingly, the test and analysis process becomes more tedious and costly. This paper introduces a holistic view of fault modelling that is carried out as a complementary step to system modelling, enabling a scalability of the test process, and providing considerable potential for automation. Event-based notions and tools are used to generate and select test cases systematically. The elements of the approach are illustrated and validated by a case study. This paper does not claim to introduce a novel theoretic approach; rather, it makes use of graph-theoretic results for a practical and simple, but nevertheless powerful, view of modelling, analysis and testing of graphical user interfaces. Copyright © 2005 John Wiley & Sons, Ltd.

It is widely accepted that conventional test methods are not necessarily adequate for testing of component-based software (CBS). As a consequence, also conventional test tools cause similar problems for the test automation of CBS based on their graphical user interfaces (GUI), because for any level of user-focused testing domain knowledge and knowledge about the implementation of the CBS are essential to run the tests. The component manufacturer, on the other side, is usually not willing to deliver the code to protect his, or her, commercial interest. For solving this conflict, this paper introduces a framework for the automation of user-oriented component testing that significantly reduces the test costs. The concept is based on black-box testing techniques and utilizes the common features of commercial capture/replay test tools.

Based on finite-state automata (FSA) and equivalent regular expressions, the paper introduces a holistic view of fault modeling that can be carried out as a complementary step to system modeling, revealing much rationalization potential. Appropriate formal notions will be used to introduce efficient algorithms to systematically generate and select test cases. The completeness of the test can be determined exploiting the link coverage of the state transition diagram of the FSA that models both the desired and undesired behavior of the system under test; this enables a precise scalability of the test and analysis process, leading to a better cost-effectiveness The elements of the approach will be narrated by realistic examples which will be used also to validate the approach.

Two facts about declarative programming prevent the application of conventional testing methods. First, the classical test coverage measures such as statement, branch or path coverage, cannot be used, since in declarative programs no control flow notion exists. Second, there is no widely accepted language available for formal specification, since predicate logic, which is the most common formalism for declarative programming, is already a very high-level abstract language. This paper presents a new approach exending previous work by the authors on test input generation for declarative programs. For this purpose, the existing program instrumentation notion is extended and a new logic coverage measure is introduced. The approach is mathematically formalized and the goal of achieving 100% program logic coverage controls the automatic test input generation. The method is depicted by means of logic programming; the results are, however, generally applicable. Finally, the concepts introduced have been used practically within a test environment. © 1998 John Wiley & Sons, Ltd.

Checklist-based code-reviews have been generally accepted as valuable means for software development and management. In order to overcome shortcomings of manual reviewing techniques, such as high costs and lack of systematization, we have already developed and implemented a knowledge-based approach for semi-automation of some steps of individual code-reviews based on checklists. The aim of this paper is to evaluate the performance of our approach for code-reviews. Therefore, two independent groups of reviewers were analyzing the same piece of code (conventional C-programs developed in the automotive industry for gear unit control), where one of the groups deployed the semi-automated approach and the other group used the traditional manual technique. The resulting empirical data were analyzed by means of software metrics and software reliability modeling. Metrics specific to review processes, e.g. average review rate, average preparation rate, etc. were adapted for our knowledge-based methodology in order to capture the particularities of our approach. The results of these empirical investigations are compared with other methods (e.g. statistical, plan-based scenario-based) for reviews' computer support.

We present a new control flow based approach to dynamic testing of sequential software. A practicable number of test cases is generated by using the boundary interior path testing strategy (J.B. Goodenough and S.L. Gerhard, 1975) and by dividing the test units into test segments (program fragments composed of one statement or a sequence of statements). The size of the test segments can be adjusted by means of a parameter, i.e. the thoroughness of the test coverage can be adapted to the needs of the tester. The selection of test cases is performed by constructing path classes for each test segment. The coverage criteria constructed by means of our approach (test segment coverage criteria) are fulfilled if at least one path from each path class is covered. A validation of our approach is given by comparing the fault detection capabilities of test segment coverage criteria with the fault detection capabilities of branch testing, multiple condition testing, LCSAJ testing and all-uses testing using n test cases for each item (e.g. branch) to be covered. The comparison demonstrates that, compared with the other testing criteria, greater fault detection probabilities can be achieved if a test segment coverage criterion is used.

Different types of code-reviews (Fagan-style code-inspections, Parnas-like active design reviews and walkthroughs) have been found to be very useful in improving the quality of software. In many cases reviewers use checklists to guide their analysis during review sessions. However, valuable, checklist-based code-reviews have the principal shortcoming of their high costs due to lack of supporting tools enabling at least partial automation of typical multiple appearing rules. This paper describes an approach towards semi-automation of some steps of individual review processes based on checklists. The method proposed is interactive, i.e. reviewers will be enabled to actualize, extend, and check the consistency and redundancy of their checklists. The basic idea underlying the approach is the usage of a rule-based system, adapting concepts of the compiler theory and knowledge engineering, for acquisition and representation of knowledge about the program. Redundant and conflicting knowledge about the program under study is recognized and solved by means of an embedded truth maintenance system. As a result of fault diagnosis, rules for fault classification are used. Software reliability models are applied to validate the results of each review session. The approach has shown promising preliminary results in analyses of conventional C-programs developed in the automotive industry.

A method is presented for systems design, specification and analysis based on predicate/transition nets (PrT nets) and logic programming. In order to evaluate and optimize the behavior of systems, the PrT net models are extended with quantitative time. The firing times are assigned to transitions and are given by arithmetic expressions which might contain variables An approach to transforming requirements driven PrT net models into logic programs is given. The generated logic programs contain the static structure and the dynamic behavior of the PrT net models and can be used for simulation and analysis purposes. Moreover they can be viewed as a precise and lucid specification of the programs to be implemented. Three examples are given which illustrate how Prolog goals can be used to validate system model properties. Example 1 describes the transformation steps for simple PrT net models. In Example 2 the generated Prolog program corresponding to a PrT net model containing formulae inside the transitions is described. Example 3 illustrates how time constraints of PrT net models with variable time can be validated.

Reliability determination for software is closely related to software testing. Testing delivers important data for software reliability models. Two important tasks of software testing are test case generation and determination of test coverage. Because of its declarative paradigm, all the well-known approaches to the above mentioned tasks are not applicable to logic programming. Implementation based testing is an approach to tackle the test problem for logic programming. We focus on test coverage aspects for logic programming. Analog to the coverage notion for conventional programming, e.g., path coverage on the control flow graph, we define a coverage measure on an abstract model of the logic program to be tested. Our abstract model is the set of goal-induced instances of program clauses. This is motivated by the computational model of resolution calculus for first-order predicate logic. Anti-unification is utilized to define the coverage measure via least upper bounds of this set. We give an algorithm for test cover determination which is prepared for coverage oriented test input generation, e.g., by declarative program instrumentation or non goal-oriented execution of logic programs using a fair interpreter. We describe syntactic and semantic aspects of our testing approach with respect to the properties of logic programming.

Abstract In this paper an approach to reliability prediction and estimation of Prolog programs is proposed. Two complexity measures describing Prolog programs are introduced. Values of the two measures are used, subsequently, to predict the reliability of Prolog programs before testing and in the early testing stages, and further, to estimate the reliability as a function of time, in order to determine whether the reliability objective is achieved. The proposed reliability determination approach is based on previous work (Azem et al., 1993), extending the prediction approach used therein through modification of the complexity measures and providing an estimation approach. It leads to improvements in the quality of predictions and estimations with respect to software reliability characteristics. The proposed approach is implemented in a reliability assessment environment, which also includes several well‐known software reliability models for comparison purposes.

The systematic approaches to testing and reliability determination of programs e.g. are applicable to imperative programming but not immediately to declarative programming, such as logic programming, which is of great importance to develop knowledge-based systems. We describe an approach to implementation-based testing and reliability determination of logic programs materialized in a product assurance environment, presently limited to two major components: the test environment PROTest and the reliability assessment environment PRORool, with the results of the former serving as input for the latter. The test environment consists of structure analysis of logic programs, automatic test case generation and execution, test coverage determination, and generation of test reports. The reliability assessment environment provides an approach to reliability prediction and estimation of Prolog programs, introducing two measures describing Prolog programs complexity, which are used to determine the program reliability. It implements also several well-known software reliability models for comparison purposes.

In this paper, we describe the PROTest II (PROlog Test Environment, Version 2) system to test logic programs in an interactive support environment. Logic programs are augmented with declarative information about the types and modes of the arguments of a predicate. Modes correspond to in, out, and in-out parameters. With this information PROTest II statically checks the types of Prolog programs, generates test cases, executes Prolog programs, and produces reports summarizing results including information about new test coverage metrics. Thus, PROTest II enables both static analysis and dynamic testing uniformly using a Prolog-based test language DTL/1. The strength of PROTest II stems from its idea of defining coverage in real logic programming terms, rather than adapting imperative programming ideas.

Consensus recovery block scheme and concurrent recovery scheme may become useful for real-time applications as they consider concurrency and time efficiency. The authors compare the performance of both techniques in terms of the times they consume to achieve software fault tolerance. For this purpose a simple model is used and analysed which simulates different real-time situations.>

A method to systematically integrate fault tolerance properties into the design of complex software systems is presented. The method exploits a formal specification of the system in which the amount of necessary redundancy can be determined. The system description is based on a combination of a predicate/transition net with regular expressions. The net model provides a formal overview of the system behavior in general, supporting the correct understanding of potential concurrency in the system processes. Regular expressions are used to model the sequential behavior of single-system components in detail. Both model layers provide well-defined levels of error detection; the regular expressions enable the system designer to also determine and introduce redundancy to achieve error correction. The techniques used to describe and analyze system behavior are explained using a case study that contains a stepwise-refined specification and analysis of a multistory shelving system model that has been implemented using the method presented. It is shown that the method applies to any software system which is to be protected against the considered errors.>

An approach to the optimization of software reliability is proposed. The emphasis is put on the software redundancy to achieve fault tolerance, i.e. the results of the optimization process are used to determine the optimal structure of the software to be developed. Two optimization models are formulated covering, respectively, modified recovery block scheme and multiversion programming approaches. Both cases are illustrated by simple examples. The models show that it is possible to formulate and solve some software related reliability optimization problems. They further show that the concept of redundancy to achieve fault tolerance (basic for the traditional theory of reliability) can be used in the field of software reliability optimization.>