SUMMARY Model‐based test generation techniques based on random input generation and guided simulation do not satisfy the demands of high test coverage and completeness guarantees as required by safety‐critical applications. Recently, test generation techniques based on model checking have been reported to bridge this gap. To evaluate the effectiveness of these techniques, an in‐house tool suite, AutoMOTGen, has been developed for Simulink/Stateflow and applied on real‐life case studies at General Motors. This paper outlines the test generation methodology of AutoMOTGen and gives a comparative study with a commercial, primarily random input‐based, test generation tool on the same set of examples. The results indicate that in terms of coverage, model checking‐based techniques complement the random input‐based techniques. In addition, they provide proofs for unreachability that can aid in debugging the models. Therefore, it is recommended that model checking‐based tools be utilized to complement and enhance the effectiveness of model‐based testing methods in safety‐critical systems engineering. Copyright © 2013 John Wiley & Sons, Ltd.

The current practice of component based engineering raises concerns in industry when the specification of proprietary components suffers from inaccuracy and incompleteness. Engineers face difficulties in producing quality systems when they lack knowledge of the interoperability of components. In order to address this issue, we present a novel framework for iterative refinement of specification for component based systems. The novelty is the use of a preliminary behavioral model as a source for triggering refinement iterations. Moreover, it exploits rigorous formal techniques to achieve high-level system validation as an integral part of the refinement procedure. The framework has been evaluated on an automotive system in which the embedded software control units were developed by third-party vendors. The final results produced an improved formal system specification that identified several behaviors that were previously unknown.

We present the CoGenTe tool for automated black-box testing of code generators. A code generator is a program that takes a model in a high-level modeling language as input, and outputs a program that captures the behaviour of the model. Thus, a code generator's input and output are complex objects having not just syntactic structure but execution semantics, too. Hence, traditional test generation methods that take only syntax into account are not effective in testing code generators. CoGenTe amends this by incorporating various coverage criteria over semantics. This enables it to generate test-cases with a higher potential of revealing subtle semantic errors in code generators. CoGenTe has uncovered such issues in widely used real-life code generators: (i) lexical analyzer generators Flex and JFlex, and (ii) The MathWorks' simulator/code generator for Stateflow.

Aimed at verifying safety properties and improving simula-tion coverage for hybrid systems models of embedded control software, we propose a technique that combines numerical simulation and symbolic methods for computing state-sets. We consider systems with linear dynamics described in the commercial modeling tool Simulink/Stateflow. Given an ini-tial state x, and a discrete-time simulation trajectory, our method computes a set of initial states that are guaranteed to be equivalent to x, where two initial states are consid-ered to be equivalent if the resulting simulation trajectories contain the same discrete components at each step of the simulation. We illustrate the benefits of our method on two case studies. One case study is a benchmark proposed in the literature for hybrid systems verification and another is a Simulink demo model from Mathworks.

This paper addresses the problem of testing auto-code generators. Auto-code generators take as input a model in certain modeling language, and produce as output a program that captures the execution semantics of the input-model. We focus on the problem of test specification for the purpose of automatically generating a test-suite. We propose a novel technique for test specification based on the execution behavior of models. We also propose an algorithm that uses such a behavioral test specification for directing test-case generation towards very specific behavioral patterns that we would like to exercise. We have implemented this technique, and have applied it for generating test-cases for a Stateflow auto-code generator.

Model-based development is increasingly becoming the method of choice for developing embedded systems for applications in automotive and aerospace industries. It relies on tool-suites consisting of a variety of model-processing tools like simulators, model-translators and code-generators. The correctness of these tools used in the development process is a key requirement for safety critical applications. This paper proposes a novel testing methodology for the rigorous verification of model processing tools. The proposed methodology takes as input the syntactic and semantic meta-model of a modeling language, expressed in the form of inference rules. Using a coverage criteria over this meta-model, it generates test-models, and test-inputs for these test-models. Apart from testing the syntactic aspects of the translation, our method aims at testing subtle semantic interactions of the modeling language that are potentially mistranslated by the model-processing tools. We illustrate the methodology with a simple prototypical process calculus. We also report on the experiments carried out with Stateflow, a variant of hierarchical state-machines implemented in the Matlab/Simulink tool-suite

We address the problem of rigorous testing of program generators. Program generators are software that take as input a model in a certain modeling language, and produce as output a program that captures the execution semantics of the input-model. In this sense, program generators are also programs and, at first sight, the traditional techniques for testing programs ought to be applicable to program generators as well. However, the rich semantic structure of the inputs and outputs of program generators poses unique challenges that have so far not been addressed sufficiently in the testing literature. We present a novel automatic test-case generation method for testing program generators. It is based on both syntax and semantics of the modeling language, and can uncover subtle semantic errors in the program generator. We demonstrate our method on flex, a prototypical lexical analyzer generator.

The development of energy and performance-efficient embedded software increasingly relies on the application of complex transformations on critical parts of the source code. Designers applying such nontrivial source code transformations are often faced with the problem of ensuring functional equivalence of the original and transformed programs. Currently, they have to rely on incomplete and time-consuming simulation. Formal automatic verification of the transformed program against the original is desirable instead. This calls for equivalence checking tools similar to the ones available for comparing digital circuits. We present such a tool to compare array-intensive programs related through a combination of important global transformations like expression propagations, loop and algebraic transformations. When the transformed program fails to pass the equivalence check, the tool provides specific feedback on the possible locations of errors.