Adequacy criteria for structural testing techniques are defined in terms of coverage (e.g. of statements or branches). For black-box testing, such criteria, making testers able to decide when to stop testing, are more difficult to define. In this paper, we propose such a definition for the particular case of specification-based testing techniques for synchronous software supplied by the Lutess testing environment. Lutess provides a framework for automatically building generators interacting with the software under test and feeding it with test input sequences. It requires a Lustre specification of both the software environment and the software safety properties. This specification is translated into an input-output automaton. A critical situation occurs when a safety property can be violated unless the software reacts adequately. Such situations correspond to particular states of the specification automaton, called suspect states. Suspect states definition can be used in two complementary ways: First, to design testing techniques able to reach several such states during testing. Second, to assess the thoroughness of a test input sequence in terms of covered suspect states. The above techniques are illustrated on a telephony software specification developed for the first Feature Interaction Detection Contest and involving 12 different telephone features. The thoroughness of the Lutess testing strategies is assessed as their ability to lead the software into suspect states.

The paper presents new techniques for specification based-testing of synchronous software with the Lutess tool. Lutess provides a framework consisting of automatically building generators which interact with the software under test and feed it with test input sequences. In the past few years, it has been established that operational profiles as well as scenarios are powerful tools, allowing for a better fault detection ability. As opposed to this last technique which relies on the ability of the human tester to specify scenarios, we propose an approach based solely on software specification to automatically generate input sequences which may correspond to fault revealing scenarios.