This paper presents research work performed on the development and the verification of dependable reflective systems based on MetaObject Protocols (MOPS). We describe our experience, we draw the lessons learned from both a design and a validation viewpoint, and we discuss some possible future trends on this topic. The main originality of this work relies on the combination of both design and validation issues for the development of reflective systems, which has led to the definition of a reflective framework for the next generation of fault-tolerant systems. This framework includes: 1) the specification of a MetaObject Protocol suited to the implementation of fault-tolerant systems and 2) the definition of a general test strategy to guide its verification. The proposed approach is generic and solves many issues related to the use and evolution of system platforms with dependability requirements. Two different instances of the specified MOP have been implemented in order to study the impact of the MOP architecture in the development of a reflective fault-tolerant system. As far as the test strategy is concerned, a different testing level is associated with each reflective mechanism defined in the MOP. For each testing level, we characterize the test objectives and the required test environments. According to this experience, several new research challenges are finally identified.

The adoption of the object-oriented (OO) technology for the development of critical software raises important testing issues. This paper addresses one of these issues: how to create effective tests from OO specification documents? More precisely, the paper describes a technique that adapts a probabilistic method, called statistical functional testing, to the generation of test cases from UML state diagrams, using transition coverage as the testing criterion. Emphasis is put on defining an automatic way to produce both the input values and the expected outputs. The technique is automated with the aid of the Rational Software Corporation's Rose RealTime tool. An industrial case study from the avionics domain, formally specified and implemented in Java, is used to illustrate the feasibility of the technique at the subsystem level. Results of first test experiments are presented to exemplify the fault revealing power of the created statistical test cases.

The separation of concerns provided by reflective architectures is of high interest for the development of dependable systems. Beyond this initial interest, the use of this technology remains questionable due to the lack of work reporting on validation aspects. This paper defines an incremental strategy for testing the cornerstone of the reflective architectures that we consider, i.e. the MetaObject protocol (MOP). The approach is aimed at reducing the testing effort by promoting a gradual increment of the observability and controllability of the MOP under test. This strategy enables reflective mechanisms that have already been tested to be re-used for verifying the remaining ones. A different testing level is associated with each reflective mechanism defined in the MOP. For each testing level, we characterize the test objectives and the required test environment. The feasibility of the approach is exemplified on a real MOP implemented using OpenC++ and extracted from the /spl Fscr//spl Rscr//spl Iscr//spl Escr//spl Nscr//spl Dscr//spl Sscr/ architecture, which is devoted to the implementation of CORBA dependable systems. First test experiments were very useful, since they revealed some implementation errors.

This paper presents an empirical study of the effectiveness of test cases generated from UML state diagrams using transition coverage as the testing criterion. The test cases production is mainly based on an adaptation of a probabilistic method, called statistical testing based on testing criteria. This technique was automated with the aid of the Rational Software Corporation's Rose RealTime tool. The test strategy investigated combines statistical test cases with (few) deterministic test cases focused on domain boundary values. Its feasibility is exemplified on a research version of an avionics system implemented in Java: the Flight Guidance System case study (14 concurrent state diagrams). Then, the results of an empirical evaluation of the effectiveness of the created test cases are presented. The evaluation was performed using mutation analysis to assess the error detection power of the test cases on more than 1500 faults seeded one by one in the Java source code (115 classes, 6500 LOC). A detailed analysis of the test results allows us to draw first conclusions on the expected strengths and weaknesses of the proposed test strategy.

One of the characteristics of object-oriented software is the complex dependency that may exist between classes due to inheritance, association and aggregation relationships. Hence, where to start testing and how to define an integration strategy are issues that require further investigation. This paper presents an approach to define a test order by exploiting a model produced during design stages (e.g., using OMT, UML), namely the class diagram. Our goal is to minimize the number of stubs to be constructed in order to decrease the cost of testing. This is done by testing a class after the classes it depends on. The novelty of the test order lies in the fact that it takes account of: (i) dynamic (polymorphism) dependencies; (ii) abstract classes that cannot be instantiated, making some testing levels infeasible. The test order is represented by a graph showing which testing levels must be done in sequence and which ones may be done independently. It also provides information about the classes involved in each level and how they are involved (e.g., instantiation or not). The approach is implemented in a tool called TOONS (Testing level generator for Object-OrieNted Software). It is applied to an industrial case study from the avionics domain.

The paper reports on a first experimental comparison of software errors generated by real faults and by 1st-order mutations. The experiments were conducted on a program developed by a student from the industrial specification of a critical software from the civil nuclear field. Emphasis was put on the analysis of errors produced upon activation of 12 real faults by focusing on the mechanisms of error creation, masking, and propagation up to failure occurrence, and on the comparison of these errors with those created by 24 mutations. The results involve a total of 3730 errors recorded from program execution traces: 1458 errors were produced by the real faults, and the 2272 others by the mutations. They are in favor of a suitable consistency between errors generated by mutations and by real faults: 85% of the 2272 errors due to the mutations were also produced by the real faults. Moreover, it was observed that although the studied mutations were simple faults, they can create erroneous behaviors as complex as those identified for the real faults. This lends support to the representativeness of errors due to mutations.

This paper is concerned with the use of statistical testing as a verification technique for complex software. Statistical testing involves exercising a program with random inputs, the test profile and the number of generated inputs being determined according to criteria based on program structure or software functionality. In case of complex programs, the probabilistic generation must be based on a black box analysis, the adopted criteria being defined from behavior models deduced from the specification. The proposed approach refers to a hierarchical specification produced in the STATEMATE environment. Its feasiblity is exemplified on a safety-critical module from the nuclear field, and the efficiency in revealing actual faults is investigated through experiments involving two versions of the module.

This paper presents the new notion of minimal detecting transition sequence (MDTS). A detectable fault f in a circuit C is detected by any MDTS in a set Df called detection set associated with f. From a prescribed set of faults, we obtain a list of detection sets. This list of detection sets is calculated once for all, for a given circuit C. Once this list bas been obtained for a circuit, it may be used either to generate a deterministic test sequence, or to calculate random testing lengths within various hypothesis (input vector probabilities).