SUMMARY Despite the existence of a number of animation tools for a variety of languages, methods for employing these tools for specification testing have not been adequately explored. Similarly, despite the close correspondence between specification testing and implementation testing, the two processes are often treated independently, and relatively little investigation has been performed to explore their relationship. This paper presents the results of applying a framework and method for the systematic testing of specifications and their implementations. This framework exploits the close correspondence between specification testing and implementation testing. The framework is evaluated on a sizable case study of the Global System for Mobile Communications 11.11 Standard, which has been developed towards use in a commercial application. The evaluation demonstrates that the framework is of similar cost‐effectiveness to the BZ‐Testing‐Tools framework and more cost‐effective than manual testing. A mutation analysis detected more than 95% of non‐equivalent specification and implementation mutants. Copyright © 2010 John Wiley & Sons, Ltd.

To be adopted by architects, modelling approaches must provide a means to leverage the software patterns and architectural styles that are relevant to development practice, instead of those proscribed by black-box CASE tools. Architecture-Centric Model-Driven Software Development (AC-MDSD) is a modelling approach that provides architectural control of the generated application. However, AC-MDSD primarily focuses on generating infrastructure code. We apply AC-MDSD to web engineering and contribute a technique to define and generate system behaviour that goes beyond the create/read/update/delete infrastructure functionality. We use UML profiles augmented with OCL to specify the behaviour. We provide an example to illustrate the approach and outcomes.

Much of the research into testing model-driven systems has focused on checking the models used to drive code generation, and testing the transformations that are used. This is predicated upon the use of full code generation. However, much of the model-driven system development used in practice is based on the use of partial code generation, where the developer implements sections of the system by modifying or adding to the generated code. In these situations, it is important to test for problems in or caused by this manually modified code. In this paper, we present a pragmatic approach to testing model-driven systems based on partial code generation. Our approach uses model-based testing techniques based on reuse of development models to drive test case generation.

Abstract Many real‐time systems are safety‐and security‐critical systems and, as a result, tools and techniques for verifying them are extremely important. Simulation and testing such systems can be exceedingly time‐consuming and these techniques provide only probabilistic measures of correctness. There are a number of model‐checking tools for real‐time systems. Although they provide formal verification for models, we still need to implement these models. To increase the confidence in real‐time programs written in real‐time Java, this paper proposes a model‐based approach to the development of such programs. First, models can be mechanically verified, to check whether they satisfy particular properties, by using current real‐time model‐checking tools. Then, programs can be derived from the model by following a systematic approach. We introduce a timed automata to RTSJ Tool (TART), a prototype tool to automatically generate real‐time Java code from the model. Finally, we show the applicability of our approach by means of four examples: a gear controller, an audio/video protocol, a producer/consumer and the Fischer protocol. Copyright © 2011 John Wiley & Sons, Ltd.

Abstract Grammars are traditionally used to recognize or parse sentences in a language, but they can also be used to generate sentences. In grammar‐based test generation (GBTG), context‐free grammars are used to generate sentences that are interpreted as test cases. A generator reads a grammar G and generates L(G), the language accepted by the grammar. Often L(G) is so large that it is not practical to execute all of the generated cases. Therefore, GBTG tools support ‘tags’: extra‐grammatical annotations which restrict the generation. Since its introduction in the early 1970s, GBTG has become well established: proven on industrial projects and widely published in academic venues. Despite the demonstrated effectiveness, the tool support is uneven; some tools target specific domains, e.g. compiler testing, while others are proprietary. The tools can be difficult to use and the precise meaning of the tags are sometimes unclear. As a result, while many testing practitioners and researchers are aware of GBTG, few have detailed knowledge or experience. We present YouGen, a new GBTG tool supporting many of the tags provided by previous tools. In addition, YouGen incorporates covering‐array tags, which support a generalized form of pairwise testing. These tags add considerable power to GBTG tools and have been available only in limited form in previous GBTG tools. We provide semantics for the YouGen tags using parse trees and a new construct, generation trees. We illustrate YouGen with both simple examples and a number of industrial case studies. Copyright © 2010 John Wiley & Sons, Ltd.

In previous work, we have proposed a model based approach to developing real-time Java programs from timed automata. This approach allows us to verify the timed automata model mechanically by using current real-time model checking tools. Programs are then derived from the model by following a systematic approach. TART (timed automata to RTSJ Tool) is a prototype tool to support this approach. This paper presents TART, including its limitations, and discusses its application on four examples.

Reviewing many usability evaluation methods (UEMs) can be challenging and so selecting the appropriate UEM for a particular context, such as software process descriptions (SPDs), is difficult. Consequently, systematic analysis of UEMs based on a reusable set of evaluation criteria and strategies is essential. In this paper, we employ the feature analysis - screening mode of the DESMET methodology to analyse a range of UEMs and to suggest several UEM candidates for evaluating the usability of SPD.

Abstract The refinement calculus for logic programs is a framework for deriving logic programs from specifications. It is based on a wide-spectrum language that can express both specifications and code, and a refinement relation that models the notion of correct implementation. In this paper we extend and generalise earlier work on contextual refinement. Contextual refinement simplifies the refinement process by abstractly capturing the context of a subcomponent of a program, which typically includes information about the values of the free variables. This paper also extends and generalises module refinement. A module is a collection of procedures that operate on a common data type; module refinement between a specification module A and an implementation module C allows calls to the procedures of A to be systematically replaced with calls to the corresponding procedures of C. Based on the conditions for module refinement, we present a method for calculating an implementation module from a specification module. Both contextual and module refinement within the refinement calculus have been generalised from earlier work and the results are presented in a unified framework.

Numerous software verification and validation (V&V) techniques and tools exist to analyse requirements, designs and implementations of software systems. These V&V technologies range from relatively lightweight ones, such as inspection and testing, to more heavyweight technologies based on formal methods and theorem proving. For complex systems, a significant part of the cost and effort for development and maintenance is associated with V&V activities, and this almost always involves selecting and applying a mix of V&V technologies. Unfortunately, little is known about the cost-effectiveness of individual technologies or how to derive the most cost-effective combination. As such, combinations for particular projects are typically selected in an ad-hoc way. In this paper, several issues related to the selection and evaluation of combinations of V&V technologies are explored, based on personal experiences with the V&V of concurrent Java components. A V&V method is presented that was systematically derived through an analysis of the possible failures that can occur in concurrent Java components. This method combines inspection, static analysis, and dynamic testing. In addition, empirical methods that use analysis of fault data and experiments to evaluate V&V combinations are presented. Finally, ideas are presented for an iterative method that can be used to assist with both the selection and evaluation of cost-effective V&V combinations in a given context.

Abstract The Java programming language supports concurrency. Concurrent programs are harder to verify than their sequential counterparts due to their inherent non‐determinism and a number of specific concurrency problems, such as interference and deadlock. In previous work, we have developed the ConAn testing tool for the testing of concurrent Java components. ConAn has been found to be effective at testing a large number of components, but there are certain classes of failures that are hard to detect using ConAn. Although a variety of other verification tools and techniques have been proposed for the verification of concurrent software, they each have their strengths and weaknesses. In this paper, we propose a method for verifying concurrent Java components that includes ConAn and complements it with other static and dynamic verification tools and techniques. The proposal is based on an analysis of common concurrency problems and concurrency failures in Java components. As a starting point for determining the concurrency failures in Java components, a Petri‐net model of Java concurrency is used. By systematically analysing the model, we come up with a complete classification of concurrency failures. The classification and analysis are then used to determine suitable tools and techniques for detecting each of the failures. Finally, we propose to combine these tools and techniques into a method for verifying concurrent Java components. Copyright © 2006 John Wiley & Sons, Ltd.

Statistical testing involves the testing of software by selecting test cases from a probability distribution that is intended to represent the software's operational usage. In this paper, we describe and evaluate a framework for statistical testing of software components that incorporates test case execution and output evaluation. An operational profile and a test oracle are essential for the statistical testing of software components because they are used for test case generation and output evaluation respectively. An operational profile is a set of input events and their associated probabilities of occurrence expected in actual operation. A test oracle is a mechanism that is used to check the results of test cases. We present four types of operational profiles and three types of test oracles, and empirically evaluate them using the framework by applying them to two software components. The results show that while simple operational profiles may be effective for some components, more sophisticated profiles are needed for others. For the components that we tested, the fault-detecting effectiveness of the test oracles was similar.

Testing concurrent software is notoriously difficult due to problems with non-determinism and synchronisation. While tools and techniques for the testing of sequential components are well-understood and widely used, similar tools and techniques for concurrent components are not commonly available. This tutorial will look at the problems associated with testing concurrent components and propose techniques for dealing with these problems. The ConAn (Concurrency Analyser) testing tool supports these techniques for the testing of concurrent Java components and will be discussed and demonstrated in the tutorial. The limitations of the techniques and ConAn, as well as additional V&V tools and techniques to address these limitations will be presented.

A test oracle is a mechanism that is used during testing to determine whether a software component behaves correctly or not. The test oracle problem is widely acknowledged in the software testing literature and many methods for test oracle development have been proposed. Most of these methods use specifications or other resources to develop test oracles. A passive test oracle checks the behaviour of the component, but does not reproduce this behaviour. In this paper, we present a technique that develops passive test oracles for components using their APIs. This simple technique can be applied to any software component that is accessed through an API. In an initial experiment, we found that test oracles developed this way were more effective at finding faults with a relatively small number of test cases than test oracles developed from a formal specification and developed as a parallel implementation.

We describe the "STSC" prototype tool that supports the statistical testing of software components. The tool supports a wide range of operational profiles and test oracles for test case generation and output evaluation. The tool also generates appropriate values for different types of input parameters of operations. STSC automatically generates a test driver from an operational profile. This test driver invokes a test oracle that is implemented as a behaviour-checking version of the implementation. To evaluate the flexibility and usability of the tool, it has been applied to several case studies using different types of operational profiles and test oracles.

The testing of concurrent software components can be difficult due to the inherent non-determinism present in these components. For example, if the same test case is run multiple times, it may produce different results. This non-determinism may lead to problems with determining expected outputs. In this paper, we present and discuss several possible solutions to this problem in the context of testing concurrent Java components using the ConAn testing tool. We then present a recent extension to the tool that provides a general solution to this problem that is sufficient to deal with the level of non-determinism that we have encountered in testing over 20 components with ConAn.

While object-oriented programming offers great solutions for today's software developers, this success has created difficult problems in class documentation and testing. In Java, two tools provide assistance: Javadoc allows class interface documentation to be embedded as code comments and JUnit supports unit testing by providing assert constructs and a test framework. This paper describes JUnitDoc, an integration of Javadoc and JUnit, which provides better support for class documentation and testing. With JUnitDoc, test cases are embedded in Javadoc comments and used as both examples for documentation and test cases for quality assurance. JUnitDoc extracts the test cases for use in HTML files serving as class documentation and in JUnit drivers for class testing. To address the difficult problem of testing inheritance hierarchies, JUnitDoc provides a novel solution in the form of a parallel test hierarchy. A small controlled experiment compares the readability of JUnitDoc documentation to formal documentation written in Object-Z. Copyright © 2005 John Wiley & Sons, Ltd.

Achieving consistency between a specification and its implementation is an important part of software development In previous work, we have presented a method and tool support for testing a formal specification using animation and then verifying an implementation of that specification. The method is based on a testgraph, which provides a partial model of the application under test. The testgraph is used in combination with an animator to generate test sequences for testing the formal specification. The same testgraph is used during testing to execute those same sequences on the implementation and to ensure that the implementation conforms to the specification. So far, the method and its tool support have been applied to software components that can be accessed through an application programmer interface (API). In this paper, we use an industrially-based case study to discuss the problems associated with applying the method to a software system with a graphical user interface (GUI). In particular, the lack of a standardised interface, as well as controllability and observability problems, make it difficult to automate the testing of the implementation. The method can still be applied, but the amount of testing that can be carried on the implementation is limited by the manual effort involved.

An operational profile is a quantification of the expected use of a system. Determining an operational profile for software is a crucial and difficult part of software reliability assessment in general and it can be even more difficult for software components. This paper presents a systematic method for deriving an operational profile for software components. The method uses both actual usage data and intended usage assumptions to derive a usage structure, usage distribution and characteristics of parameters (including relationships between parameters). A usage structure represents the flow and interaction of operation calls. Statecharts are used to model the usage structures. A usage distribution represents probabilities of the operations. The method is illustrated on two Java classes but can be applied to any software component that is accessed through an application program interface (API).

Testing concurrent software is difficult due to problems with inherent nondeterminism. In previous work, we have presented a method and tool support for the testing of concurrent Java components. In this paper, we extend that work by presenting and discussing techniques for testing Java thread interrupts and timed waits. Testing thread interrupts is important because every Java component that calls wait must have code dealing with these interrupts. For a component that uses interrupts and timed waits to provide its basic functionality, the ability to test these features is clearly even more important. We discuss the application of the techniques and tool support to one such component, which is a nontrivial implementation of the readers-writers problem.

Summary form only given. The Java programming language supports concurrency. Concurrent programs are harder to verify than their sequential counterparts due to their inherent nondeterminism and a number of specific concurrency problems such as interference and deadlock. In previous work, we proposed a method for verifying concurrent Java components based on a mix of code inspection, static analysis tools, and the ConAn testing tool. The method was derived from an analysis of concurrency failures in Java components, but was not applied in practice. In this paper, we explore the method by applying it to an implementation of the well-known readers-writers problem and a number of mutants of that implementation. We only apply it to a single, well-known example, and so we do not attempt to draw any general conclusions about the applicability or effectiveness of the method. However, the exploration does point out several strengths and weaknesses in the method, which enable us to fine-tune the method before we carry out a more formal evaluation on other, more realistic components.

Software configuration management is the discipline of managing large collections of software development artefacts from which software products are built. Software configuration management tools typically deal with artefacts at fine levels of granularity - such as individual source code files - and assist with coordination of changes to such artefacts. This paper describes a lightweight tool, designed to be used on top of a traditional file-based configuration management system. The add-on tool support enables users to flexibly define new hierarchical views of product structure, independent of the underlying artefact-repository structure. The tool extracts configuration and change data with respect to the user-defined hierarchy, leading to improved visibility of how individual subsystems have changed. The approach yields a range of new capabilities for build managers, and verification and validation teams. The paper includes a description of our experience using the tool in an organization that builds large embedded software systems.

A test oracle provides a means for determining whether an implementation behaves according to its specification. A passive test oracle checks that the correct behaviour has been implemented, but does not implement the behaviour itself. In previous work, we have presented a method that allows us to derive passive C++ test oracles from formal specifications written in Object-Z. We describe the "Warlock" prototype tool that supports the method. Warlock is built on top of an existing Object-Z type checker and generates oracle code for a substantial subset of the Object-Z language. We describe the architecture of Warlock and its application to a number of Object-Z specifications. We also discuss its current limitations.

This paper describes an ongoing collaboration between Boeing Australia Limited and the University of Queensland to develop and deliver an introductory course on software engineering for Boeing Australia. The aim of the course is to provide a common understanding for all Boeing Australia's engineering staff of the nature of software engineering and the practices used throughout Boeing Australia. It is meant as an introductory course that can be presented to people with varying backgrounds, such as recent software engineering graduates, systems engineers, quality assurance personnel, etc. The paper describes the structure and content of the course, and the evaluation techniques used to collect feedback from the participants and the corresponding results. The course has been well-received by the participants, but the feedback from the course has indicated a need for more advanced courses in specific areas.

Achieving consistency between a specification and its implementation is an important part of software development. In this paper, we present a method for generating passive test oracles that act as self-checking implementations. The implementation is verified using an animation tool to check that the behavior of the implementation matches the behavior of the specification. We discuss how to integrate this method into a framework developed for systematically animating specifications, which means a tester can significantly reduce testing time and effort by reusing work products from the animation. One such work product is a testgraph: a directed graph that partially models the states and transitions of the specification. Testgraphs are used to generate sequences for animation, and during testing, to execute these same sequences on the implementation.

Formal specifications can precisely and unambiguously define the required behavior of a software system or component. However, formal specifications are complex artifacts that need to be verified to ensure that they are consistent, complete, and validated against the requirements. Specification testing or animation tools exist to assist with this by allowing the specifier to interpret or execute the specification. However, currently little is known about how to do this effectively.This article presents a framework and tool support for the systematic testing of formal, model-based specifications. Several important generic properties that should be satisfied by model-based specifications are first identified. Following the idea of mutation analysis, we then use variants or mutants of the specification to check that these properties are satisfied. The framework also allows the specifier to test application-specific properties. All properties are tested for a range of states that are defined by the tester in the form of a testgraph, which is a directed graph that partially models the states and transitions of the specification being tested. Tool support is provided for the generation of the mutants, for automatically traversing the testgraph and executing the test cases, and for reporting any errors. The framework is demonstrated on a small specification and its application to three larger specifications is discussed. Experience indicates that the framework can be used effectively to test small to medium-sized specifications and that it can reveal a significant number of problems in these specifications.

Concurrent programs are hard to test due to the inherent nondeterminism. This paper presents a method and tool support for testing concurrent Java components. Tool support is offered through ConAn (Concurrency Analyser), a tool for generating drivers for unit testing Java classes that are used in a multithreaded context. To obtain adequate controllability over the interactions between Java threads, the generated driver contains threads that are synchronized by a clock. The driver automatically executes the calls in the test sequence in the prescribed order and compares the outputs against the expected outputs specified in the test sequence. The method and tool are illustrated in detail on an asymmetric producer-consumer monitor. Their application to testing over 20 concurrent components, a number of which are sourced from industry and were found to contain faults, is presented and discussed.

When upgrading software in mission-critical or safety-related industrial control systems, it is imperative to ensure that system integrity properties are preserved. Comprehensive system testing is one way to gain this assurance. This has limitations, however, in that the hardware may be too expensive to assemble a large test rig, or where a product upgrade is to be deployed in diversely configured systems. This paper describes a method that uses rigorous system analysis to justify the replacement of system testing with both static analysis of the system configuration and dynamic testing of the upgraded system components. The paper reports on industrial experience in applying this method to the OptoNet product, which is an embedded software product used in industrial control systems. System analysis techniques are used to develop a detailed understanding of how OptoNet components (RTUs) interact to realise OptoNet system behaviour. Based on this detailed understanding, recommendations for a revised assurance strategy are made. The lessons learnt in the trial application of this method to the OptoNet product are discussed, and possible extensions to the method are proposed.

This paper describes a tool that manages a hierarchical, "is a subsystem of"-structure on a set of software development artefacts and that provides configuration management (CM) for subsystems by interacting with an existing CM tool. The tool is based on a recently proposed framework for subsystem-based configuration management. The tool demonstrates the feasibility of the framework and develops it further The design of the framework and the tool was developed in collaboration with Invensys SCADA Development and it is discussed in relation to their current software development process.

Abstract With the advent of object‐oriented languages and the portability of Java, the development and use of class libraries has become widespread. Effective class reuse depends on class reliability which in turn depends on thorough testing. This paper describes a class testing approach based on modeling each test case with a tuple and then generating large numbers of tuples to thoroughly cover an input space with many interesting combinations of values. The testing approach is supported by the Roast framework for the testing of Java classes. Roast provides automated tuple generation based on boundary values, unit operations that support driver standardization, and test case templates used for code generation. Roast produces thorough, compact test drivers with low development and maintenance cost. The framework and tool support are illustrated on a number of non‐trivial classes, including a graphical user interface policy manager. Quantitative results are presented to substantiate the practicality and effectiveness of the approach. Copyright © 2002 John Wiley & Sons, Ltd.

Existing refinement calculi provide frameworks for the stepwise development of imperative programs from specifications. This paper presents a refinement calculus for deriving logic programs. The calculus contains a wide-spectrum logic programming language, including executable constructs such as sequential conjunction, disjunction, and existential quantification, as well as specification constructs such as general predicates, assumptions and universal quantification. A declarative semantics is defined for this wide-spectrum language based on executions. Executions are partial functions from states to states, where a state is represented as a set of bindings. The semantics is used to define the meaning of programs and specifications, including parameters and recursion. To complete the calculus, a notion of correctness-preserving refinement over programs in the wide-spectrum language is defined and refinement laws for developing programs are introduced. The refinement calculus is illustrated using example derivations and prototype tool support is discussed.

This paper reports on an industrial pilot project that introduces systematic, automated module testing for embedded software in distributed, real-time, control systems. The systems are used in safety-related applications, are complex in nature, and hence have strong requirements for test coverage, auditability and repeatability. This paper explores issues of isolating modules from the run-time environment, improving integration of testing into the development environment, automating testing, and improving test planning and documentation. Metrics were gathered throughout the project that allow a coarse cost-benefit evaluation. Code coverage metrics for statement and branch coverage were also gathered using a commercial code coverage analysis tool. The testing exposed a number of latent faults within the software, and the overall results of the project show that module testing is feasible for this complex, embedded software.

The Java programming language supports monitors. Monitor implementations, like other concurrent programs, are hard to test due to the inherent non-determinism. This paper presents the ConAn (Concurrency Analyser) tool for generating drivers for the testing of Java monitors. To obtain adequate controllability over the interactions between Java threads, the generated driver contains processes that are synchronized by a clock. The driver automatically executes the calls in the test sequence in the prescribed order and compares the outputs against the expected outputs specified in the test sequence. The method and tool are illustrated on an asymmetric producer-consumer monitor and their application to two other monitors is discussed.

This paper describes a method for specification-based class testing that incorporates test case generation, execution, and evaluation based on formal specifications. This work builds on previous achievements in the areas of specification-based testing and class testing by integrating the two within a single framework. The initial step of the method is to generate test templates for individual operations from a specification written in the Object-Z specification language. These test templates are combined to produce a finite state machine for the class that is used as the basis for test case execution using the ClassBench test execution framework. An oracle derived from the Object-Z specification is used to evaluate the outputs. The method is explained using a simple example and its application to a more substantial case study is also discussed. Keywords: specification-based testing, Object-Z, class testing, ClassBench, oracles 1

Structural coverage approaches to software testing are mature, having been thoroughly studied for decades. Significant tool support, in the form of instrumentation for statement or branch coverage, is available in commercial compilers. While structural coverage is sensitive to which code structures are covered, it is insensitive to the values of the variables when those structures are executed. Data coverage approaches, e.g. boundary value coverage, are far less mature. They are known to practitioners mostly as a few useful heuristics with very little support for automation. Because of its sensitivity to variable values, data coverage has significant potential, especially when used in combination with structural coverage. This paper generalizes the traditional notion of boundary coverage, and formalizes it with two new data coverage measures. These measures are used to generate test cases automatically and from these, sophisticated test suites for functions from the C++ Standard Template Library. Finally, the test suites are evaluated with respect to both structural coverage and discovery of seeded faults. Copyright © 1999 John Wiley & Sons, Ltd.

In this paper, we present an approach that combines specification-based testing and class testing. In particular, we provide a method for generating Finite State Machines (FSMs) from formal, object-oriented specifications, and use the ClassBench testing framework to build a test suite from those formally generated FSMs. We briefly outline our approach and focus on one step in the approach; the transformation of the formally derived FSM into a ClassBench testgraph, which is used by ClassBench to drive the test execution. We illustrate the method with a simple bounded queue class, and discuss the application of the method to a larger example, which is a simplified model of a process scheduling system.

The paper contains a case study demonstrating a complete process for specification based class testing. The process starts with an abstract specification written in Object-Z and concludes by exercising an implementation with test cases and evaluating the results. The test cases are derived using the Test Template Framework for each individual operation. They are analysed to generate a finite state machine that can execute test sequences within the ClassBench framework. An oracle is also derived from the Object-Z specification. The case study demonstrates how a formal specification contributes to the development of practical tests that can be executed by a testing tool. It also shows how a test oracle can be derived from a specification and used by the same testing tool to evaluate test results.

A test oracle provides a means for determining whether an implementation functions according to its specification. A passive test oracle checks the behaviour of the implementation, but does not attempt to reproduce this behaviour. The paper describes the translation of formal specifications of container classes to passive test oracles. Specifically, we use Object-Z for specifications and C++ for oracles. We discuss several practical issues for the use of formal specifications in test oracle generation. We then present the translation process and illustrate it with an example based on an integer set class. Our approach is illustrated with an example based on an integer set class.

Presents an overview of the Possum specification animation system and its integration into the Cogito methodology and toolset. Possum allows interpretation (or animation) of specifications written in Sum, the specification language of Cogito. We distinguish two potential uses for Possum and illustrate each of these with an example. The first is the use of Possum for specification verification, where the analysis of properties of specifications by the specification designer is emphasised. The second use is specification validation, where the specification is checked against the informal requirements of the system.

We describe part of an STL conformance test suite under development. Test suites for all of the STL containers have been written, demonstrating the feasibility of thorough and highly automated testing of industrial component libraries. We describe affordable test suites that provide good code and boundary value coverage, including the thousands of cases that naturally occur from combinations of boundary values. We show how two simple oracles can provide fully automated output checking for all the containers. We refine the traditional categories of black-box and white-box testing to specification-based, implementation-based and implementation-dependent testing, and show how these three categories highlight the key cost/thoroughness trade-offs.

We present an overview of the Possum specification animation system, an addition to the Cogito methodology and toolset. Possum allows interpretation (or animation) of specifications written in SUM, which is the specification language used in Cogito. We give an account of the functionality of Possum, illustrated by some simple examples, and describe the way in which Possum is used in a typical Cogito development. The current capabilities and limitations of Possum are reviewed from a technical perspective and an overview of other systems that support the animation of formal specification languages is presented.

This paper describes the translation of Object-Z specifications of container classes to C++ test oracle classes. It presents a three-stage translation process and describes how the derived test oracles are integrated into the ClassBench testing framework. The method caters for object-oriented features such as inheritance and aggregation. Translation issues and the limitations of the method are discussed. Our approach is illustrated with an example based on an integer set class.

In contrast to the explosion of activity in object-oriented design and programming, little attention has been given to object testing. We present a novel approach to automated testing designed especially for collection classes. In the ClassBench methodology, a testgraph partially models the states and transitions of the Class-Under-Test (CUT) state/transition graph. To determine the expected behavior for the test cases generated from the testgraph, the tester develops an oracle class, providing essentially the same operations as the CUT but supporting only the testgraph states and transitions. Surprisingly thorough testing is achievable with simple testgraphs and oracles. The ClassBench framework supports the tester by providing a testgraph editor, automated testgraph traversal, and a variety of utility classes. Test suites can be easily configured for regression testing–where many test cases are run–and debugging–where a few test cases are selected to isolate the bug. We present the ClassBench methodology and framework in detail, illustrated on both simple examples and on test suites from commercial collection class libraries. © 1997 John Wiley & Sons, Ltd.

article Free Access Share on State abstraction and modular software development Authors: Daniel Hoffman University of Victoria, Department of Computer Science, P.O. Box 3055, Victoria, B. C., V8W 3P6 Canada University of Victoria, Department of Computer Science, P.O. Box 3055, Victoria, B. C., V8W 3P6 CanadaView Profile , Paul Strooper University of Queensland, Department of Computer Science, St. Lucia, Qld. 4072, Australia University of Queensland, Department of Computer Science, St. Lucia, Qld. 4072, AustraliaView Profile Authors Info & Claims ACM SIGSOFT Software Engineering NotesVolume 20Issue 4Oct. 1995 pp 53–61https://doi.org/10.1145/222132.222139Published:01 October 1995Publication History 4citation1,735DownloadsMetricsTotal Citations4Total Downloads1,735Last 12 Months13Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

In contrast to the explosion of activity in object-oriented design and programming, little attention has been given to object testing. In our approach, a driver class and an oracle class are developed for each class-under-test (CUT). The driver class is based on a test-graph which partially models the CUT as a state machine, but with vastly fewer states and transitions. The oracle class provides essentially the same operations as the CUT, but supports only the testgraph states and transitions. Surprisingly thorough testing is achievable with simple testgraphs and oracles. The key is designing the two together, to avoid tests for which input generation and output checking are unaffordable. We summarize recent experience with the testgraphs framework, including handling of operations on pairs of objects and the application of testgraphs to test part of the testgraphs implementation.>

Tools and techniques for writing scripts in Prolog that automatically test modules implemented in C are presented. Both the input generation and the test oracle problems are addressed, focusing on a balance between the adequacy of the test inputs and the cost of developing the output oracle. The authors investigate automated input generation according to functional testing, random testing, and a novel approach based on trace invariants. For each input generation scheme, a mechanism for generating the expected outputs has been developed. The methods are described and illustrated in detail. Script development and maintenance costs appear to be reasonable, and run-time performance appears to be acceptable.>