We present experience with improving the requirements engineering process for a telecommunications electronic clearinghouse that connects multiple trading partners. Software failures and faults were traced back to problems in the semi-structured requirements, which were written as message specifications (called "business rules") in a nonstandardized format using MS Word. To remedy these problems, we first designed a highly structured representation of the. essential information in Extensible Markup Language (XML), supported by XML Schema for precisely defining the requirements structure. Second, we developed XLEdit-a custom GUI for editing the XML documents in a friendly way while preserving their information structure. Third, to assist the requirements engineers with automated migration of the existing business rules, we developed a converter for translating from MS Word to the target XML format. The investment in the technology for structured requirements editing and validation is expected to increase customer satisfaction and dramatically improve the efficiency of the clearinghouse service.

Abstract We present experience gained in automating tests for an operations support system. A major portion of the effort was devoted to extending a commercial test tool so that testers could easily manipulate graphical user interface (GUI) objects on two implementations of the application. For this purpose, we developed a test automation library as support infrastructure for writing tests. The challenges and tradeoffs are discussed such as simplicity/complexity for a tester versus a library developer, hiding/exposing window hierarchy to the tester, providing common methods for different types of GUI objects, transparently manipulating custom GUI widgets, and coping with data‐dependent test cases. We discuss the requirements of test code reusability, maintainability, and portability, and describe the solutions we found. In addition, we offer observations about benefits and pitfalls of test automation, our recommendations for maximizing return on investment, and results from automating a variety of tests. Copyright © 2002 John Wiley & Sons, Ltd.

Model-based testing is a new and evolving technique for generating a suite of test cases from requirements.Testers using this approach concentrate on a data model and generation infrastructure instead of hand-crafting individual tests.Several relatively small studies have demonstrated how combinatorial test generation techniques allow testers to achieve broad coverage of the input domain with a small number of tests.We have conducted several relatively large projects in which we applied these techniques to systems with millions of lines of code.Given the complexity of testing, the modelbased testing approach was used in conjunction with test automation harnesses.Since no large empirical study has been conducted to measure efficacy of this new approach, we report on our experience with developing tools and methods in support of model-based testing.The four case studies presented here offer details and results of applying combinatorial test-generation techniques on a large scale to diverse applications.Based on the four projects, we offer our insights into what works in practice and our thoughts about obstacles to transferring this technology into testing organizations.

The paradigm of model based testing shifts the focus of testing from writing individual test cases to developing a model from which a test suite can be generated automatically. We report on our experience with model based testing of a highly programmable system that implements intelligent telephony services in the US telephone network. Our approach used automatic test case generation technology to develop sets of self checking test cases based on a machine readable specification of the messages in the protocol under test. The AETG/sup TM/ software system selected a minimal number of test data tuples that covered pairwise combinations of tuple elements. We found the combinatorial approach of covering pairwise interactions between input fields to be highly effective. Our tests revealed failures that would have been difficult to detect using traditional test designs. However, transferring this technology to the testing organization was difficult. Automatic generation of cases represents a significant departure from conventional testing practice due to the large number of tests and the amount of software development involved.

The use of empirical data to understand and improve software products and software engineering processes is gaining ever increasing attention. Empirical data from products and processes is needed to help an organization understand and improve its way of doing business in the software domain. Additional motivation for collecting and using data is provided by the need to conform to guidelines and standards which mandate measurement, specifically the SEI’s Capability Maturity Model and ISO 9000–3. Some software engineering environments (SEEs) offer automated support for collecting and, in a few cases, using empirical data. Measurement will clearly play a significant role in future SEEs. The paper surveys the trend towards supporting measurement in SEEs and gives details about several existing research and commercial software systems.