Abstract Spreadsheets are very common for information processing to support decision making by both professional developers and non-technical end users. Moreover, business intelligence and artificial intelligence are increasingly popular in the industry nowadays, where spreadsheets have been used as, or integrated into, intelligent or expert systems in various application domains. However, it has been repeatedly reported that faults often exist in operational spreadsheets, which could severely compromise the quality of conclusions and decisions based on the spreadsheets. With a view to systematically examining this problem via survey of existing work, we have conducted a comprehensive literature review on the quality issues and related techniques of spreadsheets over a 35.5-year period (from January 1987 to June 2022) for target journals and a 10.5-year period (from January 2012 to June 2022) for target conferences. Among other findings, two major ones are: (a) Spreadsheet quality is best addressed throughout the whole spreadsheet life cycle, rather than just focusing on a few specific stages of the life cycle. (b) Relatively more studies focus on spreadsheet testing and debugging (related to fault detection and removal) when compared with spreadsheet specification, modeling, and design (related to development). As prevention is better than cure, more research should be performed on the early stages of the spreadsheet life cycle. Enlightened by our comprehensive review, we have identified the major research gaps as well as highlighted key research directions for future work in the area.

This paper extends the choce relation framework, abbreviated as choc'late, which assists software testers in the application of category/choice methods to testing. choc'late assumes that the tester is able to construct a single choice relation table from the entire specification; this table then forms the basis for test case generation using the associated algorithms. This assumption, however, may not hold true when the specification is complex and contains many specification components. For such a specification, the tester may construct a preliminary choice relation table from each specification component, and then consolidate all the preliminary tables into a final table to be processed by choc'late for test case generation. However, it is often difficult to merge these preliminary tables because such merging may give rise to inconsistencies among choice relations or overlaps among choices. To alleviate this problem, we introduce a DividE-and-conquer methodology for identifying categorieS, choiceS, and choicE Relations for Test case generation, abbreviated as dessert. The theoretical framework and the associated algorithms are discussed. To demonstrate the viability and effectiveness of our methodology, we describe case studies using the specifications of three real-life commercial software systems.

An early step for most black-box testing methods is to identify a set of categories and choices (or their equivalents) from the specification. The identification is often performed in an ad hoc manner, thus the quality of categories and choices is in doubt. Poorly identified categories and choices will affect the comprehensiveness of test cases. In this paper, we describe several comparative studies using three commercial specifications and discuss the major results. The objectives of our studies are (a) to investigate the differences in the types and amounts of mistakes made between inexperienced and experienced software testers in an ad hoc identification approach and (b) to determine the extent of mistake reduction after discussing the mistakes with the software testers and providing them with an identification checklist.

Various black-box methods for the generation of test cases have been proposed in the literature. Many of these methods, including the category-partition method and the classification-tree method, follow the approach of partition testing, in which the input domain is partitioned into subdomains according to important aspects of the specification, and test cases are then derived from the subdomains. Though comprehensive in terms of these important aspects, execution of all the test cases so generated may not be feasible under the constraint of tight testing resources. In such circumstances, there is a need to select a smaller subset of test cases from the original test suite for execution. In this paper, we propose the use of white-box information to guide the selection of test cases from the original test suite generated by a black-box testing method. Furthermore, we have developed some techniques and algorithms to facilitate the implementation of our approach, and demonstrated its viability and benefits by means of a case study.