Many empirical studies have been performed to evaluate software testing methods in past decades. However, we are still not able to generalize the results as most studies are not complete and differ significantly from one another. To contribute to the existing knowledge base of software testing methods, we performed an empirical study to evaluate 3 testing methods: (1) code reading by stepwise abstraction, (2) functional testing using equivalence partitioning and boundary value analysis, and (3) structural testing using 100% branch, multiple-condition, loop, and relational-operator coverage using a well-defined and standard schema. A controlled experiment is performed with 18 subjects who applied the 3 defect detection techniques to 3 C programs in a fractional factorial experimental design to observe failures and isolate faults. The experimental results show that (1) the techniques are equally effective in terms of observing failures and finding faults, (2) the effectiveness of the techniques depends on the nature of the program, (3) all the testing techniques are equally efficient in case of failure observation, (4) the techniques are different in their efficiency in terms of fault isolation where code reading performed better than that of structural and functional testing, and (5) with respect to the fault types, all the techniques were equally effective in observing failures and isolating faults except in case of cosmetic faults where functional testing performed better than the other 2 techniques The effectiveness and efficiency of testing techniques were significantly influenced by the type of program. The results presented in this paper contribute to an empirical knowledge base of testing methods and may be helpful for the software engineers to decide the appropriate techniques in improving the software testing process.

Context: Many empirical studies have been carried out to evaluate software testing methods in the past decades. However, we are still not able to generalize the results, as most studies are not complete and differ significantly from one another. Objective: To contribute to the existing knowledge base of testing techniques by evaluating three software testing methods a) code reading by stepwise abstraction, b) functional testing using equivalence partitioning and boundary value analysis, and c) structural testing using 100% branch, multiple-condition, loop, and relational-operator coverage using a well-defined and standard schema. Method: A controlled experiment was carried out with eighteen subjects who applied three techniques to three C programs in a fractional factorial experimental design. Results: There is no difference in techniques in terms of failure observation and fault finding effectiveness and the effectiveness depends on the program rather than technique. In case of failure observation efficiency, all the techniques performed equally; however, in terms of fault isolation efficiency, code reading performed better than structural testing which in turn out performed functional testing. With respect to the fault types, all techniques performed equally except in case of cosmetic faults where functional testing performed better than other two testing methods. Conclusion: We conclude that all techniques are all equivalent in terms of effectiveness. However, the techniques differ partially in terms of efficiency. The effect of the program was significant in almost all cases. We need to build some standardized and better laboratory packages which should represent actual software engineering practices. Carry out experiments on such packages will help in deriving realistic results.

In this paper, we describe a controlled experiment carried out to compare three software testing methods: code reading, functional testing and structural testing. The experiment was carried out with eighteen subjects who applied three techniques to three C programs in a fractional factorial experimental design. The main results of the study are that all testing techniques are equivalent in terms of effectiveness; however the techniques differ partially in terms of efficiency.