A challenging problem in software testing is finding the optimal point at which costs justify the stop-test decision. We first present an economic model that can be used to evaluate the consequences of various stop-test decisions. We then discuss two approaches for assessing performance, automated load test generation in the context of empirical testing and performance modeling, and illustrate how these techniques can affect the stop-test decision. We then illustrate the application of these two techniques to evaluating the performance of Web servers that performs significant server-side processing through object-oriented (OO) computing. Implications of our work for Web server performance evaluation in general are discussed.

Recently, a class of experimental designs has been devised that guarantee input domain coverage up to all combinations of k test factors taken t at a time.With such designs, all pairwise combinations (or triplets or quadruplets, etc.) are selected at least once.To evaluate their applicability to software testing, we analyzed the extent to which software coverage (i.e., code execution) achieved by these designs for t=1, ... ,k is representative of that achieved by exhaustively testing all factor combinations.The block coverage obtained for t;5;2 was comparable with that achieved by exhaustively testing all factor combinations but higher-order values of t were required for path coverage.Implications of these results for software testing are discussed.

Operational testing is expensive and time-consuming in applications with highly critical but infrequent event occurrences, and procedures are needed for obtaining valid software reliability estimates. Two such procedures are accelerated stress testing and experimental design techniques. Examples of accelerated stress in software include: a higher request arrival rate, increasing data traffic "burstiness", and so on. Furthermore, when the software is tested by executing the system under emergency conditions or exception events, it is critical that experimental design techniques are used to account for "carry-over" effects in failure performance. An application of software reliability testing of a telecommunication system node controller is used to describe the issues and our overall approach to software reliability assessment. The reliability projections based on our approach are compared to reliability estimates derived analytically and through simulation. Our results suggest that these statistical techniques are useful techniques for measuring reliability performance under critical events that have a low probability of occurrence.

In this paper, we present a generic approach for quantitatively evaluating the performance implications of a software architecture using the concept of operational profile from software reliability engineering. In our approach, the System Under Development (SUD) modes of operation are analyzed to specify a mode's rate of arrivals of external events. The requests' process flow (i.e., the rate of arrival of internal requests at server processes) is then derived from the software architecture. Finally, the product of the internal arrival rates and the amount of server resources to service a request is used to calculate a mode's server resource utilization. Application of this technique to an actual AT&T telecommunications operations system, together with implications for engineering software architectures that are robust under variation in modes of operation are discussed.

This panel session will discuss how Systems Engineers and Software Architects can benefit porn using SRE. Members of the panel will discuss how their respective organizations use (or anticipate using) SRE earlier in the software development life-cycle. The intent is to stimulate and encourage those involved in Systems Engineering and Software Architecture to adopt and use SRE in their work.

Describes our practical experiences in applying software reliability engineering (SRE) technology to telecommunications network operations systems. These are real-time, high-availability systems used to provision, maintain and restore AT&T's backbone telecommunications transport network. In this paper, we describe how we used SRE to ensure the quality and high reliability of three such systems prior to production. Key steps in applying SRE are: deriving an operational profile, developing a simulator test environment, and executing load, stress and stability tests. To reduce test intervals and increase system reliability, we apply accelerated testing and experimental design. In addition, our SRE program also incorporates customer-focused techniques such as operational scenario testing.

Software systems typically contain design and code defects that manifest themselves as software failures at various points during program execution. These software faults can be viewed as causing failures to occur according to some chance mechanism that is often taken to be a Poisson process. Individual per-fault failure sequences (resulting from delayed fault detection and software correction) during test execution were analyzed to determine whether they satisfied Poisson process assumptions. The system analyzed, a large, complex, industrial software system, was executed under a defined operational profile corresponding to a controlled command and autonomous event mix characteristic of the system's actual usage. The failure events, defined as severe, system-wide affecting events attributed to software faults, typically occurred following many cycles of interacting system features in the expected user mode. Execution times between failure events were calculated and then statistically analyzed for conformance to Poisson process assumptions. The results indicated that first-time failure events conformed to the assumptions of a Poisson process and were consistent with a reliability growth Poisson model but that per-fault failure sequences attributed to specific faults tended to violate Poisson assumptions due to failure clustering.>