A new measure of performance, which uses both application integrity and traditional network response time, is proposed. Modern networked application services rely on a stack of network protocols and a host of other services many of which cross-organizational and corporate boundaries. We point out that traditional software quality assurance techniques don't scale up for post-deployment integrity checks for such applications and services. A new methodology to do non-stop post-production monitoring of networked application services for transactional integrity and time delay measurement is proposed. Specifically we describe the Telcordia/spl trade/ Application Assurance System, which we have created for measuring real-time performance of web-based applications used in commercial settings. The system measures both post-production application integrity and time delay. The measurements are carried out by sending synthetic end-user transactions and analyzing the responses. Statistical models for analyzing the data using single monitoring site as well as multiple monitoring sites are described. Creating synthetic end-user transactions is crucial for our method. The paper presents a method for generation of 'highly efficient' end-user transactions from a graphical model of the functionality of the system. Highly efficient transactions are generated using combinatorial designs. The graphical model is incrementally created using a recorder. We give several empirical examples of efficacy of this system and uses for finding performance problems.

Reports on an ongoing statistical analysis of actual CCSN traffic data. The data consist of approximately 170 million signaling messages collected from a variety of different working CCS subnetworks. The key findings from the analysis concern: 1) the characteristics of both the telephone call arrival process and the signaling message arrival process; 2) the tail behavior of the call holding time distribution; and 3) the observed performance of the CCSN with respect to a variety of performance and reliability measurements.>

Developers of large software systems must decide how long software should be tested before releasing it. A common and usually unwarranted assumption is that the code remains frozen during testing. We present a stochastic and economic framework to deal with systems that change as they are tested. The changes can occur because of the delivery of software as it is developed, the way software is tested, the addition of fixes, and so on. Specifically, we report the details of a real time trial of a large software system that had a substantial amount of code added during testing. We describe the methodology, give all of the relevant details, and discuss the results obtained. We pay particular attention to graphical methods that are easy to understand, and that provide effective summaries of the testing process. Some of the plots found useful by the software testers include: the Net Benefit Plot, which gives a running chart of the benefit; the Stopping Plot, which estimates the amount of additional time needed for testing; and diagnostic plots. To encourage other researchers to try out different models, all of the relevant data are provided.>