Web applications provide essential and ubiquitous services across diverse application domains. Online reliability prediction forecasts the probability of a request being successfully processed within a given timeframe, providing valuable information for users and engineers. Many approaches have been proposed for software reliability modeling. However, conventional methods tend to rely heavily on historical data for assessing software reliability, often overlooking the dynamic nature of web applications and their implications for reliability predictions.This paper introduces AdaRel, an online reliability prediction system designed to adapt to the dynamic behavior of web applications, thereby enhancing prediction accuracy. AdaRel employs a suite of prediction models and periodically evaluates and selects the best-performing model to forecast reliability for each observation period. Additionally, it monitors the system’s behavior and strategically mitigates the adverse impacts on prediction accuracy when detecting an anomaly.In three case studies, AdaRel’s predictive accuracy consistently surpassed that of the individual algorithms. The results confirm that AdaRel’s performance is robust and dependable, irrespective of the distinctive attributes of the web applications it assesses.

Most modern software systems are continuously evolving, with changes frequently taking place in the core components or the execution context. These changes can adversely introduce regression faults, causing previously working functions to fail. Regression testing is essential for maintaining the quality of evolving complex software, but it can be overly time-consuming when the size of the test suite is large, or the execution of the test cases takes a long time. There are extensive research studies on selective regression testing aiming at minimizing the size of the regression test suite while maximizing the detection of the regression faults. However, most of the existing techniques focus on the regression faults caused by the code changes, the impact of the context changes on the non-modified software has barely been explored. This paper presents a context-aware regression test selection (CARTS) approach that not only accounts for the modification of code but also changes in the execution context, including libraries, external APIs, and databases. After a change, CARTS uses the program invariants denoted in the pre- and postconditions of a function to determine if the function is affected by the change and selects all the test cases that executed the modified code as well as the non-modified functions whose preconditions are affected by the change. To evaluate the effectiveness of our approach, we conducted empirical studies on multi-release open-source software and case studies on real-world systems that have ongoing changes in code as well as in the execution context. The results of our controlled experiments show that with an average of 32.5% of the regression test cases, CARTS selected all the fault-revealing test cases. In the case studies, all the fault-revealing test cases were selected by using an average of 25.3% of the regression test suite. These results suggest that CARTS can be effective for selecting fault-revealing test cases for both code and execution context changes.

While automatic online software anomaly detection is crucial for ensuring the quality of production software, current techniques are mostly inefficient and ineffective. For online software, its inputs are usually provided by the users at runtime and the validity of the outputs cannot be automatically verified without a predefined oracle. Furthermore, some online anomalous behavior may be caused by the anomalies in the execution context, rather than by any code defect, which are even more difficult to detect. Existing approaches tackle this problem by identifying certain properties observed from the executions of the software during a training process and using them to monitor online software behavior. However, they may require a large execution overhead for monitoring the properties, which limits the applicability of these approaches for online monitoring. We present a methodology that applies effective algorithms to select a close to optimal set of anomaly-revealing properties, which enables online anomaly detection with minimal execution overhead. Our empirical results show that an average of 76.5% of anomalies were detected by using at most 5.5% of execution overhead.

Reliability is an essential quality requirement for web services. Existing techniques for measuring reliability of web services mainly focus on failures caused by code-based defects. For data-centric web services, the reliability of services can be significantly affected by the quality of data used to provide services. However, the impact of data quality on the reliability of web services has rarely been explored. We present an approach to estimate data quality and to incorporate data quality with the reliability of software components in the reliability estimation of web services. To demonstrate the proposed approach, we present a case study on a government web service. In this study we observed that more than 60% of service failures reported over a three-month period were caused by invalid data. The results show that by taking into account data quality, the reliability estimate of the web service is more accurate than the traditional reliability measurement.

Reliability is an essential quality requirement for service-oriented systems. A number of models have been developed for predicting reliability of traditional software, in which code-based defects are the main concern for the causes of failures. Service-oriented software, however, shares many common characteristics with distributed systems and web applications. In addition to residual defects, the reliabilities of these types of systems can be affected by their execution context, message transmission media, and their usages. We present a case study to demonstrate that the reliability of a service varies on an hourly basis, and reliability forecasts should be recalibrated accordingly. In this study, the failure behavior of a required external service, used by a provided service, was monitored for two months to compute the initial estimates, which then continuously re-computed based on the learning of the new failure patterns. These reliabilities are integrated with the reliability of the component in the provided service. The results show that with this progressive re-calibration we provide more accurate reliability forecasts for the service.

Web applications have become major driving forces for world business. Effective and efficient testing of evolving Web applications is essential for providing reliable services. In this paper, we present a user session based testing technique that clusters user sessions based on the service profile and selects a set of representative user sessions from each cluster. Then each selected user session is tailored by augmentation with additional requests to cover the dependence relationships between Web pages. The created test suite not only can significantly reduce the size of the collected user sessions, but is also viable to exercise fault sensitive paths. We conducted two empirical studies to investigate the effectiveness of our approach- one was in a controlled environment using seeded faults, and the other was conducted on an industrial system with real faults. The results demonstrate that our approach consistently detected the majority of the known faults by using a relatively small number of test cases in both studies.

Large-scale multi-tier web applications are inherently dynamic, complex, heterogeneous and constantly evolving. Maintaining such applications is important yet inevitably expensive. First, the size of the test suite of an evolving system will be continuously growing. Second, to ensure that the changes will not affect the quality of the systems, regression testing is frequently performed. To effectively and efficiently maintain web applications after each change, obsolete test cases must be removed and regression testing should selectively re-test. To this end there is a need for an inter-tier change impact analysis, which requires a coherent model rendering inter-tier dependence information. We present a technique that makes use of an integrated inter-connection dependence model to analyze cross-tier change impacts. These are then used to select affected test cases for regression testing and to identify repairable test cases for reuse, or to discard obsolete non-repairable test cases. Our empirical study shows that with this technique, the maintenance cost of the target system can be significantly reduced.

We present a dependence model based on which an efficient slicing algorithm is applied to compute forward slices for impact analysis and for regression test selection. The dependence model not only captures control and data dependence relationships, but also depicts dependence between components and their execution contexts. The model is constructed in a hierarchical fashion, from inter-statement to inter-component level, where at each level detailed dependence relationships are analyzed, then summarized information is published for the next level of composition. With this model, our slicing algorithm identifies all the affected elements that can be utilized to effectively select regression test cases. The analysis shows that our approach is more efficient than the existing approaches in terms of the execution time and the space requirements. And the promising results of regression testing obtained from the case study demonstrate the great potential of our technique in effective maintenance of complex component-based systems.

Parallel I/O systems frequently use collective I/O to improve the performance of applications that have small and noncontiguous data access patterns. The performance of a collective I/O implementation depends on how well its I/O libraries and parallel file system cooperate. We present a novel server directed file domain allocation (SDFDA) approach motivated by a study of interaction between MPICH/ROMIO and PVFS for noncontiguous file access. In our SDFDA approach, the file domains are bound to processes, according to the data layout scheme of I/O servers, as well as the number and placement of the ROMIO aggregators. We conducted an empirical study on an MPICH/PVFS cluster platform and compared the performance of our SDFDA approach with the existing ROMIO allocation approach. The results show that by providing a light correlation between MPICH/ROMIO and PVFS, our approach conforms I/O distribution to PVFS file striping and achieves higher performance improvement for noncontiguous file access.

A number of Markov-based software reliability models have been developed for measuring software reliability. However, the application of these models is strictly limited to software that satisfies the Markov properties. The objective of our work is to expand the application domain of the Markov-based models, so that most software can be modeled and software reliability can be measured at the architecture level. To overcome the limitations of Markov properties, our model takes execution history into account and addresses both deterministic and probabilistic software behaviors. Each state represents the executions of one or more components depending on the architectural styles. In addition, the executions of one component are depicted by using distinctive states, when such executions are influenced by past states. Furthermore, we construct loops to eliminate the likelihood of unlimited state expansion and utilize a binomial tree structure to account for all the different execution paths. We show that Markov models are applicable even to software that does not fully satisfy the Markov properties. Therefore, we significantly improve the state of the art in architecture-based software reliability modeling.

Component-based software engineering is increasingly being adopted for software development. Although much work has been proposed for building component-based-systems, techniques for testing component-based systems have not been well developed. We present a test model that depicts a generic infrastructure of component-based systems and suggests key test elements. The test model is realized using a component interaction graph (CIG) in which the interactions and the dependence relationships among components are illustrated. By utilizing the CIG, we propose a family of test adequacy criteria which allow optimization of the balancing among budget, schedule, and quality requirements typically necessary in software development. The methodology proposed is efficient and effective, as demonstrated by promising results obtained from a case study.

Existing software reliability-growth models often over-estimate the reliability of a given program. Empirical studies suggest that the over-estimations exist because the models do not account for the nature of the testing. Every testing technique has a limit to its ability to reveal faults in a given system. Thus, as testing continues in its region of saturation, no more faults are discovered and inaccurate reliability-growth phenomena are predicted from the models. This paper presents a technique intended to solve this problem, using both time and code coverage measures for the prediction of software failures in operation. Coverage information collected during testing is used only to consider the effective portion of the test data. Execution time between test cases, which neither increases code coverage nor causes a failure, is reduced by a parameterized factor. Experiments were conducted to evaluate this technique, on a program created in a simulated environment with simulated faults, and on two industrial systems that contained tenths of ordinary faults. Two well-known reliability models, Goel-Okumoto and Musa-Okumoto, were applied to both the raw data and to the data adjusted using this technique. Results show that over-estimation of reliability is properly corrected in the cases studied. This new approach has potential, not only to achieve more accurate applications of software reliability models, but to reveal effective ways of conducting software testing.

Component based software engineering has been increasingly adopted for software development. Such an approach using reusable components as the building blocks for constructing software, on one hand, embellishes the likelihood of improving software quality and productivity; on the other hand, it consequently involves frequent maintenance activities, such as upgrading third party components or adding new features. The cost of maintenance for conventional software can account for as much as two-thirds of the total cost, and it can likely be even more for maintaining component based software. Thus, an effective maintenance technique for component based software is strongly desired. The authors present a technique that can be applied on various maintenance activities over component based software systems. The technique proposed utilizes a static analysis to identify the interfaces, events and dependence relationships that would be affected by the modification in the maintenance activity. The results obtained from the static analysis along with the information of component interactions recorded during the execution of each test case are used to guide test selection in the maintenance phase. The empirical results show that with 19% effort our technique detected 71% of the faults in an industrial component based system, which demonstrates the great potential effectiveness of the technique.

Presents a case study that evaluates two software quality attributes: performance and availability. We use three programs based on two architectural styles: pipe-filter and batch-sequential. The objective of this study is to identify the crucial factors that might have an influence on these quality attributes from the software architecture perspective. The benefit of this study is that early quality prediction can be facilitated by an analysis of the software architecture. The results from this study show that it is feasible to select a better architectural style based on variations in the execution environment to attain higher availability and/or better performance. Moreover, we demonstrate the effects of these variations on the quality measurements.

Traditional testing techniques often overlook object-oriented faults that are either caused by inheritance and/or polymorphism features or are introduced in object management. We present an object-flow based testing strategy that utilizes two object-flow coverage criteria (all-bindings and all-du-pairs) in testing object-oriented programs. The all-bindings criterion takes inheritance and polymorphism features into account to ensure that every binding of every object is exercised under some test. The all-du-pairs criterion is applied to monitor the behavior of every object during its lifetime by keeping track of where the object is defined (d) and where such a definition is referenced or used (u). These object-flow coverage criteria can be used to develop test cases that are able to trigger object-oriented faults. Furthermore, an integrated approach that incorporates the object-flow based testing strategy with traditional testing techniques as well as state-based testing technique is introduced. The results of our empirical study conducted on three industrial systems show that, with this approach, the reliability of the systems can be improved significantly and at least 80% of the maintenance cost can be reduced.

Regression testing is an important activity at both testing and maintenance phases. When a piece of software is modified, it is necessary to ensure the quality of the software is preserved. To this end, regression testing retests the software using the test cases selected from the original test pool. We present a regression testing technique that selects test cases by utilizing static information from the analysis of the program structure and dynamic information by tracing the function-calling sequences. To compare the effectiveness of this technique with other existing approaches, we conducted an empirical study on an industrial trial real-time system. The results show that not only, does this technique preserve all the necessary information for regression testing, but it is also much more efficient and more precise than the existing techniques.

We present an analytical model for estimating architecture-based software reliability, according to the reliability of each component, the operational profile, and the architecture of software. Our approach is based on Markov chain properties and architecture view to state view transformations to perform reliability analysis on heterogeneous software architectures. We demonstrate how this analytical model can be utilized to estimate the reliability of a heterogeneous architecture consisting of batch-sequential/pipeline, call-and-return, parallel/pipe-filters, and fault tolerance styles. In addition, we conduct an experiment on a system embedded with three architectural styles to validate this heterogeneous software reliability model.

Although object-oriented programming has been increasingly adopted for software development and many approaches for testing object-oriented programs have been proposed, the issue of reliability of object-oriented programs has not been explored. The objective of this study was to investigate the effectiveness of class testing from the perspective of reliability. The experiments in this study involved testing and measuring the reliability of a C++ program and a Java program. We introduced a class testing technique that exploits the function dependence relationship to reduce the testing effort in subclass testing and in testing polymorphism without degrading the reliability of object-oriented programs. In subclass testing, the impact of function dependence class testing on reliability was compared with two other techniques: exhaustive class testing, which flattens every class and tests every function in the class; and minimal class testing, which tests only new and re-defined functions. The results show that function dependence class testing preserves the same level of program reliability as does exhaustive class testing, while the effort is significant reduced. In polymorphism testing, we conducted an experiment to observe the relationship between the binding coverage and the reliability of the program. The results suggest that testing possible bindings is necessary, and using the function dependence relationship to determine which bindings to cover in testing is sufficient.

Presents a technique that uses coverage measures in reliability estimation for fault-tolerant programs, particularly N-version software. This technique exploits both coverage and time measures collected during testing phases for the individual program versions and the N-version software system for reliability prediction. The application of this technique to single-version software was presented in our previous research (IEEE 3rd Int. Symp. on Software Metrics, Berlin, Germany, March 1996). In this paper, we extend this technique and apply it on the N-version programs. The results obtained from the experiment conducted on an industrial project demonstrate that our technique significantly reduces the hazard of reliability overestimation for both single-version and multi-version fault-tolerant software systems.

We report a case study to investigate the effect of errors in an operational profile on reliability estimates. A previously reported tool named TERSE was used in this study to generate random flow graphs representing programs, model errors in operational profile, and compute reliability estimates. Four models for reliability estimation were considered: the Musa-Okumoto model, the Goel-Okumoto model, coverage enhanced Musa-Okumoto model, and coverage enhanced Goel-Okumoto model. It was found that the error in reliability estimates from these models grows nonlinearly with errors in operational profile. Results from this case study lend credit to the argument that further research is necessary in development of more robust models for reliability estimation.>

Currently more than forty models for estimating reliability exist. Thus, a practitioner is faced with the problem of selecting one out of many models to predict the behavior of given software. In such a situation, a method or tool to compare the reliability estimates from different models can certainly provide confidence in the selection of models and the estimates given by the selected model. To benchmark existing or new models, we present a new tool (TERSE) which can produce sets of failure data for a given program and compare the estimates produced by existing models. It can also generate random flow graphs for use by a given model. This feature offers a rich source of data for investigating effects of varying model parameters on reliability estimates and allows users to evaluate new models.