Studies on software fault-proneness were typically focused on analysis and prediction, with a few exceptions that used explanatory approach. This paper proposes a novel methodology that, for the first time, utilizes a case-control approach for building explanatory models of software fault-proneness. The files with post-release faults are treated as cases and the other files as controls. The cases and controls are matched by size and prerelease fault-proneness (i.e., Bugfixes) is treated as an exposure. The methodology incorporates software metrics as confounders and, for the first time, considers their interactions. Furthermore, the methodology rigorously handles multicollinearity and uses backward elimination to produce the simplest explanatory models. The odds ratios are quantified using conditional logistic regression which leads to efficient estimates, with tighter confidence intervals. The empirical results, based on three Eclipse releases, showed that while some metrics (i.e., Age, Bugfixes and Developers) consistently affected post-release fault-proneness in two or three releases, the effects of other metrics and interactions were release-specific. Additionally, the first-ever systematic exploration of the generalizability in prior explanatory studies showed that, similarly to our study, they experienced limited generalizability of impactful factors, which is likely due to the complex nature of the software and its development processes. Our results have several practical implications: (1) Simple models with 4–7 significant metrics and interactions can explain post-release fault-proneness; (2) The impactful metrics are simple and easy to collect (e.g., files age, the existence of prerelease faults, and developers count); (3) Due to limited generalizability, release/project-specific explanatory models are necessary.

This paper presents an empirical investigation of security vulnerabilities in four categories of transportation IoT devices: Electric Vehicle (EV) Chargers, EVs, non-EVs, and Other Traffic Devices. The results are based on data extracted from the Common Vulnerabilities and Exposures (CVEs) reported in the National Vulnerability Database (NVD). We analyzed 159 CVEs and explored the CWE (Common Weakness Enumeration) and CVSS (Common Vulnerability Scoring System) information associated with them. Our results showed that the security vulnerabilities were distributed unevenly across the 23 vulnerability classes, with the top six most common classes accounting for 74% to 89% of all vulnerabilities in each category of IoT devices. EV Chargers had the highest mean and median CVSS severity score, followed by Other Traffic Devices and EVs. Non-EVs had the lowest CVSS, which were statistically significantly different than the other categories of transportation IoT devices. The paper also presents the lessons learned and the practical implications of our empirical findings.

Despite the increased interest in studying software vulnerabilities, there is a notable lack of studies focused on exploring the generalizability of findings. This paper aims to address this gap by combining replication and synthesis. Thus, our study partially replicates earlier research on vulnerabilities in mission-critical software to explore previously observed phenomena in a different domain. Specifically, this paper focuses on open-source operating systems (OSes) and, in addition to replication, utilizes synthesis to explore the generalizability of findings across other open-source and proprietary OSes. Our results showed that from $76 \%$ to $92 \%$ of security-related bugs in each of the three OSes (Fedora, Red Hat Linux (RHL), and Ubuntu) belonged to only five out of 21 vulnerability classes: Memory Access, Memory Management, Other, Tainted Input, and Information Leak. For RHL, Cryptography replaced Information Leak. The results based on integrative synthesis showed that the dominant classes were consistent across OSes considered in this paper and prior studies. The replication results revealed that Memory Access and Other were among the five dominant classes in both mission-critical software and OSes. The remaining three top vulnerability classes varied due to domains’ specifics. The paper also presents the implications of our findings and the future research directions.

Context: Identifying security bugs in software is critical to minimize vulnerability windows. Traditionally, bug reports are submitted through issue trackers and manually analyzed, which is time-consuming. Challenges such as data scarcity and imbalance generally hinder the development of effective machine learning models that could be used to automate this task. Generative Pre-trained Transformer (GPT) models do not require training and are less affected by the imbalance problem. Therefore, they have gained popularity for various text-based classification tasks, apparently becoming a natural highly promising solution for this problem. Objective: This paper explores the potential of using GPT models to identify security bug reports from the perspective of a user of this type of models. We aim to assess their classification performance in this task compared to traditional machine learning (ML) methods, while also investigating how different factors, such as the prompt used and datasets’ characteristics, affect their results. Methods: We evaluate the performance of four state-of-the-art GPT models (i.e., GPT4All-Falcon, Wizard, Instruct, OpenOrca) on the task of security bug report identification. We use three different prompts for each GPT model and compare the results with traditional ML models. The empirical results are based on using bug report data from seven projects (i.e., Ambari, Camel, Derby, Wicket, Nova, OpenStack, and Ubuntu). Results: GPT models show noticeable difficulties in identifying security bug reports, with performance levels generally lower than traditional ML models. The effectiveness of the GPT models is quite variable, depending on the specific model and prompt used, as well as the particular dataset. Conclusion: Although GPT models are nowadays used in many types of tasks, including classification, their current performance in security bug report identification is surprisingly insufficient and inferior to traditional ML models. Further research is needed to address the challenges identified in this paper in order to effectively apply GPT models to this particular domain.

Microservice architectures offer scalability and flexibility, but due to their distributed nature and complex service structures, raise new security challenges, particularly in detecting DoS attacks. Although addressing these challenges calls for innovative attack detection approaches, developing effective solutions requires large-scale experiments and data collection to create representative datasets. This paper proposes a comprehensive framework to support research on the cybersecurity of microservice applications and the development of different methods to detect cyberattacks. The framework comprises two modules: (i) a data generation module that contains the components necessary to create datasets that reflect the behavior of microservices under attack and (ii) a model development and evaluation module suitable for different methods for detecting attacks on microservices. The framework is illustrated and validated by generating realistic high- and low-volume DoS attack data and developing models using supervised and unsupervised Machine Learning (ML) algorithms and a method based on Logic Scoring of Preference (LSP). The results indicate that supervised ML models have the best classification performance, especially with the XGBoost algorithm. Even though unsupervised ML and LSP models have worse performance, they can be used when the attack data are not available or are costly to generate.

Cyberphysical systems with autonomous functions are complex pieces of software, consisting of many components, some of which implement autonomous functionality and some may use AI or machine learning algorithms. Software bugs in an autonomous system are of particular concern, as they can have catastrophic consequences. However, detailed studies based on empirical data are rare and therefore these bugs are not well understood. This paper aims to contribute towards filling that gap by investigating the software changes and bugs in Autonomy Operating System (AOS) for Unmanned Aircraft Systems (UAS), which consist of 26 components containing about 103,000 lines of code and having a total of 772 bugfixes. Based on the data extracted from the code repository and semi-structured interviews with the developers of AOS, we explore the differences among autonomous software components, components developed using Model-based Software Engineering, and reuse with respect to change proneness, fault proneness, distribution of bugfixes among AOS components and files of these components, and characteristics of bugs of different AOS components. Our results show that the autonomous components were significantly more change prone (measured in number of commits and code churn) and fault prone (measured in bugfixes per KLoC) than non-autonomous components. The distribution of the locations of bugfixes was skewed, both at component and file level (i.e., a small number of components / files contained the majority of bugs). These evidence-based findings provide important insights to researchers and practitioners alike and can be used to efficiently improve the quality and reliability of autonomous systems.

Presents the conference keynote speech, plenary speech, or messages from conference chairs.

Detection of extremely rare cases is a challenging problem for most machine learning algorithms, especially if class overlapping is present. In this paper we propose a hybrid recommender system that uses a target rare case to state users' requirements and ranks the candidates using a similarity function which is calculated as a weighted sum of individual feature similarities. Specifically, the weight of each feature is computed as a product of its association with the class label and the outlyingness of its value. We apply this hybrid recommender system on the radio pulsar candidate selection problem, for detection of two different types of rare cases: low signal-to-noise (S/N) pulsars and Fast Radio Bursts (FRBs). Our results show that the proposed approach successfully detects both low S/N pulsars and FRBs. When there is class overlapping, as in case of low S/N pulsars, treating rare feature values as outliers and increasing their weights in the similarity function improve the detection performance. For FRBs, which compared to the low S/N pulsars are relatively more distinguishable from the non-astrophysical signals, uniform weighting outperformed the feature-weighting methods. The proposed hybrid recommender system can be used in other application domains that share similar requirements such as high recall and face similar challenges such as class imbalance and class overlapping.

Machine learning algorithms have been used extensively for software fault proneness prediction. This paper presents the first application of Group Lasso Regression (G-Lasso) for software fault proneness classification and compares its performance to six widely used machine learning algorithms. Furthermore, we explore the effects of two factors on the prediction performance: the effect of imbalance treatment using the Synthetic Minority Over-sampling Technique (SMOTE), and the effect of datasets used in building the prediction models. Our experimental results are based on 22 datasets extracted from open source projects. The main findings include: (1) G-Lasso is robust to imbalanced data and significantly outperforms the other machine learning algorithms with respect to the Recall and G-Score, i.e., the harmonic mean of Recall and (1- False Positive Rate). (2) Even though SMOTE improved the performance of all learners, it did not have statistically significant effect on G-Lasso's Recall and G-Score. Random Forest was in the top performing group of learners for all performance metrics, while Naive Bayes performed the worst of all learners. (3) When using the same change metrics as features, the choice of the dataset had no effect on the performance of most learners, including G-Lasso. Naive Bayes was the most affected, especially when balanced datasets were used.

Even though Model-based Software Engineering (MBSwE) techniques and Autogenerated Code (AGC) have been increasingly used to produce complex software systems, there is only anecdotal knowledge about the state-of-the practice. Furthermore, there is a lack of empirical studies that explore the potential quality improvements due to the use of these techniques. This paper presents in-depth qualitative findings about development and Software Assurance (SWA) practices and detailed quantitative analysis of software bug reports of a NASA mission that used MBSwE and AGC. The mission's flight software is a combination of handwritten code and AGC developed by two different approaches: one based on state chart models (AGC-M) and another on specification dictionaries (AGC-D). The empirical analysis of fault proneness is based on 380 closed bug reports created by software developers. Our main findings include: (1) MBSwE and AGC provide some benefits, but also impose challenges. (2) SWA done only at a model level is not sufficient. AGC code should also be tested and the models and AGC should always be kept in-sync. AGC must not be changed manually. (3) Fixes made to address an individual bug report were spread both across multiple modules and across multiple files. On average, for each bug report 1.4 modules, that is, 3.4 files were fixed. (4) Most bug reports led to changes in more than one type of file. The majority of changes to auto-generated source code files were made in conjunction to changes in either file with state chart models or XML files derived from dictionaries. (5) For newly developed files, AGC-M and handwritten code were of similar quality, while AGC-D files were the least fault prone.

Data collection for scientific applications is increasing exponentially and is forecasted to soon reach peta- and exabyte scales. Applications which process and analyze scientific data must be scalable and focus on execution performance to keep pace. In the field of radio astronomy, in addition to increasingly large datasets, tasks such as the identification of transient radio signals from extrasolar sources are computationally expensive. We present a scalable approach to radio pulsar detection written in Scala that parallelizes candidate identification to take advantage of in-memory task processing using Apache Spark on a YARN distributed system. Furthermore, we introduce a novel automated multiclass supervised machine learning technique that we combine with feature selection to reduce the time required for candidate classification. Experimental testing on a Beowulf cluster with 15 data nodes shows that the parallel implementation of the identification algorithm offers a speedup of up to 5X that of a similar multithreaded implementation. Further, we show that the combination of automated multiclass classification and feature selection speeds up the execution performance of the RandomForest machine learning algorithm by an average of 54% with less than a 2% average reduction in the algorithm's ability to correctly classify pulsars. The generalizability of these results is demonstrated by using two real-world radio astronomy data sets.

Although network flows have been used in areas such as network traffic analysis and botnet detection, not many works have used network flows-based features for malware detection. This paper is focused on malware detection based on using features extracted from the network traffic and system logs. We evaluated the performance of four supervised machine learning algorithms (i.e., J48, Random Forest, Naive Bayes, and PART) for malware detection and identified the best learner. Furthermore, we used feature selection based on information gain to identify the smallest number of features needed for classification. In addition, we experimented with training sets of different sizes. The main findings include: (1) Adding network flows-based features improved significantly the performance of malware detection. (2) J48 and PART were the best performing learners, with the highest F-score and G-score values. (3) Using J48, the top five features ranked by information gain attained the same performance as when using all 88 features. In the case of PART, the top fourteen features ranked by information gain led to the same performance as when all 88 features were used. None of the system logs-based features were included in these two models. (4) The classification performance when training on 75% of the data was comparable to training on 90% of the data. As little as 25% of the data can be used for training at an expense of somewhat higher, but not very significant performance degradation (i.e., less than 7% for F-score and 6% for G-score compared to when 90% of the data were used for training).

While many prior works used text mining for automating different tasks related to software bug reports, few works considered the security aspects. This paper is focused on automated classification of software bug reports to security and not-security related, using both supervised and unsupervised approaches. For both approaches, three types of feature vectors are used. For supervised learning, we experiment with multiple classifiers and training sets with different sizes. Furthermore, we propose a novel unsupervised approach based on anomaly detection. The evaluation is based on three NASA datasets. The results showed that supervised classification is affected more by the learning algorithms than by feature vectors and training only on 25% of the data provides as good results as training on 90% of the data. The supervised learning slightly outperforms the unsupervised learning, at the expense of labeling the training set. In general, datasets with more security information lead to better performance.

While some prior research work exists on characteristics of software faults (i.e., bugs) and failures, very little work has been published on analysis of software applications vulnerabilities. This paper aims to contribute towards filling that gap by presenting an empirical investigation of application vulnerabilities. The results are based on data extracted from issue tracking systems of two NASA missions. These data were organized in three datasets: Ground mission IV&V issues, Flight mission IV&V issues, and Flight mission Developers issues. In each dataset, we identified the security related software bugs and classified them in specific vulnerability classes. Then, we created the vulnerability profiles, i.e., determined where and when the security vulnerabilities were introduced and what were the dominant vulnerabilities classes. Our main findings include: (1) In IV&V issues datasets the majority of vulnerabilities were code related and were introduced in the Implementation phase. (2) For all datasets, close to 90% of the vulnerabilities were located in two to four subsystems. (3) Out of 21 primary vulnerability classes, five dominated: Exception Management, Memory Access, Other, Risky Values, and Unused Entities. Together, they contributed from around 80% to 90% of vulnerabilities in each dataset.

The papers in this special section were presented at the 2014 International Symposium on Software Reliability Engineering (ISSRE) that was held from 3 to 6 November 2014 in Naples, Italy.

SUMMARY Many papers have been published on analysis and prediction of software faults and/or failures, but few addressed the software fixes made to correct the faults and prevent failures from reoccurring. This paper contributes towards filling this gap by focusing on empirical characterization of software fixes. The results are based on the data extracted from a safety–critical NASA mission. In particular, 21 large‐scale software components (which together constitute over 8000 files and millions of lines of code) were analysed. The unique characteristic of this work is the fact that links were established from software faults (i.e. the root causes) to (potential or observed) failures and consequently to fixes made to correct these faults. Specifically, for the fixes associated with individual failures, the spread across software components and types of software artifacts being fixed was studied. Our results showed that significant number of software failures required fixes in multiple software components and/or multiple software artifacts (i.e. 15% and 26%, respectively). The results also showed that the patterns of software components that were often fixed together were significantly affected by the software architecture. Furthermore, the types of fixed software artifacts were highly correlated with fault type and they had different distributions for prerelease and post‐release failures. Copyright © 2013 John Wiley & Sons, Ltd.

While workload characterization and performance of web systems have been studied extensively, reliability has received much less attention. In this paper, we propose a framework for session reliability modeling which integrates the user view represented by the session layer and the system view represented by the service layer. A unique characteristic of the session layer is that, in addition to the user navigation patterns, it incorporates the session length in number of requests and allows us to account for heavy-tailed workloads shown to exist in real web systems. The service layer is focused on the request reliability as it is observed at the service provider side. It considers the multifier web server architecture and the way components interact in serving each request. Within this framework, we develop a session reliability model and solve it using simulation. Instead of the traditional one-factor-at-a-time sensitivity analysis, we use statistical design and analysis of experiments, which allow us to identify the factors and interactions that have statistically significant effect on session reliability. Our findings indicate that session reliability, which accounts for the distribution of failed requests within sessions, provides better representation of the user perceived quality than the request-based reliability.

Many systems, including Web and Software as a Service (SaaS) are best characterized with session-based workloads. Empirical studies have shown that Web session arrivals exhibit long range dependence and that the number of request in a session is well modeled with skewed or heavy-tailed distributions. However, models that account for session workloads characterized by empirically observed phenomena and studies of their impact on performance metrics are lacking. In this paper, we use partly-open queue to account for session-based workloads in a physically meaningful way and use simulation to analyze the behavior of the Web system under Long Range Dependent (LRD) session arrival process and skewed distribution for the number of requests in a session. Our results show that the percentage of dropped sessions, mean queue length, mean waiting time, and the useful server utilization are all affected by the LRD session arrivals and the statistics of the number of requests within a session. The impact is higher in the case of more prominent longrange dependence. Interestingly, both request arrival process and request departure process are long-range dependent, even in the case when session arrivals are Poisson.

The ever increasing number of vulnerabilities and reported attacks on Web systems clearly illustrate the need for better understanding of malicious cyber activities, which will allow better protection, detection, and service recovery in the cyberspace. In this paper we use three supervised machine learning methods, Support Vector Machines (SVM), and decision trees based J48 and PART, to classify attacker activities aimed at Web systems. The empirical analysis is based on four datasets, each in duration of four to five months, collected by high-interaction honeypots. Malicious Web sessions are characterized with forty three different features (i.e., session attributes) extracted from Web server logs. Our results show that the supervised learning methods can be used to efficiently distinguish attack sessions from vulnerability scan sessions, with very high probability of detection and very low probability of false alarms. Furthermore, we follow the principle of Occam's razor, that is, we seek for the simplest possible model that can successfully classify malicious Web sessions. Our results show that attacks differ from vulnerability scans only in a small number of features (i.e., session attributes). In particular, depending on the data set, classification of malicious activities can be performed using from four to six features without significantly affecting learners' performance compared to when all 43 features are used. Decision tree based methods J48 and PART perform better than SVM across all datasets.

There is a lack of published studies providing empirical support for the assumption at the heart of product line development, namely, that through structured reuse later products will be less fault-prone. This paper presents results from an empirical study of pre-release fault and change proneness from four products in an industrial software product line. The objectives of the study are (1) to determine the association between various software metrics, as well as their correlation with the number of faults at the component level, (2) to characterize the fault and change proneness at various degrees of reuse, and (3) to determine how existing products in the software product line affect the quality of subsequently developed products and our ability to make predictions. The research results confirm, in a software product line setting, the findings of others that faults are more highly correlated to change metrics than to static code metrics. Further, the results show that variation components unique to individual products have the highest fault density and are the most prone to change. The longitudinal aspect of our research indicates that new products in this software product line benefit from the development and testing of previous products. For this case study, the number of faults in variation components of new products is predicted accurately using a linear model built on data from the previous products.

The number of vulnerabilities and attacks on Web systems show an increasing trend and tend to dominate on the Internet. Furthermore, due to their popularity and users ability to create content, Web 2.0 applications have become particularly attractive targets. These trends clearly illustrate the need for better understanding of malicious cyber activities based on both qualitative and quantitative analysis. This paper is focused on multiclass classification of malicious Web activities using three supervised machine learning methods: J48, PART, and Support Vector Machines (SVM). The empirical analysis is based on data collected in duration of nine months by a high interaction honey pot consisting of a three-tier Web system, which included Web 2.0 applications (i.e., a blog and wiki). Our results show that supervised learning methods can be used to efficiently distinguish among multiple vulnerability scan and attack classes, with high recall and precision values for all but several very small classes. For our dataset, decision tree based methods J48 and PART perform slightly better than SVM in terms of overall accuracy and weighted recall. Additionally, J48 and PART require less than half of the features (i.e., session attributes) used by SVM, as well as they execute much faster. Therefore, they seem to be clear methods of choice.

Reliability is important to software product-line developers since many product lines require reliable operation. It is typically assumed that as a software product line matures, its reliability improves. Since post-deployment failures impact reliability, we study this claim on an open-source software product line, Eclipse. We investigate the failure trend of common components (reused across all products), highreuse variation components (reused in five or six products) and low-reuse variation components (reused in one or two products) as Eclipse evolves. We also study how much the common and variation components change over time both in terms of addition of new files and modification of existing files. Quantitative results from mining and analysis of the Eclipse bug and release repositories show that as the product line evolves, fewer serious failures occur in components implementing commonality, and that these components also exhibit less change over time. These results were roughly as expected. However, contrary to expectation, components implementing variations, even when reused in five or more products, continue to evolve fairly rapidly. Perhaps as a result, the number of severe failures in variation components shows no uniform pattern of decrease over time. The paper describes and discusses this and related results. Categories and Subject Descriptors D.2.8 [Software Engineering]: Metrics—Product metrics,

Web-based systems commonly face unique set of vulnerabilities and security threats due to their high exposure, access by browsers, and integration with databases. In this paper we present empirical analysis of attackers activities based on data collected by two high-interaction honeypots. The contributions of our work include: (1) Classification of the malicious traffic to port scans, vulnerability scans, and attacks; (2) Conducting experiments which, in addition to attackers activities aimed at individual components, allowed us to observe and study vulnerability scans and attacks that span multiple system components; and (3) Statistical characterization of the malicious traffic.

Information captured in software execution profiles can benefit verification activities by supporting more cost-effective fault localization and execution classification. This paper proposes an experimental design which utilizes execution information to quantify the effect of factors such as different programs and fault inclusions on the distinction between passed and failed execution profiles. For this controlled experiment we use well-known, benchmark-like programs. In addition to experimentation, our empirical evaluation includes case studies of open source programs having more complex fault models. The results show that metrics reflecting distinction between failing and passing executions are affected more by program than by faults included.

The widespread use of Web applications, in conjunction with large number of vulnerabilities, makes them very attractive targets for malicious attackers. The increasing popularity of Web 2.0 applications, such as blogs, wikis, and social sites, makes Web servers even more attractive targets. In this paper we present empirical analysis of attackers activities based on data collected by two high-interaction honeypots which have typical three-tier architectures and include Web 2.0 applications. The contributions of our work include in-depth characterization of different types of malicious activities aimed at Web servers that deploy blog and wiki applications, as well as formal inferential statistical analysis of the malicious Web sessions.

The four papers in this special section present new findings on different aspects of software dependability.

This paper is focused on modeling Web request and session level arrival processes. We propose a statistically rigorous approach which includes testing for non-stationarity and Gaussianity, and uses model selection criterion. Furthermore, a goodness of fit test is applied to each candidate model - ARMA, ARIMA, FARIMA, and FGN - and for validation purpose real data is compared with data simulated from the models. The results based on data extracted from six Web servers with different workload intensities show that (1) there is no one-fits-all solution and (2) servers with high workloads have both request and session traffic modeled well with FARIMA model which is capable of capturing both long-range and short-range dependence.

The assumption of independence among successive software runs, common to many software reliability models, often is a simplification of the actual behavior. This paper addresses the problem of estimating software reliability when the successive software runs are statistically correlated, that is, when an outcome of a run depends on one or more of its previous runs. First, we propose a generalization of our previous work using higher order Markov chain to model a sequence of dependent software runs. Then, we conduct an empirical study for exploring the phenomenon of dependent software runs using three software applications as case studies. Based on two statistical approaches, we show that the outcomes of software runs (i.e., success or failure) for two of the case studies are dependent on the outcome of one or more previous runs, in which case first or higher order Markov chain models are appropriate. Finally, we estimate the parameters of the appropriate models and discuss the effects of dependent software runs on the estimates of the software reliability.

The benefits of the analysis of software faults and failures have been widely recognized. However, detailed studies based on empirical data are rare. In this paper, we analyze the fault and failure data from two large, real-world case studies. Specifically, we explore: 1) the localization of faults that lead to individual software failures and 2) the distribution of different types of software faults. Our results show that individual failures are often caused by multiple faults spread throughout the system. This observation is important since it does not support several heuristics and assumptions used in the past. In addition, it clearly indicates that finding and fixing faults that lead to such software failures in large, complex systems are often difficult and challenging tasks despite the advances in software development. Our results also show that requirement faults, coding faults, and data problems are the three most common types of software faults. Furthermore, these results show that contrary to the popular belief, a significant percentage of failures are linked to late life cycle activities. Another important aspect of our work is that we conduct intra- and interproject comparisons, as well as comparisons with the findings from related studies. The consistency of several main trends across software systems in this paper and several related research efforts suggests that these trends are likely to be intrinsic characteristics of software faults and failures rather than project specific.

A software product spends more than 65% of its lifecycle in maintenance. Software systems with good maintainability can be easily modified to fix faults. We define maintainability-based risk as a product of two factors: the probability of performing maintenance tasks and the impact of performing these tasks. In this paper, we present a methodology for assessing maintainability-based risk in the context of corrective maintenance. The proposed methodology depends on the architectural artifacts and their evolution through the life cycle of the system. In order to prioritize corrective maintenance tasks, we combine components' maintainability- based risk with the severity of a failure that may happen as a result of unfixed fault. We illustrate the methodology on a case study using UML models.

Uncertainty analysis through sensitivity studies and quantification of the variance of the reliability estimate has become more common in architecture-based software reliability studies. However, up to this point no attempts have been made to explicate the results of such analysis. Our earlier work based on several medium to large scale empirical studies showed that a very few parameters have a significant impact on the variability of system reliability. This paper explains the reasons behind this phenomenon. Unlike related work that considered the impact of the parameters on software reliability either through their model sensitivity or through uncertainty of their estimates, we consider both. Furthermore, we look at all parameters, i.e., components reliabilities and probabilities of transfer of control between components. Based on theoretical and empirical arguments, we justify why a few parameters contribute most of the variance of the reliability estimate. Comparing our results with those obtained through simple model sensitivity studies shows that such studies are not always sufficient to accurately quantify the impact of critical components on variability of system reliability.

In this paper we present clustering analysis of session-based Web workloads of eight Web servers using the intrasession characteristics (i.e., number of requests per session, session length in time, and bytes transferred per session) as variables. We use K-means algorithm and the Mahalanobis distance, and analyze the heavy-tailed behavior of intra-session characteristics and their correlations for each cluster. Our results show that clustering provides an efficient way to classify tens or hundreds thousands of sessions into several coherent classes that efficiently describe Web workloads. These classes reveal phenomena that cannot be observed when studying the workload as a whole.

World Wide Web, the biggest distributed system ever built, experiences tremendous growth and change in Web sites, users, and technology. A realistic and accurate characterization of Web workload is the first, fundamental step in areas such as performance analysis and prediction, capacity planning, and admission control. Compared to the previous work, in this paper we present more detailed and rigorous statistical analysis of both request and session level characteristics of Web workload based on empirical data extracted from actual logs of four Web servers. Our analysis is focused on exploring phenomena such as self-similarity, long-range dependence, and heavy-tailed distributions. Identification of these phenomena in real data is a challenging task since the existing methods may perform erratically in practice and produce misleading results. We provide more accurate analysis of long-range dependence of the request and session arrival processes by removing the trend and periodicity. In addition to the session arrival process (i.e., inter-session characteristics), we study several intra-session characteristics using several different methods to test the existence of heavy-tailed behavior and cross validate the results. Finally, we point out specific problems associated with the methods used for establishing long-range dependence and heavy-tailed behavior of Web workloads. We believe that the comprehensive model presented in this paper is a step towards solving the Web workload puzzle

Our earlier research work on applying architecture-based software reliability models on a large scale case study allowed us to test how and when they work, to understand their limitations, and to outline the issues that need future research. In this paper we first present an additional case study which confirms our earlier findings. Then, we present uncertainty analysis of architecture-based software reliability for both case studies. The results show that Monte Carlo method scales better than the method of moments. The sensitivity analysis based on Monte Carlo method shows that (1) small number of parameters contribute to the most of the variation in system reliability and (2) given an operational profile, components' reliabilities have more significant impact on system reliability than transition probabilities. Finally, we summarize the lessons learned from conducting large scale empirical case studies for the purpose of architecture-based reliability assessment and uncertainty analysis.

In this paper we present an empirical study of architecture-based software reliability based on a large open source application which consists of 350,000 lines of C code. The goals of our study are to analyze empirically the adequacy, applicability, and accuracy of architecture-based software reliability models. For this purpose we developed innovative approaches to efficiently extract and more accurately analyze a large amount of empirical data. Applying the theoretical results on a large scale field study allows us to test how and when they work, to understand their limitations, and outline the issues that need attention in the future research studies. Thus, our results show that for a subset of failures which can clearly be attributed to single components, both the composite and hierarchical models are very accurate when compared to the actual reliability. However, the assumptions made by the existing architecture-based software reliability models do not allow accounting for the remaining failures which led to fixing faults in multiple components. These results show that in order to progress further, software reliability engineering should go through cycles of building models, testing them empirically, learning from the experiments, and refining the models to capture the newly discovered phenomena.

Performance is a nonfunctional software attribute that plays a crucial role in wide application domains spreading from safety-critical systems to e-commerce applications. Software risk can be quantified as a combination of the probability that a software system may fail and the severity of the damages caused by the failure. In this paper, we devise a methodology for estimation of performance-based risk factor, which originates from violations, of performance requirements, (namely, performance failures). The methodology elaborates annotated UML diagrams to estimate the performance failure probability and combines it with the failure severity estimate which is obtained using the functional failure analysis. We are thus able to determine risky scenarios as well as risky software components, and the analysis feedback can be used to improve the software design. We illustrate the methodology on an e-commerce case study using step-by step approach, and then provide a brief description of a case study based on large real system.

The growing availability of Internet access has led to significant increase in the use of World Wide Web. If we are to design dependable Web-based systems that deal effectively with the increasing number of clients and highly variable workload, it is important to be able to describe the Web workload and errors accurately. In this paper we focus on the detailed empirical analysis of the session-based workload and reliability based on the data extracted from actual Web logs often Web servers. First, we address the data collection process and describe the methods for extraction of workload and error data from Web log files. Then, we introduce and analyze several intra-session and inter-session metrics that collectively describe Web workload in terms of user sessions. Furthermore, we analyze Web error characteristics and estimate the request-based and session-based reliability of Web servers. Finally, we identify the invariants of the Web workload and reliability that apply through all data sets considered. The results presented in this paper show that session-based workload and reliability are better indicators of the users perception of the Web quality than the request-based metrics and provide more useful measures for tuning and maintaining of the Web servers.

Recent evidences indicate that most faults in software systems are found in only a few of a system's components [1]. The early identification of these components allows an organization to focus on defect detection activities on high risk components, for example by optimally allocating testing resources [2], or redesigning components that are likely to cause field failures. This paper presents a prototype tool called Architecture-level Risk Assessment Tool (ARAT) based on the risk assessment methodology presented in [3]. The ARAT provides risk assessment based on measures obtained from Unified Modeling Language (UML) artifacts [4]. This tool can be used in the design phase of the software development process. It estimates dynamic metrics [5] and automatically analyzes the quality of the architecture to produce architectural-level software risk assessment [3].

Many architecture-based software reliability models were proposed in the past. Regardless of the accuracy of these models, if a considerable uncertainty exists in the estimates of the operational profile and components reliabilities then a significant uncertainty exists in calculated software reliability. Therefore, the traditional way of estimating software reliability by plugging point estimates of unknown parameters into the model may not be appropriate since it discards any variance due to uncertainty of the parameters. In this paper we propose a methodology for uncertainty analysis of architecture-based software reliability models suitable for large complex component based applications and applicable throughout the software life cycle. First, we describe different approaches to build the architecture based software reliability model and to estimate parameters. Then, we perform uncertainty analysis using the method of moments and Monte Carlo simulation which enable us to study how the uncertainty of parameters propagates in the reliability estimate. Both methods are illustrated on two case studies and compared using several criteria.

In this paper we present a methodology for uncertainty analysis of the software operational profile suitable for large complex component-based applications and applicable throughout the software life cycle. Within this methodology, we develop a method for studying the sensitivity of software usage to changes in the operational profile based on perturbation theory. This method is then illustrated on three case studies: software developed for the European Space Agency, an e-commerce application, and real-time control software. Results show that components with small execution rates are the most sensitive to the changes in the operational profile. This observation is very important due to the fact that rarely executed components usually handle critical functionalities such as exception handling or recovery.

Risk assessment is an essential part in managing software development. Performing risk assessment during the early development phases enhances resource allocation decisions. In order to improve the software development process and the quality of software products, we need to be able to build risk analysis models based on data that can be collected early in the development process. These models will help identify the high-risk components and connectors of the product architecture, so that remedial actions may be taken in order to control and optimize the development process and improve the quality of the product. In this paper, we present a risk assessment methodology which can be used in the early phases of the software life cycle. We use the Unified Modeling Language (UML) and commercial modeling environment Rational Rose Real Time (RoseRT) to obtain UML model statistics. First, for each component and connector in software architecture, a dynamic heuristic risk factor is obtained and severity is assessed based on hazard analysis. Then, a Markov model is constructed to obtain scenarios risk factors. The risk factors of use cases and the overall system risk factor are estimated using the scenarios risk factors. Within our methodology, we also identify critical components and connectors that would require careful analysis, design, implementation, and more testing effort. The risk assessment methodology is applied on a pacemaker case study.

Quite often failures in network based services and server systems may not be accidental, but rather caused by deliberate security intrusions. We would like such systems to either completely preclude the possibility of a security intrusion or design them to be robust enough to continue functioning despite security attacks. Not only is it important to prevent or tolerate security intrusions, it is equally important to treat security as a QoS attribute at par with, if not more important than other QoS attributes such as availability and performability. This paper deals with various issues related to quantifying the security attribute of an intrusion tolerant system, such as the SITAR system. A security intrusion and the response of an intrusion tolerant system to the attack is modeled as a random process. This facilitates the use of stochastic modeling techniques to capture the attacker behavior as well as the system's response to a security intrusion. This model is used to analyze and quantify the security attributes of the system. The security quantification analysis is first carried out for steady-state behavior leading to measures like steady-state availability. By transforming this model to a model with absorbing states, we compute a security measure called the "mean time (or effort) to security failure" and also compute probabilities of security failure due to violations of different security attributes.

Messaging services are a useful component in distributed systems that require scalable dissemination of messages (events) from suppliers to consumers. These services decouple suppliers and consumers, and take care of client registration and message propagation, thus relieving the burden on the supplier Recently performance models for the configurable delivery and discard policies found in messaging services have been developed, that can be used to predict response time distributions and discard probabilities under failure-free conditions. However, these messaging service models do not include the effect of failures. In a distributed system, supplier, consumer and messaging services can fail independently leading to different consequences. In this paper we consider the expected loss rate associated with messaging services as a performability measure and derive approximate closed-form expressions for three different quality of service settings. These measures provide a quantitative framework that allows different messaging service configurations to be compared and design trade-off decisions to be made.

Many architecture-based software reliability models have been proposed in the past without any attempt to establish a relationship among them. The aim of this paper is to fill this gap. First, the unifying structural properties of the models are exhibited and the theoretical relationship is established. Then, the estimates provided by the models are compared using an empirical case study. The program chosen for the case study consists of almost 10,000 lines of C code divided into several components. The faulty version of the program was obtained by reinserting the faults discovered during integration testing and operational usage and the correct version was used as an oracle. A set of test cases was generated randomly accordingly to the known operational profile. The results show that 1) all models give reasonably accurate estimations compared to the actual reliability and 2) faults present in the components influence both components reliabilities and the way components interact.

Software rejuvenation is a preventive maintenance technique that has been extensively studied in recent literature. In this paper, we extend the classical result by Huang et al. (1995), and in addition propose a modified stochastic model to generate the software rejuvenation schedule. More precisely, the software rejuvenation models are formulated via the semi-Markov reward process, and the optimal software rejuvenation schedules are derived analytically in terms of the reward rate. In particular, we consider the two special cases: steady-state availability and expected cost per unit time in the steady state. Further, we develop non-parametric algorithms to estimate the optimal software rejuvenation schedules, provided that the statistically complete (unsensored) sample data of failure time is given. In numerical examples, we compare two models from the viewpoints of system availability and economic justification, and examine asymptotic properties for the statistical estimation algorithms.

In this paper, we extend the classical result by Huang, Kintala, Kolettis and Fulton (1995), and in addition propose a modified stochastic model to determine the software rejuvenation schedule. More precisely, the software rejuvenation models are formulated via the semi-Markov processes, and the optimal software rejuvenation schedules which maximize the system availabilities are derived analytically for respective cases. Further, we develop nonparametric statistical algorithms to estimate the optimal software rejuvenation schedules, provided that the statistical complete (unsensored) sample data of failure times is given. In numerical examples, we examine asymptotic properties for the statistical estimation algorithms.

Since the early 1970's a number of models have been proposed for estimating software reliability. However, the realism of many of the underlying assumptions and the applicability of these models continue to be questioned. Our research work was motivated by the fact that although there are practical situations in which the assumption of independence among successive software failures could be easily violated, much of the published literature on software reliability modeling does not seriously address this issue. In this paper we present a modeling framework based on Markov renewal processes which naturally introduces dependence among successive software runs and enables the phenomena of failure correlation to be precisely characterized. Thus, incorporating failure correlation into dependability and performability predictions contributes toward more realistic modeling of software systems in operation.

Perhaps the most stringent restriction in most software reliability models is the assumption of statistical independence among successive software failures. The authors research was motivated by the fact that although there are practical situations in which this assumption could be easily violated, much of the published literature on software reliability modeling does not seriously address this issue. The research work in this paper is devoted to developing the software reliability modeling framework that can consider the phenomena of failure correlation and to study its effects on the software reliability measures. The important property of the developed Markov renewal modeling approach is its flexibility. It allows construction of the software reliability model in both discrete time and continuous time, and (depending on the goals) to base the analysis either on Markov chain theory or on renewal process theory. Thus, their modeling approach is an important step toward more consistent and realistic modeling of software reliability. It can be related to existing software reliability growth models. Many input-domain and time-domain models can be derived as special cases under the assumption of failure s-independence. This paper aims at showing that the classical software reliability theory can be extended to consider a sequence of possibly s-dependent software runs, viz, failure correlation. It does not deal with inference nor with predictions, per se. For the model to be fully specified and applied to estimations and predictions in real software development projects, we need to address many research issues, e.g., the detailed assumptions about the nature of the overall reliability growth, way modeling-parameters change as a result of the fault-removal attempts.

Perhaps the most stringent restriction that is present in most software reliability models is the assumption of independence among successive software failures. Our research was motivated by the fact that although there are practical situations in which this assumption could be easily violated, much of the published literature on software reliability modeling does not seriously address this issue. In this paper we present a software reliability modeling framework based on Markov renewal processes which naturally introduces dependence among successive software runs. The presented approach enables the phenomena of failure clustering to be precisely characterized and its effects on software reliability to be analyzed. Furthermore, it also provides bases for a more flexible and consistent model formulation and solution. The Markov renewal model presented in this paper can be related to the existing software reliability growth models, that is, a number of them can be derived as special cases under the assumption of failure independence. Our future research is focused on developing more specific and detailed models within this framework, as well as statistical inference procedures for performing estimations and predictions based on the experimental data.

The paper presents a detailed, but efficiently solvable model of the N version programming for evaluating reliability and performability over a mission period. Employing a hierarchical decomposition we reduce the model complexity and provide a modeling framework for evaluating the NVP failure and execution time behavior and the operational environment, as well. The failure and execution rates are treated as random variables and the operational profile is analyzed on the microstructure level, looking at probabilities of occurrence, failure and execution rates for each partition of input space. The reliability submodel that represents per run behavior of NVP, includes both functional failures and timing failures thus resulting in system reliability which accounts for performance requirements. The successive runs are modeled by the performance submodel, that represents the iterative nature of the software execution. Combining the results of both submodels, we assess the performability over a mission period that represents the collective effect of multiple system attributes on the NVP effectiveness.

The paper presents performance modeling and evaluation of recovery block systems. In order to produce a dependability model for a complete fault tolerant system we consider the interaction between the faults in the alternatives and the faults in the acceptance test. The study is based on finite state continuous time Markov model, and unlike previous works, we carry out the analysis in the time domain. The undetected and total failure probabilities (safety and reliability), as well as the average recovery block execution time expressions are obtained. Derived mathematical relations between failure probabilities (i.e. reliability and safety) and modeling parameters enable us to gain a great deal of quantitative results.

Reliability performance modeling of N version programming is given. The study is based on continuous time Markov model for the general case of N versions. Derived mathematical relations between reliability performances (as a function of version execution time) and modeling parameters enable us to gain a great deal of quantitative results. The obtained results can be used to guide a design of actual systems.>