Many Just-In-Time defect prediction (JIT) techniques, which anticipate defect-prone software changes, have been proposed in recent years. Researchers have evaluated these techniques from different perspectives and have drawn inconsistent conclusions about which JIT defect prediction techniques are the most effective and efficient. This paper evaluates JIT techniques from a reliability perspective. For short-term early evaluation, we measure JIT predictive performance on early exposed defects. While for long-term evaluation, we quantify the overall reliability improvement resulted from JIT. A case study applying 11 state-of-the-art JIT methods on 18 large open-source projects has shown: 1) Different JIT methods have their own individual strengths for different purposes, 2) in general, RandomForest is the most effective method in short-term software reliability improvement, and CBS+ performs best in long-term reliability improvement; 3) JIT prediction accuracy is highly correlated to overall reliability improvement.

Heavy glycosylation of the envelope (Env) surface subunit, gp120, is a key adaptation of HIV-1; however, the precise effects of glycosylation on the folding, conformation and dynamics of this protein are poorly understood. Here we explore the patterns of HIV-1 Env gp120 glycosylation, and particularly the enrichment in glycosylation sites proximal to the disulfide linkages at the base of the surface-exposed variable domains. To dissect the influence of glycans on the conformation these regions, we focused on an antigenic peptide fragment from a disulfide bridge-bounded region spanning the V1 and V2 hyper-variable domains of HIV-1 gp120. We used replica exchange molecular dynamics (MD) simulations to investigate how glycosylation influences its conformation and stability. Simulations were performed with and without N-linked glycosylation at two sites that are highly conserved across HIV-1 isolates (N156 and N160); both are contacts for recognition by V1V2-targeted broadly neutralizing antibodies against HIV-1. Glycosylation stabilized the pre-existing conformations of this peptide construct, reduced its propensity to adopt other secondary structures, and provided resistance against thermal unfolding. Simulations performed in the context of the Env trimer also indicated that glycosylation reduces flexibility of the V1V2 region, and provided insight into glycan-glycan interactions in this region. These stabilizing effects were influenced by a combination of factors, including the presence of a disulfide bond between the Cysteines at 131 and 157, which increased the formation of beta-strands. Together, these results provide a mechanism for conservation of disulfide linkage proximal glycosylation adjacent to the variable domains of gp120 and begin to explain how this could be exploited to enhance the immunogenicity of those regions. These studies suggest that glycopeptide immunogens can be designed to stabilize the most relevant Env conformations to focus the immune response on key neutralizing epitopes.

We present a new method to identify navigation-related Web usability problems based on comparing actual and anticipated usage patterns. The actual usage patterns can be extracted from Web server logs routinely recorded for operational websites by first processing the log data to identify users, user sessions, and user task-oriented transactions, and then applying an usage mining algorithm to discover patterns among actual usage paths. The anticipated usage, including information about both the path and time required for user-oriented tasks, is captured by our ideal user interactive path models constructed by cognitive experts based on their cognition of user behavior. The comparison is performed via the mechanism of test oracle for checking results and identifying user navigation difficulties. The deviation data produced from this comparison can help us discover usability issues and suggest corrective actions to improve usability. A software tool was developed to automate a significant part of the activities involved. With an experiment on a small service-oriented website, we identified usability problems, which were cross-validated by domain experts, and quantified usability improvement by the higher task success rate and lower time and effort for given tasks after suggested corrections were implemented. This case study provides an initial validation of the applicability and effectiveness of our method.

The HIV-1 envelope (Env) spike, which consists of a compact, heterodimeric trimer of the glycoproteins gp120 and gp41, is the target of neutralizing antibodies. However, the high mutation rate of HIV-1 and plasticity of Env facilitates viral evasion from neutralizing antibodies through various mechanisms. Mutations that are distant from the antibody binding site can lead to escape, probably by changing the conformation or dynamics of Env; however, these changes are difficult to identify and define mechanistically. Here we describe a network analysis-based approach to identify potential allosteric immune evasion mechanisms using three known HIV-1 Env gp120 protein structures from two different clades, B and C. First, correlation and principal component analyses of molecular dynamics (MD) simulations identified a high degree of long-distance coupled motions that exist between functionally distant regions within the intrinsic dynamics of the gp120 core, supporting the presence of long-distance communication in the protein. Then, by integrating MD simulations with network theory, we identified the optimal and suboptimal communication pathways and modules within the gp120 core. The results unveil both strain-dependent and -independent characteristics of the communication pathways in gp120. We show that within the context of three structurally homologous gp120 cores, the optimal pathway for communication is sequence sensitive, i.e. a suboptimal pathway in one strain becomes the optimal pathway in another strain. Yet the identification of conserved elements within these communication pathways, termed inter-modular hotspots, could present a new opportunity for immunogen design, as this could be an additional mechanism that HIV-1 uses to shield vulnerable antibody targets in Env that induce neutralizing antibody breadth.

Orthogonal Defect Classification (ODC), the most influential framework for software defect classification and analysis, provides valuable in-process feedback to system development and maintenance. Conducting ODC classification on existing organizational defect reports is human intensive and requires experts' knowledge of both ODC and system domains. This paper presents AutoODC, an approach and tool for automating ODC classification by casting it as a supervised text classification problem. Rather than merely apply the standard machine learning framework to this task, we seek to acquire a better ODC classification system by integrating experts' ODC experience and domain knowledge into the learning process via proposing a novel Relevance Annotation Framework. We evaluated AutoODC on an industrial defect report from the social network domain. AutoODC is a promising approach: not only does it leverage minimal human effort beyond the human annotations typically required by standard machine learning approaches, but it achieves an overall accuracy of 80.2% when using manual classifications as a basis of comparison.

Identifying change-prone modules can enable software developers to take focused preventive actions that can reduce maintenance costs and improve quality. Some researchers observed a correlation between change proneness and structural measures, such as size, coupling, cohesion, and inheritance measures. However, the modules with the highest measurement values were not found to be the most troublesome modules by some of our colleagues in industry, which was confirmed by our previous study of six large-scale industrial products. To obtain additional evidence, we identified and compared high-change modules and modules with the highest measurement values in two large-scale open-source products, Mozilla and OpenOffice, and we characterized the relationship between them. Contrary to common intuition, we found through formal hypothesis testing that the top modules in change-count rankings and the modules with the highest measurement values were different. In addition, we observed that high-change modules had fairly high places in measurement rankings, but not the highest places. The accumulated findings from these two open-source products, together with our previous similar findings for six closed-source products, should provide practitioners with additional guidance in identifying the change-prone modules.

We characterize usage and problems for Web applications, evaluate their reliability, and examine the potential for reliability improvement. Based on the characteristics of Web applications and the overall Web environment, we classify Web problems and focus on the subset of source content problems. Using information about Web accesses, we derive various measurements that can characterize Web site workload at different levels of granularity and from different perspectives. These workload measurements, together with failure information extracted from recorded errors, are used to evaluate the operational reliability for source contents at a given Web site and the potential for reliability improvement. We applied this approach to the Web sites www.seas.smu.edu and www.kde.org. The results demonstrated the viability and effectiveness of our approach.

Markov chains have been used to model Web usages and served as the basis for statistical testing, performance evaluation, and reliability analysis. However, most of such applications of Markov chains were carried out without answering the question: "Can Web usage be accurately modeled by Markov chains?" In this paper, we propose a set of tests, which are easy to perform based on information about Web link usage frequencies gathered from Web server logs, to answer this question. We applied this approach to our university's Web site, and our results indicate that Markov chains can provide fairly accurate models of Web usages.

After examining the specific problems of testing and quality assurance for Web-based applications, we propose a strategy by integrating existing testing techniques and reliability analyses in a hierarchical framework. This strategy combines various usage models for statistical testing to perform high level testing and to guide selective testing of critical and frequently used subparts or components using traditional coverage-based structural testing. Reliability analysis and risk identification form an integral part of this strategy to help assure and improve the overall reliability for Web-based applications. Some preliminary results are included to demonstrate the general viability and effectiveness of our approach.

In this paper, we analyze web server error logs and the corresponding referral pairs from web access logs to identify and characterize common web errors. We identify missing files as the primary type of web errors and classify them according to their incoming referral links into internal, external, and user errors. We also identify major missing file types within each error category. Based on these analysis results, we recommend different quality assurance initiatives to deal with different types of web problems for the effective improvement to web reliability.

This paper presents a new approach to software reliability modeling by grouping data into clusters of homogeneous failure intensities. This series of data clusters associated with different time segments can be directly used as a piecewise linear model for reliability assessment and problem identification, which can produce meaningful results early in the testing process. The dual model fits traditional software reliability growth models (SRGMs) to these grouped data to provide long-term reliability assessments and predictions. These models were evaluated in the testing of two large software systems from IBM. Compared with existing SRGMs fitted to raw data, our models are generally more stable over time and produce more consistent and accurate reliability assessments and predictions.

Statistical testing and reliability analysis can be used effectively to assure quality for Web applications. To support this strategy, we extract Web usage and failure information from existing Web logs. The usage information is used to build models for statistical Web testing. The related failure information is used to measure the reliability of Web applications and the potential effectiveness of statistical Web testing. We applied this approach to analyze some actual Web logs. The results demonstrated the viability and effectiveness of our approach.

The paper combines relevant research in software reliability engineering and software measurement to develop an integrated approach for the early measurement and improvement of software quality. Recent research in these areas is extended 1) to select appropriate software measures based on. A formal model, 2) to construct tree-based reliability models for early problem identification and quality improvement, and 3) to develop tools to support industrial applications. Initial results applying this approach to several IBM products demonstrated the applicability and effectiveness of this approach.

This paper describes our experience and findings in constructing an operation profile for the Lockheed Martin Tactical Aircraft System's (LMTAS) Cartridge Support Software (CSS). The process is an adaptation of Musa's (1993) 5-step approach. The resulting operational profile was reviewed and evaluated by LMTAS's software product manager, system engineers, and software test engineers. An account of the findings and conclusions from the independent review and evaluation is discussed. This operational profile allowed the LMTAS software engineering team to derive some clear insights about the usage rate of the CSS functions from the customer's perspective.

The paper presents an approach to software reliability modeling using data partitions derived from tree based models. We use these data sensitive partitions to group data into clusters with similar failure intensities. The series of data clusters associated with different time segments forms a piecewise linear model for the assessment and short term prediction of reliability. Long term prediction can be provided by the dual model that uses these grouped data as input fitted to some failure count variations of the traditional software reliability growth models. These partition based reliability models can be used effectively to measure and predict the reliability of software systems and can be readily integrated into our strategy of reliability assessment and improvement using tree based modeling.

The paper examines two existing approaches to software reliability analysis, time domain reliability growth modeling and input domain reliability analysis, and presents a new approach that combines some of their individual strengths. An analysis method called tree-based modeling is used to build models based on the combined measurement data. This new approach can be used to assess the reliability of software systems, to track reliability change over time, and to identify problematic subparts characterized by certain input states or time periods. The results can also be used to guide various remedial actions aimed at reliability improvement. This approach has been demonstrated to be applicable and effective in the testing of several large commercial software systems developed in the IBM Software Solutions Toronto Laboratory.

The paper studies practical reliability measurement and modeling for large commercial software systems based on test execution data collected during system testing. The application environment and the goals of reliability assessment were analyzed to identify appropriate measurement data. Various reliability growth models were used on failure data normalized by test case executions to track testing progress and provide reliability assessment. Practical problems in data collection, reliability measurement and modeling, and modeling result analysis were also examined. The results demonstrated the feasibility of reliability measurement in a large commercial software development environment and provided a practical comparison of various reliability measurements and models under such an environment.>

A formal model of program complexity developed earlier by the authors is used to derive evaluation criteria for program complexity measures. This is then used to determine which measures are appropriate within a particular application domain. A set of rules for determining feasible measures for a particular application domain are given, and an evaluation model for choosing among alternative feasible measures is presented. This model is used to select measures from the classification trees produced by the empirically guided software development environment of R.W. Selby and A.A. Porter, and early experiments show it to be an effective process.>

This paper introduces an integrated environment for software reliability modeling. The environment consists of three major components: (1) a software reliability modeling tool SMERFS adapted to fit our development environment; (2) a general data analysis environment in S-PLUS customized for software reliability modeling; and (3) some supporting programs written in C and AWK. This environment has been successfully used in the IBM Programming Systems Toronto Lab.>

This paper summarizes our experience and findings in measuring and modeling software reliability for a large IBM software product. Four consecutive releases of this product were studied, with various reliability models fitted for individual releases. The models are evaluated by how good they can fit the observed data and how predictive they are. Model sensitivity and performance analysis are performed for across release comparisons, resulting in the models being ranked according to their robustness and accuracy in predicting failures across different data sets. Various ways of combining the multiple release information for generalization and extrapolation to successor releases are also explored.

Identification of high cost modules has been viewed as one mechanism to improve overall system reliability, since such modules tend to produce more than their fair share of problems. A decision tree model has previously been used to identify such modules. In this paper, a previously developed axiomatic model of program complexity is merged with the previously developed decision tree process for an improvement in the ability to identify such modules. This improvement has been tested using data from the NASA Software Engineering Laboratory.>