Adoption of software engineering best practices requires a good infrastructure for monitoring compliance with each practice. Quite often, practice compliance implies that a set of policies are being adhered by each development group. Monitoring each practice to the level of individual policies is cumbersome as it involves some one to follow up manually with the development groups on each policy in each practice. This results in a significant quality assurance effort in a big organization where we have large number of development groups. Moreover, the data can be error prone and outdated due to time and multiple people involved in such activities. In this paper, we propose a policy checking and reporting framework to automate the reporting to a good extent. The framework is based on a central reporting service which collects software engineering data from various sources and provides the assessment on each policy related to static analysis practice.

Global nature of multinational organizations pose a different set of challenges in an effective adoption of software engineering tools. Software projects are spread across multiple geographies and tools are deployed globally in these organizations. Global tool deployment is often motivated by an attractive enterprise cost of the tool. Moreover, it helps in harmonizing practices across development groups and useful in establishing benchmarks without further normalizations. There are various parameters to drive a successful tool deployment in a global setup. For example, a good roll out plan, knowledge sharing platform, coordination and trust building among the change agents from different geographies, and a good central support. In this paper, we present our experience of a global roll out of a software engineering tool in a large organization. It has been observed that besides tool's features, factors such as knowledge sharing, active feedback, and training plays an important role in an effective tool deployment.

No single software fault-detection technique is capable of addressing all fault-detection concerns. Similarly to software reviews and testing, static analysis tools (or automated static analysis) can be used to remove defects prior to release of a software product. To determine to what extent automated static analysis can help in the economic production of a high-quality product, we have analyzed static analysis faults and test and customer-reported failures for three large-scale industrial software systems developed at Nortel Networks. The data indicate that automated static analysis is an affordable means of software fault detection. Using the orthogonal defect classification scheme, we found that automated static analysis is effective at identifying assignment and checking faults, allowing the later software production phases to focus on more complex, functional, and algorithmic faults. A majority of the defects found by automated static analysis appear to be produced by a few key types of programmer errors and some of these types have the potential to cause security vulnerabilities. Statistical analysis results indicate the number of automated static analysis faults can be effective for identifying problem modules. Our results indicate static analysis tools are complementary to other fault-detection techniques for the economic production of a high-quality software product.

To maximize business value, commercial software organizations need to apply software reliability engineering (SRE) using a distributed model where key practices are performed by different roles in the organization. SRE practitioners need to understand the landscape of software development in order to define a program that is effective at moving the organization towards higher reliability

Software inspection has been shown to be an effective defect removal practice, leading to higher quality software with lower field failures. Automated software inspection tools are emerging for identifying a subset of defects in a less labor-intensive manner than manual inspection. This paper investigates the use of automated inspection for a large-scale industrial software system at Nortel Networks. We propose and utilize a defect classification scheme for enumerating the types of defects that can be identified by automated inspections. Additionally, we demonstrate that automated code inspection faults can be used as efficient predictors of field failures and are effective for identifying fault-prone modules.

This paper presents an empirical study that evaluates software-quality models over several releases, to address the question, "How long will a model yield useful predictions?" The classification and regression trees (CART) algorithm is introduced, CART can achieve a preferred balance between the two types of misclassification rates. This is desirable because misclassification of fault-prone modules often has much more severe consequences than misclassification of those that are not fault-prone. The case-study developed 2 classification-tree models based on 4 consecutive releases of a very large legacy telecommunication system. Forty-two software product, process and execution metrics were candidate predictors. Model 1 used measurements of the first release as the training data set; this model had 11 important predictors. Model 2 used measurements of the second release as the training data set; this model had 15 important predictors. Measurements of subsequent releases were evaluation data sets. Analysis of the models' predictors yielded insights into various software development practices. Both models had accuracy that would be useful to developers. One might suppose that software-quality models lose their value very quickly over successive releases due to evolution of the product and the underlying development processes. The authors found the models remained useful over all the releases studied.

Software quality modeling can be used by a software maintenance project to identify a limited set of software modules that probably need improvement. A model's goal is to recommend a set of modules to receive special treatment. The purpose of the paper is to report our experiences modeling software quality with classification trees, including necessary preprocessing of data. We conducted a case study on two releases of a very large legacy telecommunications system. A module was considered fault-prone if any faults were discovered by customers, and not fault-prone otherwise. Software product, process, and execution metrics were the basis for predictors. The TREEDISC algorithm for building classification trees was investigated, because it emphasizes statistical significance. Numeric data, such as software metrics, are not suitable for TREEDISC. Consequently, we transformed measurements into discrete ordinal predictors by grouping. This case study investigated the sensitivity of modeling results to various groupings. We found that robustness, accuracy, and parsimony of the models were influenced by the maximum number of groups. Models based on two sets of candidate predictors had similar sensitivity.

Software quality models are tools for focusing software enhancement efforts. Such efforts are essential for mission-critical embedded software, such as telecommunications systems, because customer-discovered faults have very serious consequences and are very expensive to repair. We present an empirical study that evaluated software quality models over several releases to address the question, "How long will a model yield useful predictions?" We also introduce the Classification And Regression Trees (CART) algorithm to software reliability engineering practitioners. We present our method for exploiting CART features to achieve a preferred balance between the two types of misclassification rates. This is desirable because misclassifications of fault-prone modules often have much more severe consequences than misclassifications of those that are not fault-prone. We developed two classification-tree models based on four consecutive releases of a very large legacy telecommunications system. Forty-two software product, process, and execution metrics were candidate predictors. The first software quality model used measurements of the first release as the training data set and measurements of the subsequent three releases as evaluation data sets. The second model used measurements of the second release as the training data set and measurements of the subsequent two releases as evaluation data sets. Both models had accuracy that would be useful to developers.

"Knowledge discovery in data bases" (KDD) for software engineering is a process for finding useful information in the large volumes of data that are a byproduct of software development, such as data bases for configuration management and for problem reporting. This paper presents guidelines for extracting innovative process metrics from these commonly available data bases. This paper also adapts the Classification And Regression Trees algorithm, CART, to the KDD process for software engineering data. To our knowledge, this algorithm has not been used previously for empirical software quality modeling. In particular, we present an innovative way to control the balance between misclassification rates. A KDD case study of a very large legacy telecommunications software system found that variables derived from source code, configuration management transactions, and problem reporting transactions can be useful predictors of software quality. The KDD process discovered that for this software development environment, out of forty software attributes, only a few of the predictor variables were significant. This resulted in a model that predicts whether modules are likely to have faults discovered by customers. Software developers need such predictions early in development to target software enhancement techniques to the modules that need improvement the most.

High software reliability is an important attribute of high-assurance systems. Software quality models yield timely predictions of quality indicators on a module-by-module basis, enabling one to focus on finding faults early in development. This paper introduces the Classification And Regression Trees (CART) a algorithm to practitioners in high-assurance systems engineering. This paper presents practical lessons learned on building classification trees for software quality modeling, including an innovative way to control the balance between misclassification rates. A case study of a very large telecommunications system used CART to build software quality models. The models predicted whether or not modules would have faults discovered by customers, based on various sets of software product and process metrics as independent variables. We found that a model based on two software product metrics had comparable accuracy to a model based on forty product and process metrics.

Software is the medium for implementing increasingly sophisticated features during the maintenance phase as successive releases are developed. Software quality models can predict which software modules are likely to have faults that will be discovered by customers. Such models are key components of a system such as Enhanced Measurement for Early Risk Assessment of Latent Defects (EMERALD). It is a sophisticated system of decision support tools used by software designers and managers at Nortel to assess risk and improve software quality and reliability of legacy software systems. This paper reports an approach to software quality modelling that is suitable for industrial systems such as EMERALD. We conducted a case study of a large legacy telecommunications system in the maintenance phase to predict whether each module will be considered fault-prone. The case study is distinctive in the following respects. (1) Fault-prone modules were defined in terms of faults discovered by customers, which represent only a small fraction of the modules in the system. (2) We developed models based on software product and process metrics that can make useful predictions at the end of the coding phase and at the time of release. (3) The modelling approach is suitable for very large systems. We anticipate that refinements of this case study's models will be incorporated into EMERALD. A similar approach could be taken for other systems. Copyright © 1999 John Wiley & Sons, Ltd.

The aim of the experiment presented in this paper is to present an insight into the evaluation of the potential benefits of introducing a function clone detection technology in an industrial software development process. To take advantage of function clone detection, two modifications to the software development process are presented. Our experiment consists of evaluating the impact that these proposed changes would have had on a specific software system if they had been applied over a 3 year period (involving 10000 person-months), where 6 subsequent versions of the software under study were released. The software under study is a large telecommunication system. In total 89 million lines of code have been analyzed. A first result showed that, against our expectations, a significant number of clones are being removed from the system over time. However, this effort is insufficient to prevent the growth of the overall number of clones in the system. In this context the first process change would have added value. We have also found that the second process change would have provided programmers with a significant number of opportunities for correcting problems before customers experienced them. This result shows a potential for improving the software system quality and customer satisfaction

In this empirical study, from a large data set of software metrics for program modules, thirty distinct partitions into training and validation sets are automatically generated with approximately equal distributions of fault prone and not fault prone modules. Thirty classification models are built for each of the two approaches considered-discriminant analysis and the evolutionary neural network (ENN) approach-and their performances on corresponding data sets are compared. The lower error proportions for ENNs on fault prone, not fault prone, and overall classification were found to be statistically significant. The robustness of ENNs follows from their superior performance on the range of data configurations used. It is suggested that ENNs can be effective in other software reliability problem domains, where they have been largely ignored.

Society relies on telecommunications to such an extent that telecommunications software must have high reliability. Enhanced measurement for early risk assessment of latent defects (EMERALD) is a joint project of Nortel and Bell Canada for improving the reliability of telecommunications software products. This paper reports a case study of neural-network modeling techniques developed for the EMERALD system. The resulting neural network is currently in the prototype testing phase at Nortel. Neural-network models can be used to identify fault-prone modules for extra attention early in development, and thus reduce the risk of operational problems with those modules. We modeled a subset of modules representing over seven million lines of code from a very large telecommunications software system. The set consisted of those modules reused with changes from the previous release. The dependent variable was membership in the class of fault-prone modules. The independent variables were principal components of nine measures of software design attributes. We compared the neural-network model with a nonparametric discriminant model and found the neural-network model had better predictive accuracy.

The genetic algorithm is applied to developing optimal or near optimal backpropagation neural networks for fault-prone/not-fault-prone classification of software modules. The algorithm considers each network in a population of neural networks as a potential solution to the optimal classification problem. Variables governing the learning and other parameters and network architecture are represented as substrings (genes) in a machine-level bit string (chromosome). When the population undergoes simulated evolution using genetic operators-selection based on a fitness function, crossover, and mutation-the average performance increases in successive generations. We found that, on the same data, compared with the best manually developed networks, evolved networks produced improved classifications in considerably less time, with no human effort, and with greater confidence in their optimality or near optimality. Strategies for devising a fitness function specific to the problem are explored and discussed.

Enhanced Measurement for Early Risk Assessment of Latent Defects (EMERALD) is a decision support system for assessing reliability risk. It is used by software developers and managers to improve telecommunications software service quality as perceived by the customer and the end user. Risk models are based on static characteristics of source code. This paper shows how a system such as EMERALD can enhance software development, testing, and maintenance by integration of: a software quality improvement strategy; measurements and models; and delivery of results to the desktop of developers in a timely manner. This paper also summarizes empirical experiments with EMERALD's models using data from large industrial telecommunications software systems. EMERALD has been applied to a very large system with over 12 million lines of source code within procedures. Experience and lessons learned are also discussed.

Telecommunications software is known for its high reliability. Society has become so accustomed to reliable telecommunications, that failures can cause major disruptions. This is an experience report on application of discriminant analysis based on 20 static software product metrics, to identify fault prone modules in a large telecommunications system, so that reliability may be improved. We analyzed a sample of 2000 modules representing about 1.3 million lines of code, drawn from a much larger system. Sample modules were randomly divided into a fit data set and a test data set. We simulated utilization of the fitted model with the test data set. We found that identifying new modules and changed modules mere significant components of the discriminant model, and improved its performance. The results demonstrate that data on module reuse is a valuable input to quality models and that discriminant analysis can be a useful tool in early identification of fault prone software modules in large telecommunications systems. Model results could be used to identify those modules that would probably benefit from extra attention, and thus, reduce the risk of unexpected problems with those modules.

Attention is given to software service quality measurement in the telecommunications switching environment. Environment dynamics and the composition of customer satisfaction are explained. Measurements are made at gaps in the software delivery process, and management is responsible for the success of the measurement process. It is concluded that software service quality measurement can be a strategic tool that can help increase a supplier's competitiveness.>