Several metrics have been proposed in the past to quantify the effectiveness of a test suite; they are usually types of coverage metrics, because it is sensible to quantify the effectiveness of a test suite by the extent to which it exercises (covers) various syntactic or semantic features of a program. Though no coverage metric has emerged as the gold standard of test suite effectiveness, mutation coverage is usually considered as a reliable measure thereof, because the ability of a test suite to detect program mutations can be an indication of its ability to detect faults. In this paper, we aim to challenge the superiority of mutation coverage, by showing empirically that the same test suite can have vastly different values of mutation coverage depending on the mutation operators that are used to generate mutants.

When we quantify the effectiveness of a test suite by its mutation coverage, we are in fact equating test suite effectiveness with fault detection: to the extent that mutations are faithful proxies of actual faults, it is sensible to consider that the effectiveness of a test suite to kill mutants reflects its ability to detect faults. But there is another way to measure the effectiveness of a test suite: by its ability to expose the failures of an incorrect program (or, equivalently, its ability to give us confidence in the correctness of a correct program). The relationship between failures and faults is tenuous at best: a fault is the adjudged or hypothesized cause of a failure. Whereas a failure is an observable, verifiable, certifiable effect, a fault is someone’s hypothesis about the possible cause of the observed effect. The same failure may be attributed to more than one fault or combination of faults. In this paper we raise two questions: is the ability to detect faults the same as the ability to reveal failures? If not, which is the better measure of test suite effectiveness? We do not give definite answers to these questions, but we use empirical data to challenge some assumptions and show why these questions are worth answering.

Abstract Invariant relations are used to analyze while loops; while their primary application is to derive the function of a loop, they can also be used to derive loop invariants, weakest preconditions, strongest postconditions, sufficient conditions of correctness, necessary conditions of correctness, and termination conditions of loops. In this paper we present two generic invariant relations that capture the semantics of loops whose loop body applies affine transformations on numeric variables.

To repair a program does not mean to make it absolutely correct; it only means to make it more-correct, in some sense, than it is. This distinction has consequences: Given that software products typically have a dozen faults per KLOC and thousands of KLOC’s, program repair tools ought to be designed in such a way as to transform an incorrect program into an incorrect, albeit more-correct, program. In the absence of a concept of relative correctness (the property of a program to be more-correct than another with respect to a specification), program repair methods have resorted to various approximations of absolute correctness. This shortcoming has been concealed by the fact that they are usually validated on programs with a single fault at a time, for which the goals of absolute correctness and relative correctness are indistinguishable. In this paper we discuss how the use of relative correctness can reduce the scale of patch generation and enhance the efficiency, precision and recall of patch validation.

Given that faults are the focus of much software quality assurance (fault avoidance, fault removal, fault tolerance, fault prediction/ forecasting), we argue that a formal definition of faults ought to help us enhance the state of the art in this field. In this paper we consider a formal semantic definition of faults, and explore the insights that this definition gives us, and how these insights can be used in practice. Some of these insights are counter-intuitive, which makes them all the more interesting/useful.

Program repair has been an active research area for over a decade and has achieved great strides in terms of scalable automated repair tools. In this paper we argue that existing program repair tools lack an important ingredient, which limits their scope and their efficiency: a formal definition of a fault, and a formal characterization of fault removal. To support our conjecture, we consider GenProg, an archetypical program repair tool, and modify it according to our definitions of fault and fault removal; then we show, by means of empirical experiments, the impact that this has on the effectiveness and efficiency of thee tool.

Software mutation is a widely used technique of software testing that consists in generating variants of a base program by applying standard modifications to its source code. One of the main obstacles in the use of software mutations is the existence of equivalent mutants, i.e. mutants whose behavior is indistinguishable from the base program, even though their source code is distinct. Despite several decades of research, the identification of equivalent mutants remains an open problem. Rather than attempting to identify individual mutants that are equivalent to the base, we argue that it is often sufficient to estimate the number of equivalent mutants; also, we argue that the number of equivalent mutants depends on two factors that must be considered in the estimation effort, namely the base program and the mutation operators that are used; in this paper, we explore the impact of mutation operators on the number of equivalent mutants.

It is so inconceivable to debug a program without testing it that these two words are used nearly interchangeably. Yet we argue that using the concept of relative correctness we can indeed remove a fault from a program and prove that the fault has been removed, by proving that the new program is more correct than the original. This is a departure from the traditional roles of proving and testing methods, whereby static proof methods are applied to a correct program to prove its correctness, and dynamic testing methods are applied to an incorrect program to expose its faults.

Relative correctness is the property of a program to be more-correct than another with respect to a specification; this property enables us to rank candidate programs in a partial ordering structure whose maximal elements are the correct programs.Whereas traditionally we think of program derivation as a process of successive correctnesspreserving transformations (using refinement) starting from the specification, we argue that it is possible to derive programs by successive correctness-enhancing transformations (using relative correctness) starting from abort.One of the attributes of our approach is that it captures in the same mathematical model, not only the derivation of programs from scratch, but also most (if not all) of the activities that arise in software evolution.Given that most software is developed nowadays by evolving existing products rather than from scratch, any advance in the technology of program transformation by correctness enhancement stands to yield significant practical benefits.

During the last two past decades, software architecture has been a rising subject of software engineering. Since, researchers and practitioners have recognized that analyzing the architecture of a software system is an important part of the software development process. Architectural evaluation not only reduces software development efforts and costs but it also enhances the quality of the software by verifying the addressability of quality requirements and identifying potential risks. To this aim, several approaches have been recently proposed to analyze system non-functional attributes from its software architecture specification. In this paper, we propose an ADL based formal method for representing and reasoning about system non-functional attributes at the architectural level. We are especially interested in analyzing performance and reliability quality attributes. We also propose to analyze the sensitivity of the system by identifying components that have the greatest impact on the system quality. The automation of our model was followed by a series of experiments that allowed us to validate our inductive reasoning to prove the capabilities of our model to represent and analyze software architectures.

In the process of trying to define what is a software fault, we have foundthat to formally define software faults we need to introduce the conceptof relative correctness, i.e. the property of a program to be more-correctthan another with respect to a given specification. A feature of a programis a fault (for a given specification)only because there exists an alternative to it that would makethe program more-correct with respect to the specification.In this paper, we explore applications of the concept of relative correctness in programtesting, program repair, and program design.Specifically, we argue that in many situations of software testing,fault removal and program repair, testing for relative correctnessrather than absolute correctness leads to clearer conclusions andbetter outcomes. Also, we find that designing programs by stepwisecorrectness-enhancing transformations rather than by stepwise correctness-preserving refinements leads to simpler programs and is more tolerant of designer mistakes.

The Supervisory Control and Data Acquisition (SCADA) system discussed in this work manages a distributed control network for the Tunisian Electric & Gas Utility. The network is dispersed over a large geographic area that monitors and controls the flow of electricity/gas from both remote and centralized locations. The availability of the SCADA system in this context is critical to ensuring the uninterrupted delivery of energy, including safety, security, continuity of operations and revenue. Such SCADA systems are the backbone of national critical cyber-physical infrastructures. Herein, we propose adapting the Mean Failure Cost (MFC) metric for quantifying the cost of unavailability. This new metric combines the classic availability formulation with MFC. The resulting metric, so-called Econometric Availability (EA), offers a computational basis to evaluate a system in terms of the gain/loss ($/hour of operation) that affects each stakeholder due to unavailability.

Even though nothing is ever that simple/ clear cut in software engineering, one would argue that by contrast with source code, which represents the detailed functional attributes of a software product, the architecture of a software product represents its non functional attributes. The architecture of a software product determines such attributes as its response time, throughput, reliability, security, maintainability, availability, etc. In this paper we present a model for reasoning about quantitative non functional attributes of software architectures, and discuss our plan to develop automated support for this model.

Despite several decades of on-and-off research efforts, the functional analysis of computer programs to an arbitrary level of thoroughness and precision remains an unfulfilled challenge. Also, despite several decades of advances in programming language design and implementation, most of the software that is written and maintained today is written in traditional imperative programs; in such programs, the most critical/ most complex control structure is the loop. In this paper, we discuss our ongoing effort to develop a tool that reviews a while loop written in an imperative language (C, C++, Java) and derives its function, by specifying the final values of all its program variables as a function of its initial values.

In this short paper we discuss the design, implementation and operation of an automated tool that computes the function of while loops written in C-like programming languages.

Current trends indicate that IT security measures will need to greatly expand to counter the ever increasingly sophisticated, well-funded and/or economically motivated threat space. Traditional risk management approaches provide an effective method for guiding courses of action for assessment, and mitigation investments. However, such approaches no matter how popular demand very detailed knowledge about the IT security domain and the enterprise/cyber architectural context. Typically, the critical nature and/or high stakes require careful consideration and adaptation of a balanced approach that provides reliable and consistent methods for rating vulnerabilities. As reported in earlier works, the Cyberspace Security Econometrics System provides a comprehensive measure of reliability, security and safety of a system that accounts for the criticality of each requirement as a function of one or more stakeholders' interests in that requirement. This paper advocates a dependability measure that acknowledges the aggregate structure of complex system specifications, and accounts for variations by stakeholder, by specification components, and by verification and validation impact.

The development, certification and evolution of dependable software requires the ability to analyze software artifacts in all their extensive detail. This, in turn, is contingent upon availability of reliable, certified tools that can rigorously analyze the behavior and properties of software artifacts. One of the most difficult challenges in the development of such a tool is the ability to derive the function of a loop from a static analysis of its source code. In this paper, we discuss the main tenets of our approach to this problem, based on a relation-theoretic refinement calculus, and outline its results, insights, and prospects.

Redundancy is a system property that generally refers to duplication of state information or system function. While redundancy is usually investigated in the context of fault tolerance, one can argue that it is in fact an intrinsic feature of a system that can be analyzed on its own without reference to fault tolerance. Redundancy may arise by design, generally to support fault tolerance, or as a natural byproduct of design, and is usually unexploited. In this paper, we tentatively explore observable forms of redundancy, as well as mathematical models that capture them.

The study of software architectures is gaining importance due to its role in various aspects of software engineering, like product line engineering, componentbased software engineering and other emerging paradigms. With the increasing emphasis on design patterns, the traditional practice of ad-hoc software construction is slowly shifting towards pattern-oriented development. Various architectural attributes like error propagation, change propagation, requirements propagation provide a wealth of information about software architectures. In this paper, we show that change propagation probability (CP) is helpful and effective in assessing the design quality of software architectures. We study two different architectures (one that employs patterns versus one that does not) for the same application. We also analyze and compare the change propagation metric with respect to other coupling-based metrics.

Summary form only given. Software architectures are an emerging discipline in software engineering as they play a central role in many modern software development paradigms. Quantifying software architectures is an important research agenda, as it allows software architects to subjectively assess quality attributes and rationalize architecture-related decisions. In this paper, we discuss the attribute of change propagation probability, which reflects the likelihood that a change that arises in one component of the architecture propagates (i.e. mandates changes) to other components.

The study of software architectures is gaining importance due to its role in various aspects of software engineering such as product line engineering, component based software engineering and other emerging paradigms. With the increasing emphasis on design patterns, the traditional practice of ad-hoc software construction is slowly shifting towards pattern-oriented development. Various architectural attributes like error propagation, change propagation, and requirements propagation, provide a wealth of information about software architectures. In this paper, we show that change propagation probability (CP) is helpful and effective in assessing the design quality of software architectures. We study two different architectures (one that employs patterns versus one that does not) for the same application. We also analyze and compare change propagation metric with respect to other coupling-based metrics.

To assist the analysis software architectures, we have developed a Web-based CASE tool that automates the steps of capturing, modeling, and inspection of software architectures, in order to derive and display the change propagation probabilities.

Risk assessment is an essential part in managing software development. Performing risk assessment during the early development phases enhances resource allocation decisions. In order to improve the software development process and the quality of software products, we need to be able to build risk analysis models based on data that can be collected early in the development process. These models will help identify the high-risk components and connectors of the product architecture, so that remedial actions may be taken in order to control and optimize the development process and improve the quality of the product. In this paper, we present a risk assessment methodology which can be used in the early phases of the software life cycle. We use the Unified Modeling Language (UML) and commercial modeling environment Rational Rose Real Time (RoseRT) to obtain UML model statistics. First, for each component and connector in software architecture, a dynamic heuristic risk factor is obtained and severity is assessed based on hazard analysis. Then, a Markov model is constructed to obtain scenarios risk factors. The risk factors of use cases and the overall system risk factor are estimated using the scenarios risk factors. Within our methodology, we also identify critical components and connectors that would require careful analysis, design, implementation, and more testing effort. The risk assessment methodology is applied on a pacemaker case study.

Because it codifies best practices, and because it supports various forms of software reuse, the discipline of software architecture is emerging as an important branch of software engineering research and practice. Because architectural-level decisions are prone to have a profound impact on finished software products, it is important to apprehend their quality attributes and to quantify them (as much as possible). In this paper, we discuss an information-theoretic approach to the definition and validation of architectural metrics, and illustrate our approach on a sample example.

We present a formal specification of fault tolerance requirements for an analytical redundancy based fault tolerant flight control system. The development of the specification is driven by the performance and fault tolerance requirements contained in the US Air Force military specification MIL-F-9490D. The design constraints imposed to the system from adopting the analytical redundancy approach are captured within the specification. We draw some preliminary conclusions from our study.

Echoing Louis Pasteur's quote, we submit the premise that it is advantageous to define measures of distance between requirements specifications because such measures open up a wide range of possibilities both in theory and in practice. The authors present a mathematical basis for measuring distances between specifications and show how their measures of distance can be used to address concrete problems that arise in the practice of software engineering.

Several cost models have been proposed in the past for estimating, predicting, and analyzing the costs of software reuse. In this paper we analyze existing models, explain their variance, and propose a tool-supported comprehensive model that encompasses most of the existing models.

No abstract available.

No abstract available.

Fault avoidance, fault removal and fault tolerance represent three successive lines of defense against the contingency of faults in software systems and their impact on system reliability. Beyond the colorful discussions of the relative merits of these techniques, the law of diminishing returns advocates that they be used in concert, where each is applied whenever it is most effective. Such a premise remains an idle act of faith so long as these techniques cannot be captured by a uniform model. This paper proposes such a model and illustrates how it can be used in practice to improve the quality of software products.

We give a formal relation-based definition of scenarios and we show how different scenarios can be integrated to obtain a more global view of user-system interactions. We restrict ourselves to the sequential case, meaning that we suppose that there is only one user (thus, the scenarios we wish to integrate cannot occur concurrently). Our view of scenarios is state-based, rather than event-based, like most of the other approaches, and can be grafted to the well-established specification language Z. Also, the end product of scenario integration, the specification of the functional aspects of the system, is given as a relation; this specification can be refined using independently developed methods. Our formal description is coupled with a diagram-based, transition-system like, presentation of scenarios, which is better suited to communication between clients and specifiers.

This paper discusses an undergraduate option in software engineering which is currently in the approval process at the University of Ottawa. The proposed option represents a close collaboration between the Department of Computer Science and the Department of Electrical and Computer Engineering. In this paper we present an analysis of the weaknesses of the graduates of existing programs and some criteria for the choice of courses in the new option. In particular, we examine why a significant amount of additional project-based experience should be required in the software engineering option. The resulting option differs from those at other universities in that it has a somewhat greater emphasis on both human factors and the object oriented approach.

Given a software library whose entries are represented by formal specifications, we distinguish between two retrieval procedures: exact retrieval, whereby, given a query K, we identify all the library components that are correct with respect to K; approximate retrieval, which is invoked when exact retrieval fails, and identifies the library components that minimize adaptation effort. To this effect, we define four measures of functional distance between specifications, and discuss algorithms that minimize these measures over a set of components; then we discuss whether these measures can be used to predict adaptation effort.

Software libraries are repositories which contain software components; as such, they represent a precious resource for the software engineer. As software libraries grow in size, it becomes increasingly difficult to maintain adequate precision and recall with informal retrieval algorithms. In this paper, we discuss the design and implementation of a storage and retrieval structure for software components that is based on formal specifications and on the refinement ordering between specifications.

Software productivity has been steadily increasing over the past 30 years, but not enough to close the gap between the demands placed on the software industry and what the state of the practice can deliver; nothing short of an order of magnitude increase in productivity will extricate the software industry from its perennial crisis. Several decades of intensive research in software engineering and artificial intelligence left few alternatives but software reuse as the (only) realistic approach to bring about the gains of productivity and quality that the software industry needs. In this paper, we discuss the implications of reuse on the production, with an emphasis on the technical challenges. Software reuse involves building software that is reusable by design and building with reusable software. Software reuse includes reusing both the products of previous software projects and the processes deployed to produce them, leading to a wide spectrum of reuse approaches, from the building blocks (reusing products) approach, on one hand, to the generative or reusable processor (reusing processes), on the other. We discuss the implication of such approaches on the organization, control, and method of software development and discuss proposed models for their economic analysis. Software reuse benefits from methodologies and tools to: (1) build more readily reusable software and (2) locate, evaluate, and tailor reusable software, the last being critical for the building blocks approach. Both sets of issues are discussed in this paper, with a focus on application generators and OO development for the first and a thorough discussion of retrieval techniques for software components, component composition (or bottom-up design), and transformational systems for the second. We conclude by highlighting areas that, in our opinion, are worthy of further investigation.>

We discuss our experience tvith integrating three key ideas of softurare engineen"ng into our data structures course: formal specifications, information hiding, and bottom up programming.

Abstract In this paper we investigate the lattice properties of the natural ordering between specifications, which expresses that a specification expresses a stronger requirement than another specification. The lattice-like structure that we uncover is used as a basis for a specification methodology.

An empirical study of the management of software maintenance is presented, based on completion of an earlier study. The survey at hand has been conducted by means of live interviews with analysts and software managers, using a translated version of B.P. Lientz and E.B. Swanson's (1978) survey form (slightly adjusted at times to take into account a difference of data processing culture). The authors accumulated a total of 150 completed survey forms, properly distributed on the various sectors of the Tunisian data processing industry. For the sake of comparison or confirmation, the authors mention a similar study that was carried out by one of the authors with some companies in Algeria. This study covers a total of sixty surveys, taken in 1988 and 1989. The final results are presented. The issues discussed are scale of effect issues, organizational issues, productivity technique issues and problem area issues.>

A recent statistical study of the various economic aspects of software maintenance management in Tunisia is presented. The study confirms a well-known results, that software maintenance is very time consuming. It was also found that software maintenance escapes the software manager's notice. In the study, only one in ten managers considers maintenance problems important or very important. In addition, it was shown that the longevity of software applications does not depend on the data processing environment. It is concluded that software maintenance is controllable. Combined with appropriately calibrated cost-estimation techniques, organizational steps developed by the community of software practitioners can be arranged into an integrated policy for software maintenance.>

Abstract The effectiveness of set theoretic concepts for the purpose of software specification is becoming more and more widely recognized. This paper presents an integrated relational methodology for software specification, and shows its effectivness when applied to a practical example.

When the state of a program in execution is accidentally altered, a recovery action may be needed before the execution can proceed on. Two approaches exist for the design of recovery actions: backward recovery consists of retrieving a previously saved correct state and restarting the computation; forward recovery consists of generating á (sufficiently) correct state from the current (not too) contaminated state. This paper presents a tentative framework for the study of forward error recovery and then discusses some preliminary results and some future research within the proposed framework.

Teaching programming to first year classes is most often considered a matter of minor importance. We think otherwise. This paper expresses our viewpoint on the matter and reports on an experiment supporting our viewpoint.

Because of its relevance to program analysis and, ultimately, to program synthesis, the area of formal verification of program correctness is an important part of the education of the programmer and the computer scientist. A case is presented in this paper to advocate the teaching of the formal verification of computer programs. This case is based on three major arguments:

Self-checking programs are programs which meet the following condition. For any legal input, either they return the correct output or they return a message indicating that the output may be incorrect. Self-checking programs are capable of recognizing irregular conditions in their state space and reporting it. Self-stabilizing programs are programs which, in addition to (or instead of) reporting irregular conditions in their state space, take action to recover from them. This correspondence describes how to formally verify that a program is self-checking or self-stabilizing and discusses the relationship between the thoroughness of recovery actions in a self-stabilizing program and the overall quality of the program.