Traditional anomaly detectors in water treatment and distribution plants identify process anomalies only after their impact has been realized, limiting their ability to prevent disruptions or damage. This paper introduces the Programmable Logic Controller (PLC) Command Validation Tool (PCAT), an anomaly-avoidance framework that verifies the correctness of control commands issued by PLCs before they reach actuators. PCAT was evaluated on the Secure Water Treatment (SWaT) testbed under realistic attack scenarios inspired by real-world incidents. The framework demonstrated the ability to raise alerts significantly earlier than existing methods—triggering one alert 2.5 seconds in advance—thereby effectively preventing anomalies before physical damage occurred. Over six hours of normal operation, PCAT recorded zero false positives, confirming its precision and reliability. These findings position PCAT as a proactive and effective solution for securing water treatment operations.

Humans have a natural ability to perform semantic associations with the surrounding objects in the environment. This allows them to create a mental map of the environment, allowing them to navigate on-demand when given linguistic instructions. A natural goal in Vision Language Navigation (VLN) research is to impart autonomous agents with similar capabilities. Recent works take a step towards this goal by creating a semantic spatial map representation of the environment without any labeled data. However, their representations are limited for practical applicability as they do not distinguish between different instances of the same object. In this work, we address this limitation by integrating instance-level information into spatial map representation using a community detection algorithm and utilizing word ontology learned by large language models (LLMs) to perform open-set semantic associations in the mapping representation. The resulting map representation improves the navigation performance by two-fold (233%) on realistic language commands with instance-specific descriptions compared to the baseline. We validate the practicality and effectiveness of our approach through extensive qualitative and quantitative experiments.

We are pleased to present the papers selected for inclusion in the special issue devoted to the 1st International Conference on Automation of Software Test, which was held virtually in colocation with the 42nd IEEE/ACM International Conference on Software Engineering (ICSE 2020). Software testing is an integral and important part of the software engineering (SE) discipline. Over the past decades, a significant amount of SE research has focused on automation of software test (AST), including the automation of test case generation, test case selection and prioritization, test execution, test verdict analysis, and debugging. AST practice has also moved forward significantly, and in recent years, many test tools, frameworks, and methodologies have been developed and have significantly enhanced the quality assurance and the productivity in SE practices. Despite the significant achievements, AST remains challenging. To achieve total automation of software testing, a huge amount of code for the test cases and the supporting test infrastructure is needed, which yields itself in turn to large maintenance costs. However, if on the one side it is now generally accepted that disciplined procedures and quality standards should be applied in production code development, on the other side, comparable levels of rigor and quality are not demanded for the code written for testing that production code. Indeed, several recent empirical studies point to how test code is affected by many problems, bugs, unjustified assumptions, hidden dependencies from other tests or environment, flakiness, and performance issues. Because of such problems, test effectiveness is impacted and several false alarms are raised in regression testing that increase the test costs. Recently, both researchers and practitioners have proposed solutions toward this problem by identifying test code smells or test code quality issues and providing techniques to automatically detect and repair test code bugs and flakiness. In consideration of this active research thread, the 1st ACM/IEEE International Conference on Automation of Software Test (AST 2020) featured “Who Tests The Tests?” as the conference theme. AST 2020 was run for the first time in the format of a colocated conference to ICSE, after a successful series of 14 workshops under the same name. This special issue offers a venue for researchers and practitioners to share the advances in software test automation, especially on test code quality. In particular, we invited the authors of the best papers of AST 2020 to submit extended versions of their conference publications. Moreover, we also encouraged the authors and participants of the AST series and the broader SE community to submit novel original papers on the themes of software test automation and test code quality, including approaches for the following: understanding the dimensions and characteristics of problems with unreliable, low-quality test code; identifying and preventing test code smells, test code bugs, and flaky tests; impact of test code quality in scaling up test automation of very large, complex system; metrics for test code quality and robustness; automated repair of test code bugs and flakiness; and employing Artificial Intelligence and Machine Learning methods to help test automation. Each of the 10 submitted papers underwent a rigorous review process by independent referees to ensure that the paper had a sound, novel, and original contribution to the field of software test automation. Finally, five papers were accepted for inclusion in the special issue; three of these are extended versions of the best papers of AST 2020, and two are novel external submissions. In particular, the paper titled “Quantum Software Testing—State of the Art,” by Antonio García de la Barrera, Ignacio García-Rodríguez de Guzmán, Macario Polo, and Mario Piattini, provides a systematic mapping study assessing the state of the art in testing of quantum computing applications, indeed an emerging new paradigm that promises exponential speed up in solving highly demanding computational problems. In “An Empirical Study on How Sapienz Achieves Coverage and Crash Detection”, Iván Arcuschin, Juan Pablo Galeotti, and Diego Garbervetsky report the results from an empirical study aiming at better understanding how the main features of Sapienz, a powerful tool for automated testing of Android applications using evolutionary algorithms, impact its effectiveness. Many program analysis and testing tools need to incorporate string solver mechanisms. After observing that adequate tools and benchmarks for comparing existing solvers were lacking, in “ZaligVinder: A Generic Test Framework for String Solvers”, Mitja Kulczynski, Florin Manea, Dirk Nowotka, and Danny Bøgsted Poulsen propose an extensible framework gathering several string solver benchmarks that can be used for analysis and debugging purposes. “ExVivoMicroTest: Ex-Vivo Testing of Microservices” by Luca Gazzola, Maayan Goldstein, Leonardo Mariani, Marco Mobilio, Itai Segall, Alessandro Tundo, and Luca Ussi focuses on ex vivo testing of microservices during deployment. The authors claim that regression testing of such services may not be adequate prior to deployment due to a lack of knowledge regarding new use scenarios. The authors then propose a technique, named ExVivoMicroTest, that analyzes the behavior of deployed microservices and generates test cases for testing future updates. In “Fight Silent Horror Unit Test Methods by Consulting a TestWizard”, Maura Cerioli, Giovanni Lagorio, Maurizio Leotta, and Filippo Ricca focus on a practical problem, that is, the existence of incorrect tests. While a totally automated solution to identify invalid tests seems impractical, the idea of TestWizard is to assess individual tests' quality from the point of view of their coherence to specifications. We would like to thank the Editors-in-Chief of the Journal of Software: Evolution and Process for giving us the opportunity to publish this special issue. We appreciate all reviewers for their efforts in providing thoughtful and constructive comments to improve the quality of the publications. We also thank the authors of all submissions and publications of this special issue.

A method referred to as PbNN is proposed to detect cyber-physical attacks through the identification of resulting anomalies in the process dynamics of the underlying ICS. Unlike existing anomaly detectors based on an abstract knowledge acquired from operational data, PbNN utilizes the design knowledge of ICS to learn the complex relationships among the correlated components. Such relationships are accurately modeled using operational data through the application of the deep convolution neural network. The proposed detector was implemented and evaluated in an operational secure water treatment plant by launching several real-time stealthy and coordinated attacks. The results indicate that PbNN outperforms the existing state-of-the-art machine learning anomaly detectors when compared using detection accuracy and the rate of false alarms.

Programmable Logic Controllers (PLCs) are a core component of an Industrial Control System (ICS). However, if a PLC is compromised or the commands sent across a network from the PLCs are spoofed, consequences could be catastrophic. In this work, a novel technique to authenticate PLCs is proposed that aims at raising the bar against powerful attackers while being compatible with real-time systems. The proposed technique captures timing information for each controller in a non-invasive manner. It is argued that Scan Cycle is a unique feature of a PLC that can be approximated passively by observing network traffic. An attacker that spoofs commands issued by the PLCs would deviate from such fingerprints. To detect replay attacks a PLC Watermarking technique is proposed. PLC Watermarking models the relation between the scan cycle and the control logic by modeling the input/output as a function of request/response messages of a PLC. The proposed technique is validated on an operational water treatment plant (SWaT) and smart grid (EPIC) testbeds. Results from experiments indicate that PLCs can be distinguished based on their scan cycle timing characteristics.

A cyber-physical attack against critical infrastructures, such as water treatment and distribution plants, could lead to process anomalies. Several design and data centric approaches were developed to detect such anomalies when the physical processes of the underlying plant move from a normal to a malicious state. Although these approaches are necessary for the continued and reliable plant operation, they might not be sufficient to prevent service disruption or damage to components. This is because the impact of anomalies is already realized before the attack is detected. In this paper, we propose PCAT – PLC Command Analysis Tool that validates the control commands issued by the programmable logic controller (PLC). PCAT stops rogue attack commands before they reach the target actuators. In a case study, PCAT was deployed and validated on the operational water treatment plant named SWaT. The experimental results attest the performance of PCAT in ensuring the security, safety, and service status of the plant before it enters an anomalous state.

Abstract Modern critical infrastructure, such as a water treatment plant, water distribution system, and power grid, are representative of Cyber Physical Systems (CPSs) in which the physical processes are monitored and controlled in real time. One source of complexity in such systems is due to the intra-system interactions and inter-dependencies. Consequently, these systems are a potential target for attackers. When one or more of these infrastructure are attacked, the connected systems may also be affected due to potential cascading effects. In this paper, we report a study to investigate the cascading effects of cyber-attacks on two interdependent critical infrastructure namely, a Secure water treatment plant (SWaT) and a Water Distribution System (WADI).

Abstract Gradual increase in the number of successful attacks against Industrial Control Systems (ICS) has led to an urgent need to create defense mechanisms for accurate and timely detection of the resulting process anomalies. Towards this end, a class of anomaly detectors, created using data-centric approaches, are gaining attention. Using machine learning algorithms such approaches can automatically learn the process dynamics and control strategies deployed in an ICS. The use of these approaches leads to relatively easier and faster creation of anomaly detectors compared to the use of design-centric approaches that are based on plant physics and design. Despite the advantages, there exist significant challenges and implementation issues in the creation and deployment of detectors generated using machine learning for city-scale plants. In this work, we enumerate and discuss such challenges. Also presented is a series of lessons learned in our attempt to meet these challenges in an operational plant.

Industrial Control Systems (ICS) monitor and control physical processes. ICS are found in, among others, critical infrastructures such as water treatment plants, water distribution systems, and the electric power grid. While the existence of cyber-components in an ICS leads to ease of operations and maintenance, it renders the system under control vulnerable to cyber and physical attacks. An experimental study was conducted withreplay attackslaunched on an operational water distribution (WADI) plant to understand under what conditions an attacker/attack can remain undetected while stealing water. A detection method, based on an input-output Linear Time-invariant system model of the physical process, was developed and implemented in WADI to detect such attacks. The experiments reveal the strengths and limitations of the detection method and challenges faced by an attacker while attempting to steal water from a water distribution system.

The rise in attempted and successful attacks on critical infrastructure, such as power grid and water treatment plants, has led to an urgent need for the creation and adoption of methods for detecting such attacks often launched either by insiders or state actors. This paper focuses on one such method that aims at the detection of attacks that compromise one or more actuators and sensors in a plant either through successful intrusion in the plant's communication network or directly through the plant computers. The method, labelled as Distributed Attack Detection (DAD), detects attacks in real-time by identifying anomalies in the behavior of the physical process in the plant. Anomalies are identified by using monitors that are implementations of invariants derived from the plant design. Each invariant must hold either throughout the plant operation, or when the plant is in a given state. The effectiveness of DAD was assessed experimentally on an operational water treatment plant named SWaT that is a near-replica of commercially available large treatment plants. The method used in DAD was found to be effective in detecting stealthy and coordinated attacks.

A hackfest named SWaT Security Showdown (S3) has been organized consecutively for two years. S3has enabled researchers and practitioners to assess the effectiveness of methods and products aimed at detecting cyber attacks launched in real-time on an operational water treatment plant, namely, Secure Water Treatment (SWaT). In S3, independent attack teams design and launch attacks on SWaT while defence teams protect the plant passively and raise alarms upon attack detection. Attack teams are scored according to how successful they are in performing attacks based on specific intents while the defense teams are scored based on the effectiveness of their methods to detect the attacks. This paper focuses on the first two instances of S3and summarizes the benefits of hackfest and the performance of an attack detection mechanism, named Water Defense, that was exposed to attackers during S3.

Fingerprinting of various physical and logical devices has been proposed for uniquely identifying users or devices of mainstream IT systems such as PCs, laptops, and smart phones. However, the application of such techniques in Industrial Control Systems (ICS) is less explored for reasons such as a lack of direct access to such systems and the cost of faithfully reproducing realistic threat scenarios. This work addresses the feasibility of using fingerprinting techniques in the context of realistic ICS related to water treatment and distribution systems. A model-free sensor fingerprinting scheme ( NoiSense ) and a model-based sensor fingerprinting scheme ( NoisePrint ) are proposed. Using extensive experimentation with sensors, it is shown that noise patterns due to microscopic imperfections in hardware manufacturing can uniquely identify sensors with accuracy as high as 97%. The proposed technique can be used to detect physical attacks, such as the replacement of legitimate sensors by faulty or manipulated sensors. For NoisePrint , a combined fingerprint for sensor and process noise is created. The difference (called residual), between expected and observed values, i.e., noise, is used to derive a model of the system. It was found that in steady state the residual vector is a function of process and sensor noise. Data from experiments reveals that a multitude of sensors can be uniquely identified with a minimum accuracy of 90% based on NoisePrint . Also proposed is a novel challenge-response protocol that exposes more powerful cyber-attacks, including replay attacks.

A particularly good software testing strategy is to achieve the underlying testing goal while solving the problems of tradeoffs between testing effectiveness and efficiency. To improve the fault detection effectiveness of software testing, the principle of feedback control theory was adopted, which motivated the proposal of dynamic random testing (DRT). The main idea behind DRT is using the testing results to guide the test case selection to increase the selection probabilities of the subdomains with higher fault detection rates. Previous works show that DRT strategy can achieve better effectiveness than random testing strategy and random partition testing strategy, and has significantly lower computational costs than adaptive testing strategy. However, the essential factors that affect the performance of DRT, i.e., adjusting parameters, initial profile, and test case classification have not been thoroughly investigated. Besides, some experimental assumptions are inconsistent with real scenarios. Therefore, this paper gives a series of investigations on DRT with a set of practical subject programs. More specifically, the effectiveness and efficiency of DRT are presented, and the extended experiments on DRT with relevant factors are conducted. The results indicate that the effectiveness of DRT is robust to different initial profiles and affected noticeably by the adjusting parameter settings and test case classification methods.

A novel scheme is proposed to authenticate sensors and detect data integrity attacks in a Cyber Physical System (CPS). The proposed technique uses the hardware characteristics of a sensor and physics of a process to create unique patterns (herein termed as fingerprints) for each sensor. The sensor fingerprint is a function of sensor and process noise embedded in sensor measurements. Uniqueness in the noise appears due to manufacturing imperfections of a sensor and due to unique features of a physical process. To create a sensor's fingerprint a system-model based approach is used. A noise-based fingerprint is created during the normal operation of the system. It is shown that under data injection attacks on sensors, noise pattern deviations from the fingerprinted pattern enable the proposed scheme to detect attacks. Experiments are performed on a dataset from a real-world water treatment (SWaT) facility. A class of stealthy attacks is designed against the proposed scheme and extensive security analysis is carried out. Results show that a range of sensors can be uniquely identified with an accuracy as high as 98%. Extensive sensor identification experiments are carried out on a set of sensors in SWaT testbed. The proposed scheme is tested on a variety of attack scenarios from the reference literature which are detected with high accuracy

An attack detection scheme is proposed to detect data integrity attacks on sensors in Cyber-Physical Systems (CPSs). A combined fingerprint for sensor and process noise is created during the normal operation of the system. Under sensor spoofing attack, noise pattern deviates from the fingerprinted pattern enabling the proposed scheme to detect attacks. To extract the noise (difference between expected and observed value) a representative model of the system is derived. A Kalman filter is used for the purpose of state estimation. By subtracting the state estimates from the real system states, a residual vector is obtained. It is shown that in steady state the residual vector is a function of process and sensor noise. A set of time domain and frequency domain features is extracted from the residual vector. Feature set is provided to a machine learning algorithm to identify the sensor and process. Experiments are performed on two testbeds, a real-world water treatment (SWaT) facility and a water distribution (WADI) testbed. A class of zero-alarm attacks, designed for statistical detectors on SWaT are detected by the proposed scheme. It is shown that a multitude of sensors can be uniquely identified with accuracy higher than 90% based on the noise fingerprint.

Industrial Control Systems (ICS) such as water and power are critical to any society. Process anomaly detection mechanisms have been proposed to protect such systems to minimize the risk of damage or loss of resources. In this paper, a graphical model-based approach is proposed for profiling normal operational behavior of an operational ICS referred to as SWaT (Secure Water Treatment). Timed automata are learned as a model of regular behaviors shown in sensors signal like fluctuations of water level in tanks. Bayesian networks are learned to discover dependencies between sensors and actuators. The models are used as a one-class classifier for process anomaly detection, recognizing irregular behavioral patterns and dependencies. The detection results can be interpreted and the abnormal sensors or actuators localized due to the interpretability of the graphical models. This approach is applied to a dataset collected from SWaT. Experimental results demonstrate the model's superior performance on both precision and run-time over methods including support vector machine and deep neural networks. The underlying idea is generic and applicable to other industrial control systems such as power and transportation.

An increase in the number of attacks on cyberphysical systems (CPS) has raised concerns over the vulnerability of critical infrastructure such as water treatment, oil, gas plants, against cyber attacks. Such systems are controlled by an Industrial Control System (ICS) that includes controllers communicating with each other, and with physical sensors and actuators, using a communications network. This paper focuses on a Multiple Security Domain Nondeducibility (MSDND) model to identify the vulnerable points of attack on the system that hide critical information rather than steal it, such as in the STUXNET virus. It is shown how MSDND analysis, conducted on a realistic multi-stage water treatment testbed, is useful in enhancing the security of a water treatment plant. Based on the MSDND analysis, this work offers a thorough documentation on the vulnerable points of attack, invariants used for removing the vulnerabilities, and suggested design decisions that help in developing invariants to mitigate attacks.

Effectiveness of seven methods for detecting stealthy attacks on Cyber Physical Systems (CPS) was investigated using an experimental study. The Amigobot robot was used as the CPS. The experiments were conducted in simulation as well as on the physical robot. Three types of stealthy attacks were implemented: surge, bias, and geometric. Two variations of Cumulative Sum (CUSUM) method for detecting attacks were evaluated: partial and full physics. Four attack scenarios were implemented. Results from the experiments indicate that stealthy attacks could remain undetected by the CUSUM methods for some attack scenarios. In addition to the CUSUM-based methods, a set of five methods to complement CUSUM were implemented and their effectiveness assessed. While the additional methods do improve the effectiveness of CUSUM-based methods, some attacks remained undetected regardless of which method, or a combination of methods, was used for detection due to the amount of variation in sensor measurements between different runs in simulation and in the physical robot.

Industrial control systems (ICS) include devices, systems, networks and controls to operate industrial processes. ICS are found in many critical infrastructure systems such as transportation, energy and water processing. Given the wide proliferation of ICS, there is a heightened risk of cyber attacks on such infrastructure. It is well known that many such ICS lack appropriate access control, and there are guidelines on what such controls should be. However, there is a lack of case studies that point to specific access control shortfalls in real systems, relate such shortfalls to how an ICS could be compromised, and propose enhancements. This paper reports one such case study conducted on a water distribution plant built by professionals, and to professional standards. The study points to specific instances of inadequacy of access control in such systems that allow malicious entities to compromise system security. At the end of the study a comparison is presented between NIST Guidelines and the state of the water distribution system.

A distributed detection method is proposed to detect single stage multi-point (SSMP) attacks on a Cyber Physical System (CPS). Such attacks aim at compromising two or more sensors or actuators at any one stage of a CPS and could totally compromise a controller and prevent it from detecting the attack. However, as demonstrated in this work, using the flow properties of water from one stage to the other, a neighboring controller was found effective in detecting such attacks. The method is based on physical invariants derived for each stage of the CPS from its design. The attack detection effectiveness of the method was evaluated experimentally against an operational water treatment testbed containing 42 sensors and actuators. Results from the experiments point to high effectiveness of the method in detecting a variety of SSMP attacks but also point to its limitations. Distributing the attack detection code among various controllers adds to the scalability of the proposed method.

An attacker model is proposed for Cyber Physical Systems (CPS). The attack models derived from the attacker model are used to generate parameterized attack procedures and functions that target a specific CPS. The proposed models capture both physical and cyber attacks and unify a number of existing attack models into a common framework useful for researchers in the experimental assessment of attack detection techniques. The generality of the models is shown by mapping a broad variety of existing attack models to the models proposed here, as well as generating attacks that are not found in the CPS design literature. The models have been used extensively in understanding the impact of cyber attacks on a water treatment system and in the design and assessment of detection mechanisms.

Complex and critical public infrastructure includes systems for water treatment, water distribution, power generation and distribution, and private and mass transportation. Such systems share one design characteristic: they include a complex cyber component to control a complex physical component. The cyber component is a source of attraction for people with malicious intent; and the number of attempts and successes to cripple, disrupt, or damage critical infrastructure continues to increase. This talk will focus on (a) novel attacker models for public infrastructure, (b) detection methods for collaborative smart replay and extreme attacks, and (c) a network of interconnected testbeds for understanding the impact of cyber attacks and validation of attack detection and post-detection control schemes.

An experiment was conducted to investigate the effectiveness of the Cumulative Sum (CUSUM) approach for detecting cyber attacks on Cyber Physical Systems (CPS). The Amigobot robot was used as the CPS in this study. Three types of stealthy attacks were considered, namely, surge, bias, and geometric. While a similar study has been reported earlier using a simulated chemical plant, the objective of the study reported here was to replicate the previous study in a realistic CPS environment and investigate whether the detection method performs differently. Cyber attacks were implemented on the Amigobot through its wireless control mechanism by changing the readings obtained from one of its sonar sensors. In addition, the investigation focused on understanding the impact of attack timing and duration on (a) detection effectiveness of the CUSUM method and (b) system safety. Analysis of experimental data indicates differences between results reported in the previous simulation-based study and those reported here.

According to a study commissioned by the National Institute of Standards and Technology in 2002, software bugs cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 percent of the nation's gross domestic product (GDP). The same study also found that more than one-third of these costs, or an estimated $22.2 billion, could be eliminated by an improved testing infrastructure. These numbers would be significantly higher if the study were conducted today.

Expert search has made rapid progress in modeling, algorithms and evaluations in the recent years. However, there is very few work on analyzing how users interact with expert search systems. In this paper, we conduct analysis of an expert search query log. The aim is to understand the special characteristics of expert search usage. To the best of our knowledge, this is one of the earliest work on expert search query log analysis. We find that expert search users generally issue shorter queries, more common queries, and use more advanced search features, with fewer queries in a session, than general Web search users do. This study explores a new research direction in expert search by analyzing and exploiting query logs.

This special issue contains extended versions of three papers from the first IEEE International Conference on Software Testing Verification and Validation (ICST 2008). These three papers were selected based on reviews from members of the program committee and subsequently subjected to another round of review and revision. All three papers focus on automated test generation and include significant empirical evaluation. However, they apply very different approaches to automation: model based, random, and white-box. The first paper by Ciupa, Pretschner, Oriol, Leitner, and Meyer offers an empirical investigation of a variant of random testing applied to object-oriented software. The nature of object-oriented systems complicates random testing since there is a need to introduce objects as well as to randomly generate input. The authors applied an approach in which an object was produced using a constructor call followed by routines that changed the object's state. This has the advantage of only generating valid objects. The (Eiffel) software to which random testing was applied contained contracts and these provided the oracle. The authors performed multiple runs of the same duration and examined the faults found. Interestingly, they found that the actual number of faults found did not vary too much but that different runs found different faults. The authors also classified faults and investigated the faults found by manual testing, random testing, and those in user incident reports. They discovered that these processes found different types of faults and hence are complementary. The second paper, by Bardin and Herrmann, investigates automated testing based on machine code rather than, for example, source code. One of the main motivations for this work is that often source code is not available, especially if parts of development have been outsourced. An additional motivation is that the machine code is closer, than the source code, to the physical system that actually runs. The authors observe that machine code is more difficult to analyse than source code in structured programming languages since, for example, the control flow is not explicit. However, they show that path-based approaches can be adapted and describe a tool (OSMOSE) that achieves this. The approach essentially can be seen as finding the path predicate (precondition) for a suitable path through the machine code and then solving this to find test input. The tool combines static and dynamic analysis in a manner inspired by concolic execution. Finally, they report on the results of experiments using OSMOSE. Our final paper by Grieskamp, Kicillof, Stobie, and Braberman discusses the development of interoperability documentation for Windows protocols within Microsoft and the use of test-driven approaches to verify that the documents accurately represent the protocols. The approaches used not only include manual testing but also a significant amount of model-based testing (MBT) using Microsoft's tool Spec Explorer. Vendors in India and China carried out the testing. A major focus of the paper is the use of MBT in this project. There are two main positive messages regarding MBT. First, it was found that the testers were able to apply MBT after an initial training course. Second, MBT was found to scale and to be significantly more efficient than manual testing. On a final note, we would like to express our gratitude to the many able and hard working people who were involved in the organization of ICST 2008 and this special issue. The steering committee members provided valuable advice and we were assisted by industry chairs Robert Eschbach, Per Runeson, and Clay Williams, student papers chairs Henry Muccini and Tao Xie, and by the general chair Lionel Briand. The success of ICST 2008 and also this special issue is testament to the efforts of these individuals as well as those who reviewed the papers for ICST 2008 and the special issue in a thorough and timely manner. We are also indebted to the ICST 2008 publicity chairs Harita Bhaskar, Yvan Labiche, Yves Le Traon, and T.H. Tse and the web chair Sudipto Ghosh. Finally, we thank all the authors who spent valuable time in preparing papers for ICST 2008 and this special issue.

This article is a brief overview of the current state of the software testing education in the Canadian and American universities. In doing so, the authors hope to pinpoint the strengths, and areas for improvement and to encourage a systematic software testing curriculum development. We also present some recommendations for new software testing educators.

Generative models such as statistical language modeling have been widely studied in the task of expert search to model the relationship between experts and their expertise indi-cated in supporting documents. On the other hand, dis-criminative models have received little attention in expert search research, although they have been shown to outper-form generative models in many other information retrieval and machine learning applications. In this paper, we propose a principled relevance-based discriminative learning frame-work for expert search and derive specific discriminative models from the framework. Compared with the state-of-the-art language models for expert search, the proposed re-search can naturally integrate various document evidence and document-candidate associations into a single model without extra modeling assumptions or effort. An extensive set of experiments have been conducted on two TREC En-terprise track corpora (i.e., W3C and CERC) to demonstrate the effectiveness and robustness of the proposed framework.

We propose an approach for conformance testing of implementations required to enforce access control policies specified using the Temporal Role-Based Access Control (TRBAC) model. The proposed approach uses Timed Input-Output Automata (TIOA) to model the behavior specified by a TRBAC policy. The TIOA model is transformed to a deterministic se-FSA model that captures any temporal constraint by using two special events Set and Exp. The modified W-method and integer-programming-based approach are used to construct a conformance test suite from the transformed model. The conformance test suite so generated provides complete fault coverage with respect to the proposed fault model for TRBAC specifications.

Conformance testing procedures for generating tests from the finite state model representation of Role-Based Access Control (RBAC) policies are proposed and evaluated. A test suite generated using one of these procedures has excellent fault detection ability but is astronomically large. Two approaches to reduce the size of the generated test suite were investigated. One is based on a set of six heuristics and the other directly generates a test suite from the finite state model using random selection of paths in the policy model. Empirical studies revealed that the second approach to test suite generation, combined with one or more heuristics, is most effective in the detection of both first-order mutation and malicious faults and generates a significantly smaller test suite than the one generated directly from the finite state models.

The effectiveness of statecharts as a tool to express the desired behavior of security protocols and a source of tests for their implementations was investigated. Specifically, TLS protocol was modeled as a statechart and tests generated from its flattened version. The GnuTLS implementation of the protocol was then tested against the generated tests. The MC/DC coverage of different components of the implementation varied from 51% to 81%. A "what if" analysis revealed that while some defects in the uncovered code will not lead to any security vulnerability due to in-built fault tolerance, others might lead to improper authentication, integrity failure, session hijacking, denial of service, and loss of confidentiality. The analysis suggests that statecharts alone might not be an adequate tool as a source of tests for implementations of security protocols and that tests so generated must be augmented through other formal means such as random testing, stress testing, and code coverage analysis.

A software development process modeling framework is constructed under the formalism of state modeling. The approach interconnects instances of a general development-activity model into a composite system. Simulation results are presented, and implications for control-theoretic decision support are briefly discussed.

Presents the introductory welcome message from the conference proceedings.

Provides a listing of current committee members and society officers.

While research into model based testing and verification (MSTV) is almost as old as the field of Computer Science, there has been a recent surge in the application and evaluation of MSTV technologies. MSTV has found takers in the embedded systems such as those found in medical devices, engine controllers, plant controllers, and mobile devices. The current panel is organized to focus on the successes of MSTV, its strengths, and its shortcomings. The panel will focus on the following in the context of MSTV: 1) what is "Model based testing and verification"?; 2) state-of-the-art and state-of-practice; 3) economic benefits; 4) strengths and weaknesses; and 5) research directions.

A quantitative, adaptive process control technique is described using an industrially validated model of the software system test phase as concrete target to be controlled. The technique combines the use of parameter correction and model predictive control to overcome the problems induced by modeling errors, parameter estimation errors, and limits on the resources available for productivity improvement.

A data collection and grooming tool named DIG is reported. The tool allows the management of a software process to be able to extract data from a variety of project repositories, groom the data, and generate reports. The reports are aimed at assisting the management in the control of the software development process. DIG allows management to build a complete representation of the productivity data in order to better tease out the parameters and understand the causes of variation in the data. In addition, the data obtained and groomed by DIG may be used to calibrate and apply process control models.

Summary form only given. The explosive growth of Internet and wireless infrastructure allows software development collaboratively from multiple locations. Technology and standards have evolved to the point where it is feasible to perform full life cycle software development through multiple site cooperation. In fact, most large software corporations have distributed or even global software development, an example of which is the current trend of outsourcing to India and China. Much research in the area of CSCW concentrated on the area of improving efficiency of groupware. Topics include group awareness, multi-user interfaces, concurrency control, communication and coordination within the group, shared information space and the support of a heterogenous, open environment which integrates existing single-user applications, and etc. The focus is on application of CSCW technology to software engineering regarding collaborative software development, including design, workflow, testbeds, and etc. In particular, we would like to discuss one fundamental question, i.e. whether CSCW software development is more efficient than single-location software development. The topics of our discussion include, but are not limited to the following: 1. What is the impact of CSCW on software reliability, as compared with single-site software development? 2. What is the impact of CSCW on software productivity as compared with single site software development? 3. What are the benefits of CSCW for software engineering? 4. What are the new research topics in CSCW for software engineering? 5. How can CSCW be used in software design, development and testing?.

We describe two case studies to investigate the application of a state variable model to control the system test phase of software products. The model consists of two components: a feedback control portion and a model parameter estimation portion. The focus in this study is on the assessment of the goodness of the estimates and predictions of the model parameters and their utility in the management of the system test phase. Two large network management applications developed and tested at Sun Microsystems served as the subjects in these studies. Unlike the release of products based on marketing or deadline pressure, estimates of the number of residual defects are used to control the quality of the product being released. The estimates of the number of defects in the application when the test phase began and at the current checkpoint are obtained. In addition a prediction is made regarding the reduction in the number of remaining defects over the remaining period. The estimates and predictions assist the management in planning the test phase and allow inferring the level of customer support needed subsequent to product release. The results of both case studies are satisfactory and, when viewed in light of other studies conducted at Sun Microsystems, show the applicability of the state variable model to the management of the software test process.

A collection of one or more devices, each described by its digital device manual and reachable over a network, is a ConnectedSpace. A set of safety policies may be enforced on a ConnectedSpace to ensure the safety of the environment in which the ConnectedSpace is deployed. The enforcement of these safety policies by one or more safely controllers governs the behavior of the devices within the ConnectedSpace. We propose a policy-based partitioning scheme for synthesizing k distributed safety controllers such that: (a) each device is guaranteed to be controlled by no more than two controllers, and (b) each policy is guaranteed to be enforced by exactly one controller. We present an experimental evaluation of our scheme. The experimental results show that the scheme is scalable with respect to the number of devices and the number of policies. We also show how safety controllers that are correct with respect to the policies, are synthesized using the theory of supervisory control.

Statistical Process Control (SPC) is a powerful tool to control the quality of processes. It assists management personnel in the identification of problems and actions to be taken to bring a process into a stable state. SPC has been applied in various fields, including the Software Development Process. However, some processes are better characterized by factors that exhibit an exponential behavior. The use of such factors for process control limits the application of traditional SPC techniques. The Software Test Process (STP) characterized by the decay in the number of remaining errors, failure intensity, and an increase in code coverage, is one such process.A variant of the traditional SPC technique is proposed. This variant uses logarithmic transformation to allow the statistical control of processes whose dominant behavior is best described by an exponential. An evaluation of the proposed transformation carried out using simulation and a case study from an industrial project, encourages the application of the proposed variant to the STP.

We report on the sensitivity analysis of a state variable model (Model S) proposed earlier. Model S captures the dominant behavior of the system test phase of the software test process. Sensitivity analysis is a mathematical methodology to compute changes in the system behavior due to changes in system parameters or variables. This is particularly important when parameters are calibrated using noisy or small data sets. Nevertheless, by mathematically quantifying the effects of parameter variations on the behavior of the model, and thereby the STP, one can easily and quickly evaluate the effect of such variations on the process performance without having to perform extensive simulations. In all cases studied, model S behaved according to empirical observations which serves to validate the model. It is also shown that sensitivity analysis can suggest structural improvements in a model when the model does not behave as expected.

We report a study to determine the impact of four types of disturbances on the failure intensity of a software product undergoing system test. Hardware failures, discovery of a critical fault, attrition in the test team, are examples of disturbances that will likely affect the convergence of the failure intensity to its desired value. Such disturbances are modeled as impulse, pulse, step, and white noise. Our study examined, in quantitative terms, the impact of such disturbances on the convergence behavior of the failure intensity. Results from this study reveal that the behavior of the state model, proposed elsewhere, is consistent with what one might predict. The model is useful in that it provides a quantitative measure of the delay one can expect when a disturbance occurs.

A novel approach to model the system test phase of the software life cycle is presented. This approach is based on concepts and techniques from control theory and is useful in computing the effort required to reduce the number of errors and the schedule slippage under a changing process environment. Results from these computations are used, and possibly revised, at specific checkpoints in a feedback-control structure to meet the schedule and quality objectives. Two case studies were conducted to study the behavior of the proposed model. One study reported here uses data from a commercial project. The outcome from these two studies suggests that the proposed model might well be the first significant milestone along the road to a formal and practical theory of software process control.

This paper presents the design and architecture of a prototype implementation of a XML based policy-driven management information server. It also describes the usage of such a server in building a flexible, extensible architecture for managing heterogeneous distributed systems.

A closed-loop feedback control model of the software test process (STP) is described. The model is grounded in the well established theory of automatic control. It offers a formal and novel procedure for using product reliability or failure intensity as a basis for closed loop control of the STP. The reliability or the failure intensity of the product is compared against the desired reliability at each checkpoint and the difference fed back to a controller. The controller uses this difference to compute changes necessary in the process parameters to meet the reliability, or failure intensity objective at the terminal checkpoint (the deadline). The STP continues beyond a checkpoint with a revised set of parameters. This procedure is repeated at each checkpoint until the termination of the STP. The procedure accounts for the possibility of changes (during testing), in reliability or failure intensity objective, the checkpoints, and the parameters that characterize the STP. The effectiveness of this procedure was studied using commercial data available in the public domain and also from the data generated through simulation. In all cases, the use of feedback control produces adequate results allowing the achievement of the objectives.

Many architecture-based software reliability models have been proposed in the past without any attempt to establish a relationship among them. The aim of this paper is to fill this gap. First, the unifying structural properties of the models are exhibited and the theoretical relationship is established. Then, the estimates provided by the models are compared using an empirical case study. The program chosen for the case study consists of almost 10,000 lines of C code divided into several components. The faulty version of the program was obtained by reinserting the faults discovered during integration testing and operational usage and the correct version was used as an oracle. A set of test cases was generated randomly accordingly to the known operational profile. The results show that 1) all models give reasonably accurate estimations compared to the actual reliability and 2) faults present in the components influence both components reliabilities and the way components interact.

A model is proposed to assist software test managers in controlling the behavior and progress of the Software Test Process (STP) by allowing them to compare predicted behavior against observed progress made at various checkpoints. The model, whose parameters are based on measured data and process characteristics, generates the predicted behavior. An algorithm for the parameter estimation is set forth. The error between the predicted and desired behavior is used to drive a parametric control algorithm that tells the manager how to correct for schedule deviations.

The paper reports the results of a sensitivity analysis of a state variable model of the Software Test Process (STP). Given a state model of the STP, a sensitivity matrix is calculated using tensor algebra. The sensitivity matrix allows computation of output variations to small perturbations in the model parameters. The results confirm that the model behaves in a manner very similar to what one might expect from a software test process. Results of this analysis also suggest changes and enhancements in the model to improve its accuracy in predicting the behavior of the Software Test Process.

Abstract Applications that utilize a broker‐based architecture are often composed of components that need to be tested individually and in combination. Furthermore, adequacy assessment of tests of components is useful in that it assists testers in identifying weaknesses in the tests generated so far and in offering hints on what the new tests must be. Traditional test adequacy criteria have limitations for commercial use, especially when tests for large components are to be assessed for their adequacy. This paper describes a test adequacy criterion based on interface mutation and a method, based on the criterion, to test components. This method requires the mutation of elements only from within a component's interface and not from within the code that implements the interface. The adequacy criterion based on interface mutation was evaluated empirically and compared with coverage criteria based on control flow for its relative effectiveness in revealing errors and in the cost incurred in developing test sets that satisfy the criterion. Copyright © 2001 John Wiley & Sons, Ltd.

The need for test adequacy criteria is widely recognized. Several criteria have been proposed for the assessment of adequacy of tests at the unit level. However, there remains a lack of criteria for the assessment of the adequacy of tests generated during integration testing. We present a mutation based interprocedural criterion, named Interface Mutation (IM), suitable for use during integration testing. A case study to evaluate the proposed criterion is reported. In the study, the UNIX sort utility was seeded with errors and Interface Mutation evaluated by measuring the cost of its application and its error revealing effectiveness. Alternative IM criteria using different sets of Interface Mutation operators were also evaluated. While comparing the error revealing effectiveness of these Interface Mutation-based test sets with same size randomly generated test sets, we observed that in most cases Interface Mutation based test sets are superior. The results suggest that Interface Mutation offers a viable test adequacy criteria for use at the integration level.

Describes a methodology for testing a software system for possible security flaws. Based on the observation that most security flaws are caused by a program's inappropriate interactions with the environment and are triggered by a user's malicious perturbation on the environment (which we call an "environment fault"), we view the security testing problem as the problem of testing for the fault-tolerance properties of a software system. We consider each environment perturbation as a fault, and the resulting security compromise as a failure in the toleration of such faults. Our approach is based on the well-known technique of fault injection. Environment faults are injected into the system under test, and the system's behavior is observed. A failure to tolerate faults is an indicator of a potential security flaw in the system. An environment-application interaction (EAI) fault model is proposed which guides us to decide what faults to inject. Based on EAI, we have developed a security testing methodology, and we have applied it to several applications. We have successfully identified a number of vulnerabilities, including vulnerabilities in the Windows NT operating system.

Branch coverage is an important criteria used during the structural testing of programs. We present a new program execution based approach to generate input data that exercises a selected branch in a program. The test data generation is initiated with an arbitrarily chosen input from the input domain of the program. A new input is derived from the initial input in an attempt to force execution through any of the paths through the selected branch. The method dynamically switches among the paths that reach the branch by refining the input. Using a numerical iterative technique that attempts to generate an input to exercise the branch, it dynamically selects a path that offers less resistance. We have implemented the technique and present experimental results of its performance for some programs. Our results show that our method is feasible and practical.

Many security breaches are caused by inappropriate inputs, crafted by people with malicious intents. To enhance the system security, we need either to ensure that inappropriate inputs are filtered out by the program, or to ensure that only trusted people can access those inputs. In the second approach, we certainly do not want to put such a constraint on every input; instead, we only want to restrict the access to the security-relevant inputs. This paper investigates how to identify which inputs are relevant to system security. We formulate the problem as a security relevancy problem and deploy static analysis technique to identify security-relevant inputs. Our approach is based on the dependency analysis technique; it identifies whether the behavior of any security-critical action depends on a certain input. If such a dependency relationship exists, we say that the input is security-relevant, otherwise we say the input is security-nonrelevant. This technique is applied to a security analysis project initiated by the Microsoft Windows NT Security Group. The project is intended to identify security-relevant registry keys in the Windows NT operating system. The results from this approach proved useful to enhancing Windows NT security. Our experiences and results from this project are presented in this paper.

A number of approaches have been proposed to automatically generate test data to traverse a given path in a program. We present a program execution based approach to generate test data for a given path. The technique derives a desired input for a test path by iteratively refining an arbitrarily chosen input. A set of linear constraints on the increments to the input are derived to refine the input. We solve this constraint set using a Unified Numerical Approach (UNA) developed in this paper. Our technique can generate both integer and floating point inputs as well as handle arrays and loops. We determine a basis set of paths for a program and use our technique to generate test data for this set. We implemented and experimentally evaluated our technique. We present results of generating input for scientific programs. The experimental results show that the technique is effective in that it generates input for most of the paths in the basis sets and also efficiently detects linear infeasible paths. Our experiments also show that our technique is efficient in the number of iterations required to generate test data. The time performance shows that it provides a practical method to automatically generate test data for scientific programs.

An important problem that arises in path oriented testing is the generation of test data that causes a program to follow a given path. In this paper, we present a novel program execution based approach using an iterative relaxation method to address the above problem. In this method, test data generation is initiated with an arbitrarily chosen input from a given domain. This input is then iteratively refined to obtain an input on which all the branch predicates on the given path evaluate to the desired outcome. In each iteration the program statements relevant to the evaluation of each branch predicate on the path are executed, and a set of linear constraints is derived. The constraints are then solved to obtain the increments for the input. These increments are added to the current input to obtain the input for the next iteration. The relaxation technique used in deriving the constraints provides feedback on the amount by which each input variable should be adjusted for the branches on the path to evaluate to the desired outcome.When the branch conditions on a path are linear functions of input variables, our technique either finds a solution for such paths in one iteration or it guarantees that the path is infeasible. In contrast, existing execution based approaches may require an unacceptably large number of iterations for relatively long paths because they consider only one input variable and one branch predicate at a time and use backtracking. When the branch conditions on a path are nonlinear functions of input variables, though it may take more then one iteration to derive a desired input, the set of constraints to be solved in each iteration is linear and is solved using Gaussian elimination. This makes our technique practical and suitable for automation.

Given a test set T to test a program P, there are at least two attributes of T that determine its fault detection effectiveness. One attribute is the size of T measured as the number of test cases in T. Another attribute is the code coverage measured when P is executed on all elements of T. The fault detection effectiveness of T is the ratio of the number of faults guaranteed to result in program failure when P is executed on T to the total number of faults present in P. An empirical study was conducted to determine the relative importance of the size and coverage attributes in affecting the fault detection effectiveness of a randomly selected test set for some program P. Results from this study indicate that as the size of a test set is reduced, while the code coverage is kept constant, there is little or no reduction in the fault detection effectiveness of the new test set so generated. For the study reported, of the two attributes mentioned above, the code coverage attribute of a test set is more important than its size attribute. © 1998 John Wiley & Sons, Ltd.

An important question in software testing is whether it is reasonable to apply coverage based criteria as a filter to reduce the size of a test set. An empirical study was conducted using a test set minimization technique to explore the effect of reducing the size of a test set, while keeping block coverage constant, on the fault detection strength of the resulting minimized test set. Two types of test sets were examined. For those with respect to a fixed size, no test case screening was conducted during the generation, whereas for those with respect to a fixed coverage, each subsequent test case had to improve the overall coverage in order to be included. The study reveals that no matter how a test set is generated (with or without any test case screening) block minimized test sets have a size/effectiveness advantage, in terms of a significant reduction in test set size but with almost the same fault detection effectiveness, over the original non-minimized test sets.

We report an experiment to evaluate a method, known as component based reliability estimation (CBRE), for the estimation of reliability of a software system using reliabilities of its components. CBRE involves computing path reliability estimates based on the sequence of components executed for each test input. Path reliability estimates are averaged over all test runs to obtain an estimate of the system reliability. In the experiment reported, three components of a Unix utility were seeded with errors and the reliability of each component was measured. The faulty components were then introduced systematically into the utility, in various combinations, to produce several faulty versions of the utility. For each faulty version, test cases were drawn from an operational profile to measure the component-based reliability. The true reliability of the faulty version was estimated using the frequency count approach. The goodness of CBRE was assessed in terms of the accuracy and efficiency of the estimates with respect to the true reliability. Results from this experiment suggest that CBRE yields reasonably accurate results at an efficient rate. However, the accuracy and efficiency of CBRE is sensitive to the dependency among successive calls to a component.

A criterion for assessing the adequacy of test sets during integration testing is proposed. The criterion is based on a testing technique named Interface Mutation. The technique itself is designed to be scalable with the size of the software under test; the size being measured in the number of subsystems integrated. Using Interface Mutation it is possible to assess the adequacy of tests incrementally while integrating various subsystems. Also reported are results from a pilot experiment conducted to study the cost and error defection effectiveness of Interface Mutation.

We describe the architecture and use of a tool named LISTEN. This is a general purpose tool to instrument computer programs so that during program execution aspects of program behavior are mapped to audible sound. Ongoing research aimed at investigating the usefulness of sound in various programming-related tasks and a lack of supporting tools led to the development of LISTEN. This tool is expected to find use in tasks such as program testing and debugging, software-development environments for the visually handicapped, and data analysis using aural cues. We also report our initial experience gathered during exploratory use of LISTEN and provide a summary of ongoing research using this tool.

Article Free AccessEffect of test set minimization on fault detection effectiveness Authors: W. Eric Wong Bell Communications Research, Morristown, NJ Bell Communications Research, Morristown, NJView Profile , Joseph R. Horgan Bell Communications Research, Morristown, NJ Bell Communications Research, Morristown, NJView Profile , Saul London Bell Communications Research, Morristown, NJ Bell Communications Research, Morristown, NJView Profile , Aditya P. Mathur Software Engineering Research Center, Department of Computer Sciences, Purdue University, W. Lafayette, IN Software Engineering Research Center, Department of Computer Sciences, Purdue University, W. Lafayette, INView Profile Authors Info & Claims ICSE '95: Proceedings of the 17th international conference on Software engineeringApril 1995Pages 41–50https://doi.org/10.1145/225014.225018Published:23 April 1995Publication History 135citation967DownloadsMetricsTotal Citations135Total Downloads967Last 12 Months94Last 6 weeks14 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

We report experiments conducted to investigate the correlation between code coverage and software reliability. Black-, decision-, and all-use-coverage measures were used. Reliability was estimated to be the probability of no failure over the given input domain defined by an operational profile. Four of the five programs were selected from a set of Unix utilities. These utilities range in size from 121 to 8857 lines of code, artificial faults were seeded manually using a fault seeding algorithm. Test data was generated randomly using a variety of operational profiles for each program. One program was selected from a suite of outer space applications. Faults seeded into this program were obtained from the faults discovered during the integration testing phase of the application. Test cases were generated randomly using the operational profile for the space application. Data obtained was graphed and analyzed to observe the relationship between code coverage and reliability. In all cases it was observed that an increase in reliability is accompanied by an increase in at least one code coverage measure. It was also observed that a decrease in reliability is accompanied by a decrease in at least one code coverage measure. Statistical correlations between coverage and reliability were found to vary between -0.1 and 0.91 for the shortest two of the five programs considered; for the remaining three programs the correlations varied from 0.89 to 0.99.

In a recent article by P.G. Frankl and E.J. Weyuker (see ibid., vol.19, no.3, p.962-75, 1993), results are reported that appear to establish a hierarchy of software test methods based on their respective abilities to detect faults. The methods used by Frankl and Weyuker to obtain this hierarchy constitute a new and important addition to their arsenal of tools. These tools were developed specifically to establish simple, useful comparisons of test data generation methods. This is the latest step in an ambitious test method classification program undertaken by the Frankl and Weyuker and their collaborators. The article discusses the method and goes on to present a reply to the critique.>

We report a case study to investigate the effect of errors in an operational profile on reliability estimates. A previously reported tool named TERSE was used in this study to generate random flow graphs representing programs, model errors in operational profile, and compute reliability estimates. Four models for reliability estimation were considered: the Musa-Okumoto model, the Goel-Okumoto model, coverage enhanced Musa-Okumoto model, and coverage enhanced Goel-Okumoto model. It was found that the error in reliability estimates from these models grows nonlinearly with errors in operational profile. Results from this case study lend credit to the argument that further research is necessary in development of more robust models for reliability estimation.>

Size and code coverage are two important attributes that characterize a set of tests. When a program P is executed on elements of a test set T, we can observe the fault-detecting capacity of T for P. We can also observe the degree to which T induces code coverage on P according to some coverage criterion. We would like to know whether it is the size of T or the coverage of T on P which determines the fault detection effectiveness (FDE) of T for P. We found that there is little or no reduction in the FDE of a test set when its size is reduced while the all-uses coverage is kept constant. These data suggest, indirectly, that coverage is more correlated than the size with the FDE. To further investigate this suggestion, we report an empirical study to compare the statistical correlation between (1) FDE and coverage, and (2) FDE and the size. Results from our experiments indicate that the correlation between FDE and block coverage is higher than that between FDE and size.>

Abstract Program unification is a technique for source‐to‐source transformation of code for enhanced execution performance on vector and SIMD architectures. This work focuses on simple examples of program unification to explain the methodology and demonstrate its promise as a practical technique for improved performance. Using simple examples to explain how unification is done, we outline two experiments in the simulation domain that benefit from unification, namely Monte Carlo and discrete‐event simulation. Empirical tests of unified code on a Cray Y‐MP multiprocessor show that unification improves execution performance by a factor of roughly 8 for given application. The technique is general in that it can be applied to computation‐intensive programs in various data‐parallel application domains.

Abstract Evaluation of the adequacy of a test set consisting of one or more test cases is a problem oftes encountered in software testing environments. Two test adequacy criiteria are considered, namely the data flow based all‐uses criterion and a mutation based criterion. An empirical study was conducted to compare the ‘difficulty’ of satisfying the two criteria and their costs. Similar studies conducted in the past are discussed in the light of this study. A discussion is also presented of how and why the results of this study, when viewed in conjunction with the results of earlier comparisons of testing methods, are useful to a software test team.

Currently more than forty models for estimating reliability exist. Thus, a practitioner is faced with the problem of selecting one out of many models to predict the behavior of given software. In such a situation, a method or tool to compare the reliability estimates from different models can certainly provide confidence in the selection of models and the estimates given by the selected model. To benchmark existing or new models, we present a new tool (TERSE) which can produce sets of failure data for a given program and compare the estimates produced by existing models. It can also generate random flow graphs for use by a given model. This feature offers a rich source of data for investigating effects of varying model parameters on reliability estimates and allows users to evaluate new models.

A method for integrating support for program mutation into a compiler is presented. The method is both efficient and sufficiently powerful to support program mutation software testing. Moreover, existing research suggests that this approach appears to be essential for the cost-effective application of program mutation to testing large commercial software systems. It is believed that a compiler-integrated approach will provide a significant increase in the efficiency of several existing testing tools and allow program mutation to be effectively used to test commercial software systems.>

The author has identified two problems that need to be overcome in order that some of the powerful testing techniques be used in practice: performance and effectiveness. The testing methods referred to are dataflow and mutation testing.>

A method for high-performance, software testing, called mutant unification, is described. The method is designed to support program mutation on parallel machines based on the single instruction multiple data stream (SIMD) paradigm. Several parameters that affect the performance of unification have been identified and their effect on the time to completion of a mutation test cycle and speedup has been studied. Program mutation analysis provides an effective means for determining the reliability of large software systems and a systematic method for measuring the adequacy of test data. However, it is likely that testing large software systems using mutation is computation bound and prohibitive on traditional sequential machines. Current, implementations of mutation tools are unacceptably slow and are only suitable for testing relatively small programs. The proposed unification method provides a practical alternative to the current approaches. The method also opens up a new application domain for SIMD machines.>

A new technique for source-to-source transformation of sequential programs is described. It is shown that the transformed programs so generated provide significant speedups over the original program on vector processors and vector multiprocessors. The parallelism that arises when multiple instances of a program are executed on simultaneously available data sets is exploited. This is in contrast to the existing approaches that aim at detecting parallelism within a program. The analytic model is used to prove the optimality of the complete first policy for block selection for a class of program graphs known as nonregressive graphs. Analytic and simulation models of the technique clearly indicate the speedups that could be achieved when several data sets are available simultaneously, as is the case in many fields of interest.>