To increase their dependability, distributed control systems (DCSs) need to agree in real time about which hosts have crashed, i.e., they need a real-time membership service. In this paper, we prove that such a service cannot be implemented deterministically if, besides host crashes, communication can also fail. We define implementable probabilistic variants of membership properties, which constitute what we call a synchronous membership service (SYMS). We present an algorithm, ViewSnoop, that implements SYMS with high-probability. We implement, deploy and evaluate ViewSnoop analytically as well as experimentally, within an industrial DCS framework. We show that ViewSnoop significantly improves the dependability of DCSs compared to membership schemes based on classic heartbeats, at low additional cost. Moreover, ViewSnoop distinguishes, with high probability, host crashes from message losses, enabling DCSs to counteract losses better than existing approaches.

Embedded devices are often tightly constrained by CPU power and memory size. As a consequence, developers of embedded software avoid any kind of design abstractions, thinking that they imply large overhead. This results in complex designs with high coupling between the individual entities. The code of such designs is then difficult to maintain or reuse. This article presents FASAlight, a component-based software architecture for embedded devices with a focus on three main quality attributes that support maintenance and reuse: modularity, reusability, and portability. This architecture can be effectively implemented even for low-end embedded devices and only implies at worst a 14% on CPU and small overhead on memory on a device with a 120MHz ARM CPU and 128 kB of RAM.

Research on software engineering typically focuses on mainstream languages such as Java, .NET, and C. It is validated using projects easily executable and deployable on a desktop machine. Real, embedded, legacy code is however seldom made of such clean code. This article presents such a case. We performed the analysis and testing of legacy code, which is mix of C and DSP assembly. Such combinations of technologies cannot be analyzed by regular software engineering tools, creating a de facto software engineering desert island. Our solution relies on writing a parser for the DSP code, static analyzers, and using integration test cases. To run the tests, we also automate deployment on the target hardware and run the tests from an integration server.

In industrial automation, unexpected failures often generate high direct and indirect costs. To achieve high availability, the critical parts of the system are typically redundant. M-out-of-N (MooN) redundancy is a widespread redundancy pattern because it offers low failover times. This pattern has however two major disadvantages. First, the voter, an essential entity in MooN patterns, is itself a single point of failure. Second, it introduces delay to the control process and significant network load when deployed with multiple voters. This article proposes new MooN redundancy patterns for distributed control systems. These eliminate the voter as a single point of failure and reduce the network load and delay of the MooN redundancy. For each of these patterns we introduce an algorithm, which computes a near-minimum instance for the MooN problem. We validate our solution against minimum values and present a case study giving the delays introduced through our redundancy patterns and the induced network load.

Automation systems must primarily be deterministic and reliable, especially in safety-critical environments. With recent trends such as mass customization or Industry 4.0, there is an increasing need for automation systems to be dynamic. Changing parts of the software of today's automation systems, however, typically requires rebooting the controller, which makes software updates a complex and costly endeavor often despised by operators. This article presents an approach to updating the software of automation systems at runtime without disrupting the system's operation. This is achieved with a combination of a component-based architecture, cyclic application execution, and a state transfer mechanism between the original and the updated version of a component. We validate our solution with a case study in which we update the control algorithm of a magnetic levitation device running at cycles of 1 kHz without dropping the ball.

Test selection techniques are used to reduce the human effort involved in software testing. Most research focusses on selecting efficient sets of test cases according to various coverage criteria for directed testing. We introduce a new technique to select efficient sets of sub domains from which new test cases can be sampled at random to achieve a high mutation score. We first present a technique for evolving multiple sub domains, each of which target a different group of mutants. The evolved sub domains are shown to achieve an average 160% improvement in mutation score compared to random testing with six real world Java programs. We then present a technique for selecting sets of the evolved sub domains to reduce the human effort involved in evaluating sampled test cases without reducing their fault finding effectiveness. This technique significantly reduces the number of sub domains for four of the six programs with a negligible difference in mutation score.

Many automation systems are used in safety-critical environments, in which any nondeterministic timing behavior of the software may damage equipment or even threaten the health of people. Making software behave deterministically usually relies on hardware-bound optimizations, leaving little room for abstraction layers. This, however, has a negative impact on several important software qualities such as portability, maintainability, or scalability. As a consequence, it becomes difficult to use the same software across different platforms, scale it up or down to changing requirements, or reuse code across different families of products. In this article, we show how both flexibility and robustness can be achieved for real-time automation software. To this end, we present the mechanisms behind the component-based execution framework FASA. This framework offers a platform abstraction mechanism, which allows developers to port it to various platforms in a well-defined manner while maintaining its deterministic qualities. Furthermore, FASA supports runtime reconfigurations and software updates while executing real-time applications without any disruptions. For each of these concepts, we detail our reference implementation.

With the wide support for object serialization in object-oriented programming languages, persistent objects have become commonplace and most large object-oriented software systems rely on extensive amounts of persistent data. Such systems also evolve over time. Retrieving previously persisted objects from classes whose schema has changed is, however, difficult, and may lead to invalidating the consistency of the application. The ESCHER framework addresses these issues through an IDE-integrated approach that handles class schema evolution by managing versions of the code and generating transformation functions automatically. The infrastructure also enforces class invariants to prevent the introduction of potentially corrupt objects. This paper describes a model for class attribute changes, a measure for class evolution robustness, four empirical studies, and the design and implementation of the ESCHER system.

Cloud computing provides outsourcing of resources bringing economic benefits. The outsourcing however does not allow data owners to outsource the responsibility of confidentiality, integrity and access control, as it still is the responsibility of the data owner. As cloud computing is transparent to both the programmers and the users, it induces challenges that were not present in previous forms of distributed computing. Furthermore, cloud computing enables its users to abstract away from low-level configuration such as configuring IP addresses and routers. It creates an illusion that this entire configuration is automated. This illusion is also true for security services, for instance automating security policies and access control in cloud, so that individuals or end-users using the cloud only perform very high-level (business oriented) configuration. This paper investigates the security challenges posed by the transparency of distribution, abstraction of configuration and automation of services by performing a detailed threat analysis of cloud computing across its different deployment scenarios (private, bursting, federation or multi-clouds). This paper also presents a risk inventory which documents the security threats identified in terms of availability, integrity and confidentiality for cloud infrastructures in detail for future security risks. We also propose a methodology for performing security risk assessment for cloud computing architectures presenting some of the initial results.

Automated random testing is an effective and predictable method for finding faults. While it was recently studied both in practice and in theory, no general laws were found that express the number of faults in function of the time or the number of tests performed. This article evaluates the Michaelis-Menten equation (Max * t)/(K + t) as a law for representing the number of faults found by automated random testing. Max is the number of faults it can uncover in the code, K is a constant dependent on the tested code and the strategy used and t is the number of tests. The evaluation relies on the testing of more than 6000 Java classes from the Qualitas Corpus.

Mutation testing is effective at measuring the adequacy of a test suite, but it can be computationally expensive to apply all the test cases to each mutant. Previous research has investigated the effect of reducing the number of mutants by selecting certain operators, sampling mutants at random, or combining them to form new higher-order mutants. In this paper, we propose a new approach to the mutant reduction problem using static analysis. Symbolic representations are generated for the output along the paths through each mutant and these are compared with the original program. By calculating the range of their output expressions, it is possible to determine the effect of each mutation on the program output. Mutants with little effect on the output are harder to kill. We confirm this using random testing and an established test suite. Competent programmers are likely to only make small mistakes in their programming code. We argue therefore that test suites should be evaluated against those mutants that are harder to kill without being equivalent to the original program.

Regression testing techniques are commonly used to validate the correctness of upgrades. When a regression test fails, testers must understand the erroneous behaviors that caused the failure and identify the fault that originated these erroneous behaviors. In many cases, identifying the causes of a failure is difficult and time consuming. The analysis of regression problems provides interesting opportunities to validation and verification techniques. In fact, by comparing the execution of the base version and the upgraded version of the same program it is possible to automatically deduce information about incorrect behavior of the program. In this paper we present RADAR (Regression Analysis with Diff And Recording), a dynamic analysis technique, which analyzes regression problems and automatically identifies the chain of erroneous events that lead to a failure in C/C++ programs. RADAR exploits information about changes and the availability of multiple versions of the same program to automatically distinguish correct and suspicious events. Empirical experience with industrial and open source cases shows that RADAR can effectively support testers in the investigation of regression problems. Thus, RADAR can drive and simplify the debugging process.

Automated random testing is efficient at detecting faults but it is certainly not an optimal testing strategy for every given program. For example, an automated random testing tool ignores that some routines have stronger preconditions, they use certain literal values, or they are more error-prone. Taking into account such characteristics may increase testing effectiveness. In this article, we present Evotec, an enhancement of random testing which relies on genetic algorithms to evolve a best testing strategy for contract-equipped programs. The resulting strategy is optimized for detecting more faults, satisfying more routine preconditions and establishing more object states on a given set of classes to test. Our experiment tested 92 classes over 1710 hours. It shows that Evotec detected 29% more faults than random+ and 18% more faults than the precondition-satisfaction strategy.

Abstract Intuition suggests that random testing should exhibit a considerable difference in the number of faults detected by two different runs of equal duration. As a consequence, random testing would be rather unpredictable. This article first evaluates the variance over time of the number of faults detected by randomly testing object‐oriented software that is equipped with contracts. It presents the results of an empirical study based on 1215 h of randomly testing 27 Eiffel classes, each with 30 seeds of the random number generator. The analysis of over 6 million failures triggered during the experiments shows that therelative numberof faults detected by random testing over time is predictable, but that different runs of the random test case generator detectdifferent faults. The experiment also suggests that the random testing quickly finds faults: the first failure is likely to be triggered within 30 s. The second part of this article evaluates thenatureof the faults found by random testing. To this end, it first explains a fault classification scheme, which is also used to compare the faults found through random testing with those found through manual testing and with those found in field use of the software and recorded in user incident reports. The results of the comparisons show that each technique is good at uncovering different kinds of faults. None of the techniques subsumes any of the others; each brings distinct contributions. This supports a more general conclusion on comparisons between testing strategies: thenumberof detected faults is too coarse a criterion for such comparisons—thenatureof faults must also be considered. Copyright © 2009 John Wiley & Sons, Ltd.

Testing code is one of the central techniques for quality assessment of code. Generating test cases manually, however, is costly and inherently biased by the human point of view. While this might not be an issue for end-user code, it is problematic for developing libraries. The York Extendible Testing Infrastructure (YETI) is an automated random testing infrastructure supporting multiple programming languages (Java, JML, and .NET). It tests code at random and decouples the engine from the strategies and the language used. This article presents the .NET binding.

A random testing strategy can be effective at finding faults, but may leave some routines entirely untested if it never gets to call them on objects satisfying their preconditions. This limitation is particularly frustrating if the object pool does contain some precondition-satisfying objects but the strategy, which selects objects at random, does not use them. The extension of random testing described in this article addresses the problem. Experimentally, the resulting strategy succeeds in testing 56% of the routines that the pure random strategy missed; it tests hard routines 3.6 times more often; although it misses some of the faults detected by the original strategy, it finds 9.5% more faults overall; and it causes negligible overhead.

Developers write and execute ad-hoc tests as they implement software. While these tests reflect important insights of the developers (e.g., which parts of the software need testing and what inputs should be used), they are usually not persistent and are easily forgotten. They cannot always be re-executed automatically, for example to debug or to test for regressions. Several methods that make such test cases persistent and automatically executable have been proposed. They rely on capturing state and/or events at runtime and thus induce significant overhead or require specialized hardware. In previous work we proposed a method that, in the event of a failure, extracts test cases solely from the state at the time of the failure (and not from before the failure). We call this method "failure-state extraction". Capturing the state only at the moment of failure reduces the run-time overhead to zero, but comes at a cost: state extracted in this way cannot always be used to reproduce the failure. This paper provides an experimental evaluation of failure-state extraction. The results show that the method is highly effective: in the experiment, 90% of all failures were reproducible using failure-state extraction and thus could be extracted without run-time overhead.

With the wide support for serialization in object-oriented programming languages, persistent objects have become common place. Retrieving previously ¿persisted¿ objects from classes whose schema changed is however difficult, and may lead to invalidating the consistency of the application. The ESCHER framework addresses this issues through an IDE-based approach that handles schema evolution by managing versions of the code and generating transformation functions automatically. The infrastructure also enforces class invariants to prevent the introduction of any corrupt objects. This article describes the principles behind invariant-safe schema evolution,and the design and implementation of the ESCHER system.

Intuition is often not a good guide to know which testing strategies will work best. There is no substitute for experimental analysis based on objective criteria: how many faults a strategy finds, and how fast. "Random" testing is an example of an idea that intuitively seems simplistic or even dumb, but when assessed through such criteria can yield better results than seemingly smarter strategies. The efficiency of random testing is improved if the generated inputs are evenly spread across the input domain. This is the idea of Adaptive Random Testing (ART).

Intuition suggests that random testing of object-oriented programs should exhibit a significant difference in the number of faults detected by two different runs of equal duration. As a consequence, random testing would be rather unpredictable. We evaluate the variance of the number of faults detected by random testing over time. We present the results of an empirical study that is based on 1215 hours of randomly testing 27 Eiffel classes, each with 30 seeds of the random number generator. Analyzing over 6 million failures triggered during the experiments, the study provides evidence that the relative number of faults detected by random testing over time is predictable but that different runs of the random test case generator detect different faults. The study also shows that random testing quickly finds faults: the first failure is likely to be triggered within 30 seconds.

The usual way to compare testing strategies, whether theoretically or empirically, is to compare the number of faults they detect. To ascertain definitely that a testing strategy is better than another, this is a rather coarse criterion: shouldn't the nature of faults matter as well as their number? The empirical study reported here confirms this conjecture. An analysis of faults detected in Eiffel libraries through three different techniques-random tests, manual tests, and user incident reports-shows that each is good at uncovering significantly different kinds of faults. None of the techniques subsumes any of the others, but each brings distinct contributions.

Ever growing expectations from students, university management and other stakeholders make course preparation increasingly time-consuming. Setting up a course from scratch requires producing many supporting documents such as syllabi, schedules, and course web sites listing the concepts being taught. This can be a considerable effort, taking time away from tasks with a more immediate pedagogical value, such as answering student questions and refining the concepts themselves.

Formally defining the knowledge units taught in a course helps instructors ensure a sound coverage of topics and provides an objective basis for comparing the content of two courses. The main issue is to list and define the course concepts, down to basic knowledge units. Ontology learning techniques can help partially automate the process by extracting information from existing materials such as slides and textbooks. The TrucStudio course planning tool, discussed in this article, provides such support and relies on Text2Onto to extract concepts from course material. We conducted experiments on two different programming courses to assess the quality of the results.

Progress in testing requires that we evaluate the effectiveness of testing strategies on the basis of hard experimental evidence, not just intuition or a priori arguments. Random testing, the use of randomly generated test data, is an example of a strategy that the literature often deprecates because of such preconceptions. This view is worth revisiting since random testing otherwise offers several attractive properties: simplicity of implementation, speed of execution, absence of human bias.

Curriculum and course planning is a key step in developing quality educational programs, but current practices very often lack a systematic approach. This article addresses this issue by refining and expanding the concept of Testable, Reusable Unit of Cognition (Truc). The methodology allows modeling courses and verifying compliance of a given course to a given description. It also makes it possible to describe precisely what students have previously learned and, as a result, adapt the teaching to their specific needs. The article presents a case study of comparing a subset of two introductory programming textbooks and describes the application TrucStudio that supports the methodology.

Randomized unit test cases can be very effective in detecting defects. In practice, however, failing test cases often comprise long sequences of method calls that are tiresome to reproduce and debug. We present a combination of static slicing and delta debugging that automatically minimizes the sequence of failure-inducing method calls. In a case study on the EiffelBase library, the strategy minimizes failing unit test cases on average by 96%.

One of the main shortcomings of programming courses is the lack of practice with real-world systs. As a result, students feel unprepared for industry jobs. In parallel, open source software is accepting contributions even from inexperienced programmers and achieves software that competes both in quality and functionality with industrial systs. This article describes: first, a setting in which students were required to contribute to existing open source software; second, the evaluation of this experience using a motivation measuring technique; and third, an analysis of the efficiency and commitment of students over the time. The study shows that students are at first afraid of failing the assignment, but end up having the impression of a greater achievent. It ses also that students are inclined to keep working on the project to which they contributed after the end of the course.

Although unit tests are recognized as an important tool in software development, programmers prefer to write code, rather than unit tests. Despite the emergence of tools like JUnit which automate part of the process, unit testing remains a time-consuming, resource-intensive, and not particularly appealing activity.This paper introduces a new development method, called Contract Driven Development. This development method is based on a novel mechanism that extracts test cases from failure-producing runs that the programmers trigger. It exploits actions that developers perform anyway as part of their normal process of writing code. Thus, it takes the task of writing unit tests off the developers' shoulders, while still taking advantage of their knowledge of the intended semantics and structure of the code. The approach is based on the presence of contracts in code, which act as the oracle of the test cases. The test cases are extracted completely automatically, are run in the background, and can easily be maintained over versions. The tool implementing this methodology is called Cdd and is available both in binary and in source form.

Software updates typically require stopping and restarting an application, but many systems cannot afford to halt service, or would prefer not to. Dynamic software updating (DSU) addresses this difficulty by permitting programs to be updated while they run. DSU is appealing compared to other approaches for on-line upgrades because it is quite general and requires no redundant hardware. The challenge is in making DSU practical: it should be flexible, and yet safe, efficient, and easy to use.In this paper, we present Ginseng, a DSU implementation for C that aims to meet this challenge. We compile programs specially so that they can be dynamically patched, and generate most of a dynamic patch automatically. Ginseng performs a series of analyses that when combined with some simple runtime support ensure that an update will not violate type-safety while guaranteeing that data is kept up-to-date. We have used Ginseng to construct and dynamically apply patches to three substantial open-source server programs---Very Secure FTP daemon, OpenSSH sshd daemon, and GNU Zebra. In total, we dynamically patched each program with three years' worth of releases. Though the programs changed substantially, the majority of updates were easy to generate. Performance experiments show that all patches could be applied in less than 5 ms, and that the overhead on application throughput due to updating support ranged from 0 to at most 32%.