Verifying that a stochastic system is in a certain state when it has reached equilibrium has important applications. For instance, the probabilistic verification of the long-run behavior of a safety-critical system enables assessors to check whether it accepts a human abort-command at any time with a probability that is sufficiently high. The stochastic system is represented as probabilistic model, a long-run property is asserted and a probabilistic verifier checks the model against the property.

Coverage metrics try to quantify how well a software artifact is tested. High coverage numbers instill confidence in the software and might even be necessary to obtain certification. Unfortunately, achieving high coverage numbers does not imply high quality of the test suite. One shortcoming is that coverage metrics do not measure how well test suites cover systems in production. We look at coverage from an operational perspective. We evaluate test suite quality by comparing runs executed during testing with runs executed in production. Branch frequency spectra are employed to capture the behavior during runtime. Differences in the branch frequency spectra between field executions and testing runs indicate test suite deficiencies. This post-release test suite quality assurance mechanism can be used to (1) build confidence by pooling coverage information from many execution sites and (2) guide test suite augmentation in order to prepare the test suite for the next release cycle.

Functional testing is widespread and supported by a multitude of tools, including tools to mine functional specifications. In contrast, non-functional attributes like performance are often less well understood and tested. While many profiling tools are available to gather raw performance data, interpreting this raw data requires expert knowledge and a thorough understanding of the underlying software and hardware infrastructure. In this work we present an approach that mines performance specifications from running systems autonomously. The tool creates performance models during runtime. The mined models are analyzed further to create compact and comprehensive performance assertions. The resulting assertions can be used as an evidence-based performance specification for performance regression testing, performance monitoring, or as a foundation for more formal performance specifications.

While there is a good understanding of functional requirements and the need to test them during development, non-functional requirements are more elusive. Defining non-functional requirements can end up in a major undertaking consuming significant resources. Even after defining non-functional requirements, chances are they do not reflect the real-world usage of a deployed system. Differences can occur due to workload, hardware, or utilised third-party libraries. To tackle these challenges we propose a fully automatic compliance monitoring solution for non-functional properties. The proposed system mines stable behavioural patterns of the system and automatically extracts assertions that can be used to detect deviations of expected non-functional behaviour. We especially focus on non-functional properties that require runtime observation, e.g. execution time, performance, throughput.The full automation of the process enables a deployment in the field, giving rise to a distributed non-functional behaviour extraction system.

Spatial memory errors (like buffer overflows) are still a major threat for applications written in C. Most recent work focuses on memory safety - when a memory error is detected at runtime, the application is aborted. Our goal is not only to increase the memory safety of applications but also to increase the application's availability. Therefore, we need to tolerate spatial memory errors at runtime. We have implemented a compiler extension, Boundless, that automatically adds the tolerance feature to C applications at compile time. We show that this can increase the availability of applications. Our measurements also indicate that Boundless has a lower performance overhead than SoftBound, a state-of-the-art approach to detect spatial memory errors. Our performance gains result from a novel way to represent pointers. Nevertheless, Boundless is compatible with existing C code. Additionally, Boundless provides a trade-off to reduce the runtime overhead even further: We introduce vulnerability specific patching for spatial memory errors to tolerate only known vulnerabilities. Vulnerability specific patching has an even lower runtime overhead than full tolerance.

Software bugs and hardware errors are the largest contributors to downtime, and can be permanent (e.g. deterministic memory violations, broken memory modules) or transient (e.g. race conditions, bitflips). Although a large variety of dependability mechanisms exist, only few are used in practice. The existing techniques do not prevail for several reasons: (1) the introduced performance overhead is often not negligible, (2) the gained coverage is not sufficient, and (3) users cannot control and adapt the mechanism. Aaron tackles these challenges by detecting hardware and software errors using automatically diversified software components. It uses these software variants only if CPU spare cycles are present in the system. In this way, Aaron increases fault coverage without incurring a perceivable performance penalty. Our evaluation shows that Aaron provides the same throughput as an execution of the original application while checking a large percentage of requests - whenever load permits.