Hybrid fault models are known to be an effective means for enhancing the robustness of consensus-based replicated systems. However, existing hybridization approaches suffer from limited flexibility with regard to the composition of crash-tolerant and Byzantine fault-tolerant system parts and/or are associated with a significant diversification overhead. In this paper we address these issues with ShellFT, a framework that leverages the concept of micro replication to allow system designers to freely choose the parts of the replication logic that need to be resilient against Byzantine faults. As a key benefit, such a selective hybridization makes it possible to develop hybrid solutions that are tailored to the specific characteristics and requirements of individual use cases. To illustrate this flexibility, we present three custom ShellFT protocols and analyze the complexity of their implementations. Our evaluation shows that compared with traditional hybridization approaches, ShellFT is able to decrease diversification costs by more than 70 %.

When put under stress, traditional state-machine replication protocols typically exhibit response times that by far exceed the average level of normal-case operation. The common way to mitigate such overload-induced tail latency is to overprovision computing and network resources. However, this method often leads to large amounts of resources left unused over extended periods of time, especially in application scenarios in which high loads are mostly limited to short phases. In this paper, we circumvent the need for overprovisioning with Idem, a replication protocol specifically designed to process client requests with low latency even during load spikes. Most notably, Idem replicas avoid overload by proactively rejecting requests in a collaborative manner. In contrast to centralized overload-prevention strategies, the collaboration among replicas allows Idem to always timely notify clients about rejections of their requests, not only under favorable conditions but also in the presence of replica crashes.

Consensus is a fundamental building block for constructing reliable and fault-tolerant distributed services. Many Byzantine fault-tolerant consensus protocols designed for partially synchronous systems adopt a pessimistic approach when dealing with adversaries, ensuring safety even under the worst-case scenarios that adversaries can create. Following this approach typically results in either an increase in the message complexity (e.g., PBFT) or an increase in the number of communication steps (e.g., HotStuff). In practice, however, adversaries are not as powerful as the ones assumed by these protocols. Furthermore, it might suffice to ensure safety and liveness properties with high probability. To accommodate more realistic and optimistic adversaries and improve the scalability of BFT consensus, we propose ProBFT (Probabilistic Byzantine Fault Tolerance). ProBFT is a leader-based probabilistic consensus protocol with a message complexity of [EQUATION] and an optimal number of communication steps that tolerates Byzantine faults in permissioned partially synchronous systems. It is built on top of well-known primitives, such as probabilistic Byzantine quorums and verifiable random functions. ProBFT guarantees safety and liveness with high probability even with faulty leaders, as long as a supermajority of replicas is correct and using only a fraction (e.g., 20%) of messages exchanged in PBFT. We provide a detailed description of ProBFT's protocol and its analysis.

Byzantine fault-tolerant (BFT) state-machine replication offers resilience against a wide spectrum of faults including hardware crashes, software failures, and attacks. Unfortunately, having been mostly designed for use on large servers, existing implementations of such replication protocols consume vast amounts of memory and therefore are not available to embedded systems that consist of highly resource-constrained devices. In this paper we address this problem with TinyBFT, the first BFT state-machine replication library specifically developed to run on nodes comprising 1 MB of RAM or less. To achieve this, TinyBFT relies on a memory-efficient implementation of the PBFT protocol that allocates all of its memory statically and thus, in contrast to common state-of-the-art PBFT-based libraries, has a guaranteed worst-case memory consumption that is known at compile time. Experiments show that our library provides sufficiently low latency even on tiny ESP32-C3 microcontrollers.

State-machine replication protocols represent the foundation of many fault-tolerant services. Unfortunately, their inherent complexity makes existing implementations notoriously difficult to debug and test. To address this problem, we propose a novel design approach, micro replication, whose main goal is to reduce bugs and enable replication protocols with improved debuggability properties. At its core, our concept consists of a set of principles that, if followed during protocol design, later significantly facilitate crucial tasks such as bug-source isolation, state-information retrieval, as well as root-cause identification. To achieve this, micro replication organizes a protocol as a composition of specialized modules (“micro replicas”) that each encapsulate a particular protocol phase or mechanism, and therefore are easier to test and monitor than traditional monolithic replicas. Besides discussing the underlying ideas of our approach, to show its feasibility we also present and evaluate Mirador, the first micro-replicated Byzantine fault-tolerant protocol.

Minimizing end-to-end latency in geo-replicated systems usually makes it necessary to compromise on resilience, resource efficiency, or throughput performance, because existing approaches either tolerate only crashes, require additional replicas, or rely on a global leader for consensus. In this paper, we eliminate the need for such tradeoffs by presenting ISOS, a leaderless replication protocol that tolerates up to f Byzantine faults with a minimum of 3 f + 1 replicas. To reduce latency in wide-area environments, ISOS relies on an efficient consensus algorithm that allows all participating replicas to propose new requests and thereby enables clients to avoid delays by submitting requests to their nearest replica. In addition, ISOS minimizes overhead by limiting message ordering to requests that conflict with each other (e.g., due to accessing the same state parts) and by already committing them after three communication steps if at least f + 1 replicas report each conflict. Our experimental evaluation with a geo-replicated key-value store shows that these properties allow ISOS to provide lower end-to-end latency than existing protocols, especially for use-case scenarios in which the clients of a system are distributed across multiple locations.

Existing approaches to tolerate Byzantine faults in geo-replicated environments require systems to execute complex agreement protocols over wide-area links and consequently are often associated with high response times. In this paper we address this problem with Spider, a resilient replication architecture for geo-distributed systems that leverages the availability characteristics of today's public-cloud infrastructures to minimize complexity and reduce latency. Spider models a system as a collection of loosely coupled replica groups whose members are hosted in different cloud-provided fault domains (i.e., availability zones) of the same geographic region. This structural organization makes it possible to achieve low response times by placing replica groups in close proximity to clients while still enabling the replicas of a group to interact over short-distance links. To handle the inter-group communication necessary for strong consistency Spider uses a reliable group-to-group message channel with first-in-first-out semantics and built-in flow control that significantly simplifies system design.

Replicated systems tolerating arbitrary (Byzantine) faults require periodic and deterministic application-state checkpoints to perform essential tasks such as initializing new replicas, enabling faulty replicas to recover, and garbage-collecting old agreement-protocol messages. Existing techniques to create checkpoints in these systems make it necessary to temporarily suspend request execution in order to capture a consistent checkpoint, causing significant service disruptions for applications with large states. Unfortunately, state-of-the-art approaches from the domain of crash-tolerant systems also are not directly applicable, because the checkpoints they produce are not comparable across replicas and therefore cannot be validated in an environment in which replicas may fail arbitrarily and do not trust each other. In this paper, we address these problems by proposing deterministic fuzzy checkpoints (DFC), a novel technique that enables all correct replicas in a system to create consistent and matching checkpoints in parallel to processing requests. As a consequence, DFC increases service availability while still allowing replicas to verify the correctness of a checkpoint before applying it to their local states. In addition to our general approach, we present different alternatives to implement DFC within a replication library and furthermore discuss support for the creation of differential checkpoints. Experiments with a key-value store show that DFC is able to snapshot states of 3 GB while sustaining high performance throughout the entire checkpointing process.

Various protocols and architectures have been proposed to make Byzantine fault tolerance (BFT) increasingly practical. However, the deployment of such systems requires dedicated client-side functionality. This is necessary as clients have to connect to multiple replicas and perform majority voting over the received replies to outvote faulty responses. Deploying custom client-side code is cumbersome, and often not an option, especially in open heterogeneous systems and for well-established protocols (e.g., HTTP and IMAP) where diverse client-side implementations co-exist. We propose Troxy, a system which relocates the BFT-specific client-side functionality to the server side, thereby making BFT transparent to legacy clients. To achieve this, Troxy relies on a trusted subsystem built upon hardware protection enabled by Intel SGX. Additionally, Troxy reduces the replication cost of BFT for read-heavy workloads by offering an actively maintained cache that supports trustworthy read operations while preserving the consistency guarantees offered by the underlying BFT protocol. A prototype of Troxy has been built and evaluated, and results indicate that using Troxy (1) leads to at most 43% performance loss with small ordered messages in a local network environment, while (2) improves throughput by 130% with read-heavy workloads in a simulated wide-area network.

Although internal devices (e.g., memory, timers) and external devices (e.g., transceivers, sensors) significantly contribute to the energy consumption of an embedded real-time system, their impact on the worst-case response energy consumption (WCRE) of tasks is usually not adequately taken into account. Most WCRE analysis techniques, for example, only focus on the processor and therefore do not consider the energy consumption of other hardware units. Apart from that, the typical approach for dealing with devices is to assume that all of them are always activated, which leads to high WCRE overestimations in the general case where a system switches off the devices that are currently not needed in order to minimize energy consumption. In this paper, we present SysWCEC, an approach that addresses these problems by enabling static WCRE analysis for entire real-time systems, including internal as well as external devices. For this purpose, SysWCEC introduces a novel abstraction, the power-state-transition graph, which contains information about the worst-case energy consumption of all possible execution paths. To construct the graph, SysWCEC decomposes the analyzed real-time system into blocks during which the set of active devices in the system does not change and is consequently able to precisely handle devices being dynamically activated or deactivated.

Energy-efficient cloud and data-center applications utilize just enough resources (e.g., threads, cores) to provide the performance required at the current point in time. Unfortunately, building such applications is inherently difficult in the face of varying workloads and further complicated by the fact that existing programming and execution platforms are not energy aware. Consequently, programmers are usually forced to choose between two unfavorable options: to lose performance and/or waste energy by relying on a static resource pool, or to significantly increase the complexity of their applications by implementing additional functionality to control resource usage at runtime. In this paper we present Empya, an energy-aware programming and execution platform that frees application programmers from the need to take care of energy efficiency. During execution, Empya constantly monitors both the performance as well as the energy consumption of an application and dynamically adjusts the system configuration to achieve the best energy–performance tradeoff for the current workload. In contrast to existing approaches, Empya combines techniques from different software and hardware levels to effectively and efficiently minimize the resource footprint of an application during periods of low utilization. This allows Empya to enable significant energy savings, as shown by our experimental evaluations of a key–value store and a variety of MapReduce applications.

Energy-neutral real-time systems harvest the entire energy they use from their environment. In such systems, energy must be treated as an equally important resource as time, which creates the need to solve a number of problems that so far have not been addressed by traditional real-time systems. In particular, this includes the scheduling of tasks with both time and energy constraints, the monitoring of energy budgets, as well as the survival of blackout periods during which not enough energy is available to keep the system fully operational. In this article, we address these issues presenting E n OS, an operating-system kernel for energy-neutral real-time systems. E n OS considers mixed time criticality levels for different energy criticality modes, which enables a decoupling of time and energy constraints when one is considered less critical than the other. When switching the energy criticality mode, the system also changes the set of executed tasks and is therefore able to dynamically adapt its energy consumption depending on external conditions. By keeping track of the energy budget available, E n OS ensures that in case of a blackout the system state is safely stored to persistent memory, allowing operations to resume at a later point when enough energy is harvested again.

Coordination services are essential building blocks of today's data centers as they provide processes of distributed applications with means to reliably exchange data. Consequently, coordination services must deliver high performance to ensure that they do not become a bottleneck for the applications depending on them. Unfortunately, the design of existing services such as ZooKeeper prevents them from scaling with the number of cores on a machine. In this paper, we address this problem with Agora, a high-performance coordination service that is able to both effectively and efficiently utilize multi-core machines. Agora relies on a primary-backup replication architecture that partitions the workload on each server to achieve parallelism while still providing similar consistency guarantees as ZooKeeper. Our evaluation shows that Agora scales with the number of cores and thus can fully utilize the network resources available.

With the advent of trusted execution environments provided by recent general purpose processors, a class of replication protocols has become more attractive than ever: Protocols based on a hybrid fault model are able to tolerate arbitrary faults yet reduce the costs significantly compared to their traditional Byzantine relatives by employing a small subsystem trusted to only fail by crashing. Unfortunately, existing proposals have their own price: We are not aware of any hybrid protocol that is backed by a comprehensive formal specification, complicating the reasoning about correctness and implications. Moreover, current protocols of that class have to be performed largely sequentially. Hence, they are not well-prepared for just the modern multi-core processors that bring their very own fault model to a broad audience. In this paper, we present Hybster, a new hybrid state-machine replication protocol that is highly parallelizable and specified formally. With over 1 million operations per second using only four cores, the evaluation of our Intel SGX-based prototype implementation shows that Hybster makes hybrid state-machine replication a viable option even for today's very demanding critical services.

Precisely evaluating the accuracy of worst-case execution time (WCET) analysis tools through benchmarking is inherently difficult and in general involves a significant amount of manual intervention. In this paper, we address this problem with ALADDIN, a tooling framework that enables fully-automated evaluations of WCET analyzers. To provide comprehensive results based on benchmarks with known WCETs, ALADDIN incorporates the GENE benchmark generator. Our demonstration shows how ALADDIN evaluates two state-of-the-art WCET analyzers: the commercial tool aiT and the open-source tool PLATIN.

Being able to comprehensively evaluate the individual strengths and weaknesses of worst-case execution time (WCET) analysis tools through benchmarking is essential for improving their accuracy. Unfortunately, a lack of knowledge about the detailed characteristics, actual complexities, and internal structures of existing benchmarks often prevents finegrained assessments, and sometimes even results in misleading conclusions. In this paper we present GENE, a tool that addresses these problems by automatically generating WCET benchmarks with known properties and predefined complexities. Due to the WCETs of benchmarks created by GENE being available, this approach for example makes it possible to precisely determine the accuracy of a WCET analyzer. In addition, the fact that GENE controls the program patterns of a benchmark enables fine-grained evaluations of the particular abilities and deficiencies of different WCET analyzers, as we demonstrate for aiT and PLATIN using multiple hardware platforms.

Energy-neutral real-time systems harvest the entire energy they use from their environment, making it essential to treat energy as an equally important resource as time. As a result, such systems need to solve a number of problems that so far have not been addressed by traditional real-time systems. In particular, this includes the scheduling of tasks with both time and energy constraints, the monitoring of energy budgets, as well as the survival of blackout periods during which not enough energy is available to keep the system fully operational. In this paper, we address these issues presenting ENOS, an operating-system kernel for energy-neutral real-time systems. ENOS considers mixed time criticality levels for different energy criticality modes, which enables a decoupling of time and energy constraints during phases when one is considered less critical than the other. When switching the energy criticality mode, the system also changes the set of tasks to be executed and is therefore able to dynamically adapt its energy consumption depending on external conditions. By keeping track of the energy budget available, ENOS ensures that in case of a blackout the system state is safely stored to persistent memory, allowing operations to resume at a later point when enough energy is harvested again.

Comprehensive evaluations of the effectiveness of worst-case execution time (WCET) analyzers require a selection of benchmarks that pose a challenge to these tools. In this paper, we identify pitfalls that are associated with selecting such benchmarks based on complexity metrics (e.g., the number of loops contained in a program), which in part are caused by the fact that complexity measures are not necessarily stable in the face of compiler optimizations. To address these problems, we are developing a tool that automatically assesses the resilience of a benchmark against compiler optimizations by tracking complexity measures across different optimization levels. In combination with information on the data dependency of control flows, which is also provided by our tool, this allows users to find and discard benchmarks that appear challenging for WCET analyzers at the source-code level, but in fact are trivial at the machine-code level where the actual analysis is performed.

One of the main reasons why Byzantine fault-tolerant (BFT) systems are currently not widely used lies in their high resource consumption:$3f+1$replicas are required to tolerate only$f$faults. Recent works have been able to reduce the minimum number of replicas to$2f+1$by relying on trusted subsystems that prevent a faulty replica from making conflicting statements to other replicas without being detected. Nevertheless, having been designed with the focus on fault handling, during normal-case operation these systems still use more resources than actually necessary to make progress in the absence of faults. This paper presentsResource-efficient Byzantine Fault Tolerance(ReBFT), an approach that minimizes the resource usage of a BFT system during normal-case operation by keeping$f$replicas in a passive mode. In contrast to active replicas, passive replicas neither participate in the agreement protocol nor execute client requests; instead, they are brought up to speed by verified state updates provided by active replicas. In case of suspected or detected faults, passive replicas are activated in a consistent manner. To underline the flexibility of our approach, we applyReBFTto two existing BFT systems: PBFT and MinBFT.

The fact that energy is a scarce resource in many embedded real-time systems creates the need for energy-aware task schedulers, which not only guarantee timing constraints but also consider energy consumption. Unfortunately, existing approaches to analyze the worst-case execution time (WCET) of a task usually cannot be directly applied to determine its worst-case energy consumption (WCEC) due to execution time and energy consumption not being closely correlated on many state-of-the-art processors. Instead, a WCEC analyzer must take into account the particular energy characteristics of a target platform. In this paper, we present 0g, a comprehensive approach to WCEC analysis that combines different techniques to speed up the analysis and to improve results. If detailed knowledge about the energy costs of instructions on the target platform is available, our tool is able to compute upper bounds for the WCEC by statically analyzing the program code. Otherwise, a novel approach allows 0g to determine the WCEC by measurement after having identified a set of suitable program inputs based on an auxiliary energy model, which specifies the energy consumption of instructions in relation to each other. Our experiments for three target platforms show that 0g provides precise WCEC estimates.

Most services inside a data center are distributed systems requiring coordination and synchronization in the form of primitives like distributed locks and message queues. We argue that extensibility is a crucial feature of the coordination infrastructures used in these systems. Without the ability to extend the functionality of coordination services, applications might end up using sub-optimal coordination algorithms, possibly leading to low performance. Adding extensibility, however, requires mechanisms that constrain extensions to be able to make reasonable security and performance guarantees. We propose a scheme that enables extensions to be introduced and removed dynamically in a secure way. To avoid performance overheads due to poorly designed extensions, it constrains the access of extensions to resources. Evaluation results for extensible versions of ZooKeeper and DepSpace show that it is possible to increase the throughput of a distributed queue by more than an order of magnitude (17x for ZooKeeper, 24x for DepSpace) while keeping the underlying coordination kernel small.

Consensus protocols employed in Byzantine fault-tolerant systems are notoriously compute intensive. Unfortunately, the traditional approach to execute instances of such protocols in a pipelined fashion is not well suited for modern multi-core processors and fundamentally restricts the overall performance of systems based on them. To solve this problem, we present the consensus-oriented parallelization (COP) scheme, which disentangles consecutive consensus instances and executes them in parallel by independent pipelines; or to put it in the terminology of our main target, today's processors: COP is the introduction of superscalarity to the field of consensus protocols. In doing so, COP achieves 2.4 million operations per second on commodity server hardware, a factor of 6 compared to a contemporary pipelined approach measured on the same code base and a factor of over 20 compared to the highest throughput numbers published for such systems so far. More important, however, is: COP provides up to 3 times as much throughput on a single core than its competitors and it can make use of additional cores where other approaches are confined by the slowest stage in their pipeline. This enables Byzantine fault tolerance for the emerging market of extremely demanding transactional systems and gives more room for conventional deployments to increase their quality of service.

One of the main reasons why Byzantine fault-tolerant (BFT) systems are not widely used lies in their high resource consumption: 3f+1 replicas are necessary to tolerate only f faults. Recent works have been able to reduce the minimum number of replicas to 2f+1 by relying on a trusted subsystem that prevents a replica from making conflicting statements to other replicas without being detected. Nevertheless, having been designed with the focus on fault handling, these systems still employ a majority of replicas during normal-case operation for seemingly redundant work. Furthermore, the trusted subsystems available trade off performance for security; that is, they either achieve high throughput or they come with a small trusted computing base.

Traditional agreement-based Byzantine fault-tolerant (BFT) systems process all requests on all replicas to ensure consistency. In addition to the overhead for BFT protocol and state-machine replication, this practice degrades performance and prevents throughput scalability. In this paper, we propose an extension to existing BFT architectures that increases performance for the default number of replicas by optimizing the resource utilization of their execution stages.