Byzantine Fault-Tolerant (BFT) protocols play an important role in blockchains. As the deployment of such systems extends to wide-area networks, the scalability of BFT protocols becomes a critical concern. Optimizations that assign specific roles to individual replicas can significantly improve the performance of BFT systems. However, such role assignment is highly sensitive to faults, potentially undermining the optimizations' effectiveness.

Many blockchain platforms use committee-based consensus for scalability, finality, and security. In this consensus scheme, a committee decides which blocks get appended to the chain, typically through several voting phases. Platforms typically leverage the committee members' recorded votes to reward, punish, or detect failures. A common approach is to let the block proposer decide which votes to include, opening the door to possible attacks. For example, a malicious proposer can omit votes from targeted committee members, resulting in lost profits and, ultimately, their departure from the system. This paper presents Iniva, an inclusive and incentive-compatible vote aggregation scheme that prevents such vote omission attacks. Iniva relies on a tree overlay with carefully selected fallback paths, making it robust against process failures without needing reconfiguration or additional redundancy. Our analysis shows that Iniva significantly reduces the chance to omit individual votes while ensuring that omitting many votes incurs a significant cost. In addition, our experimental results show that Iniva enjoys robustness, scalability, and reasonable throughput.

A certification system is responsible for issuing digital credentials, which attest claims about a subject, e.g., an academic diploma. Such credentials are valuable for individuals and society, and widespread adoption requires a trusted certification system. Trust can be gained by being transparent when issuing and verifying digital credentials. However, there is a fundamental tradeoff between privacy and transparency. For instance, admitting a student to an academic program must preserve the student’s privacy, i.e., the student’s grades must not be revealed to unauthorized parties. At the same time, other applicants may demand transparency to ensure fairness in the admission process. Thus, building a certification system with the right balance between privacy and transparency is challenging. This paper proposes a novel design for a certification system that provides sufficient transparency and preserves privacy through selective disclosure of claims such that authorized parties can verify them. Moreover, unauthorized parties can also verify the correctness of the certification process without compromising privacy. We achieve this using an incremental Merkle tree of cryptographic commitments to users' credentials. The commitments are added to the tree based on verifying zero-knowledge issuance proofs. Users store credentials off-chain and can prove the ownership and authenticity of credentials without revealing their commitments. Further, our approach enables users to prove statements about the credential’s claims in zero-knowledge. Our design offers a cost-efficient solution, reducing the amount of linkable on-chain data by up to 79% per credential compared to prior work, while maintaining transparency.

In cryptographic decentralized storage systems, files are split into chunks and distributed across a network of peers. These storage systems encode files using Merkle trees, a hierarchical data structure that provides integrity verification and lookup services. A Merkle tree maps the chunks of a file to a single root whose hash value is the file's content-address.

Sensitive data, such as personally identifiable information and human genetic data, should be protected against unauthorized accesses. When such sensitive data is stored and analyzed in a secure and trusted e-infrastructure, the workflow typically includes uploading raw data to the e-infrastructure, processing and downloading the results. However, accessing sensitive data imposes a significant added complexity due to the security restrictions of the e-infrastructure. This complexity results in additional manual work for researchers to perform data processing especially for use cases where raw data is generated incessantly, and instant result collection is a requirement. Stroll is a universal filesystem-based interface for seamless job submission to high performance computing clusters. Users interact with the cluster through simple read and write operations. Stroll supports simple and composite jobs. This paper describes the client-server architecture of Stroll that enables job submission to isolated sensitive data clusters. Stroll client-server is currently implemented in a pilot project for job submission in the Services for Sensitive Data e-infrastructure at the University of Oslo, and is part of an ongoing effort for building a Nordic federated sensitive data cloud.

This paper introduces Gorums, a novel RPC framework for building fault tolerant distributed systems. Gorums offers a flexible and simple quorum call abstraction, used to communicate with a set of processes, and to collect and process their responses. Gorums provides separate abstractions for (a) selecting processes for a quorum call and (b) processing replies. These abstractions simplify the main control flow of protocol implementations, especially for quorum-based systems, where only a subset of the replies to a quorum call need to be processed. To show that Gorums can be used in practical systems, we implemented EPaxos' latency-efficient quorum system, and ran experiments using a key-value storage. Our results show that Gorums' abstractions can provide additional performance benefits to EPaxos.

Emerging cloud-based network services must deliver both good performance and high availability. Achieving both of these goals requires content replication across multiple sites. Many cloud-based services either require or would benefit from the semantics and simplicity of strong consistency. However, replication techniques for strong consistency can severely limit the availability of replicated services when recovering large data objects over wide-area links. To address this problem, we present the design and implementation of ZORFU, a hierarchical system architecture for replication across data centers. The primary contribution of ZORFU is a local recovery technique that significantly increases availability of replicated strongly consistent services. Local recovery achieves this by reducing the recovery time by an order of magnitude, while imposing only a negligible latency overhead. Experimental results show that ZORFU can recover a 100 MB object in 4 ms.

We compare different algorithms for reconfigurable atomic storage in the data-centric model. We present the first experimental evaluation of two recently proposed algorithms for reconfiguration without consensus and compare them to established algorithms for reconfiguration both with and without consensus. Our evaluation reveals that the new algorithms offer a significant improvement in terms of latency and overhead for reconfiguration without consensus. Our evaluation also shows that reconfiguration without consensus, can obtain similar results to that of consensus-based reconfiguration, which relies on a stable leader. Moreover, the new algorithms also substantially reduces the overhead compared to consensus-based reconfiguration without a leader. While our analysis confirms our intuition that batching reconfiguration requests serves to reduce the overhead of reconfigurations, our evaluation also shows that it is equally important to separate reconfigurations from read and write operations. Specifically, we found that using read and write operations to assist in completing concurrent reconfigurations is in fact detrimental to the reconfiguration performance.

With the advent of recent technological advances, more demanding tele-immersive applications have started to emerge. In the World Opera application, artists from different opera houses across the globe can participate in a single united performance, and interact almost as if they were co-located. One of the main design challenges in this application domain is to assess to what extent the inevitable failures of some of the numerous and complex hardware, software, and network components affect the quality of experience for the user. This challenge cannot be addressed by traditional system-centric methods for dependability evaluation, which do not take personalized user perspective into account when considering meaningful and acceptable degradation of services. In this paper, we propose a novel method to assess the quality of experience in presence of failures, based on a new metric called perceived reliability. The method takes the human perspective into account and allows considering factors such as human perception of video and audio, characteristics of the audience, as well as performance elements and artistic content. This method can help system designers and engineers compare architectural variants and determine the dependability budget. We show the feasibility of our method by applying it to a World Opera performance. To this end, we construct a SAN-based model and run simulations in the Möbius framework. The obtained results provide useful guidelines for system engineers towards improving the quality of experience of World Opera performances despite the presence of failures.

We investigate methods for handling failures in a Paxos State Machine and introduce Replacement, a novel approach to handle failures. Replacement is fully decentralized and does not rely on consensus. This allows failed replicas to be replaced quickly, avoiding the bottleneck of a single leader. Instead of handling failures in the order proposed by a leader, concurrent replacements are combined to guarantee that all failed replicas are replaced. Replacement also allows the state machine to process client requests during failure handling, even while disagreeing on the current configuration. As our evaluation shows, this enables Replacement to quickly handle failures, with minimal disruption in the processing of client requests.

Building robust network services that can withstand a wide range of failure types is a fundamental problem in distributed systems. The most general approach, called Byzantine fault tolerance, can mask arbitrary failures. Yet it is often considered too costly to deploy in practice, and many solutions are not resilient to performance attacks. To address this concern we leverage two key technologies already widely deployed in cloud computing infrastructures: replicated state machines and intrusion detection systems. First, we have designed a general framework for constructing Byzantine failure detectors based on an intrusion detection system. Based on such a failure detector, we have designed and built a practical Byzantine fault-tolerant protocol, which has costs comparable to crash-resilient protocols like Paxos. More importantly, our protocol is particularly robust against several key attacks such as flooding attacks, timing attacks, and fairness attacks, that are typically not handled well by Byzantine fault masking procedures.

Advanced industrial robots usually consist of several independent control systems. Particularly, robots that perform process-intensive tasks like painting, gluing, and sealing have dedicated process control systems that are more or less loosely coupled with the motion control system. Testing the software for such systems is challenging because physical systems are necessary to test many of their characteristics. This paper proposes a method for automated testing of such robot systems. Our approach draws on previous work on continuous integration, combined with constraint programming techniques for test sequence generation. In ABB Robotics' process control system for robotized painting, many tests are only conducted every six months, during the release test. With our automated test approach, we expect to reduce the round-trip time, from code change to test completion, to less than one day.

We derive a consensus protocol for a hybrid failure model. In this model, clients are Byzantine faulty and servers are crash faulty. We argue that this model is well suited to environments where the servers run within one administrative domain, and the clients run outside of this domain. Our consensus protocol, which is derived from crash Paxos, provides low latency for client requests, tolerates any number of (Byzantine) faulty clients, up to 1/3 (crash) faulty servers, and does not rely on computing costly signatures in the common case. It can be used to build state machine replication that provides a highly available service.

The emerging publish/subscribe communication paradigm for building large-scale distributed event notification systems, has been shown to exhibit excellent performance and scalability characteristics. Moreover, some work also focus on providing reliability and availability guarantees in the face of node crash and link failures. Such publish/subscribe systems are commonly used in cloud computing infrastructures. However, addressing the dependability concern due to malicious attacks or unintentional software errors, which can potentially corrupt the system, has largely been left untouched by researchers. In this paper, we first identify some of the potential problem areas related to Byzantine behavior in the publish/subscribe paradigm. Secondly, we propose several directions of research for designing a Byzantine fault-tolerant publish/subscribe system suitable for use as a cloud computing infrastructure.

Abstract This paper presents the design and implementation of Jgroup/ARM, a distributed object group platform with autonomous replication management along with a novel measurement‐based assessment technique that is used to validate the fault‐handling capability of Jgroup/ARM. Jgroup extends Java RMI through the group communication paradigm and has been designed specifically for application support in partitionable systems. ARM aims at improving the dependability characteristics of systems through a fault‐treatment mechanism. Hence, ARM focuses on deployment and operational aspects, where the gain in terms of improved dependability is likely to be the greatest. The main objective of ARM is to localize failures and to reconfigure the system according to application‐specific dependability requirements. Combining Jgroup and ARM can significantly reduce the effort necessary for developing, deploying and managing dependable, partition‐aware applications. Jgroup/ARM is evaluated experimentally to validate its fault‐handling capability; the recovery performance of a system deployed in a wide area network is evaluated. In this experiment multiple nearly coincident reachability changes are injected to emulate network partitions separating the service replicas. The results show that Jgroup/ARM is able to recover applications to their initial state in several realistic failure scenarios, including multiple, concurrent network partitionings. Copyright © 2007 John Wiley & Sons, Ltd.

Performing experimental evaluation of fault tolerant distributed systems is a complex and tedious task, and automating as much as possible of the execution and evaluation of experiments is often necessary to test a broad spectrum of possible executions of the system to obtain good coverage. The confidence of the results obtained from an experimental evaluation depends on the degree of control over the environment in which experiments are being executed. Typically, an uncontrolled environment is exposed to numerous sources of external influence that can affect the obtained results. Automated and repeated executions can be used to reduce the impact of such influences. In this paper, a framework for experimental validation and performance evaluation of fault management in a fault tolerant distributed system is presented. The framework provides a facility to execute experiments in a configured target system. It is based on injecting faults or other events needed to test the fault handling capability of the system. Relevant events are logged and collected for postprocessing and analysis, e.g. to construct a single global timeline of events occurring at different nodes in the target system. This timeline of events can then be used to validate the behavior a system, and to evaluate its performance.

In a distributed fault-tolerant server system realized according to the open group model, inconsistency will (temporarily) arise between the dynamic membership of the replicated service and its client-side representation in the event of server failures and recoveries. The paper proposes techniques for maintaining this consistency and discuss their performance implications in failure/recovery scenarios where clients load balance requests on the servers. Comparative performance measurements is carried out for two of the proposed techniques. The results indicate that the performance impact of lacking consistency is easily kept small, and that the cost of the technique is small.

Change management is indispensable in most distributed software systems, which are continuously being modified throughout their life cycle. Managing the changes at runtime in highly available distributed systems is especially challenging as upgrade of a running system should not deteriorate its availability characteristics. We present a distributed algorithm that allows one to dynamically upgrade an actively replicated server so that the server is operational, even during the upgrade process. The algorithm makes use of the core functionality of an underlying group communication system that has been extended with a recovery mechanism. Its design enables dependable upgrades of replicated software in the presence of replica crashes. The presented mechanisms are part of the dynamic upgrade management framework aiming at supporting and managing dependable upgrades of distributed systems on the fly.

Recent peer-to-peer (P2P) systems are characterized by decentralized control, large scale and extreme dynamism of their operating environment. As such, they can be seen as instances of complex adaptive systems (CAS) typically found in biological and social sciences. We describe Anthill, a framework to support the design, implementation and evaluation of P2P applications based on ideas such as multi-agent and evolutionary programming borrowed from CAS. An Anthill system consists of a dynamic network of peer nodes; societies of adaptive agents travel through this network, interacting with nodes and cooperating with other agents in order to solve complex problems. Anthill can be used to construct different classes of P2P services that exhibit resilience, adaptation and self-organization properties. We also describe preliminary experiences with Anthill in implementing a file sharing application.