In a system containing several distributed servers, messages of random sizes generated at different locations must be disseminated and processed in the same order by all hosts. A ring protocol is defined, where a number of folders carrying messages circulate in one direction without overtaking each other. A model involving parallel queues is analysed in the steady state and is solved approximately, allowing the computation of performance measures. A number of example systems are evaluated numerically and by simulations, leading to a heuristic for choosing the optimal number of folders.

Distributed OLTP systems execute the high-overhead, two-phase commit (2PC) protocol at the end of every distributed transaction. Epoch-based commit proposes that 2PC be executed only once for all transactions processed within a time interval called an epoch. Increasing epoch duration allows more transactions to be processed before the common 2PC. It thus reduces 2PC overhead per transaction, increases throughput but also increases average transaction latency. Therefore, required is the ability to choose the right epoch size that offers the desired trade-off between throughput and latency. To this end, we develop two analytical models to estimate throughput and average latency in terms of epoch size taking into account load and failure conditions. Simulations affirm their accuracy and effectiveness. We then present epoch-based multi-commit which, unlike epoch-based commit, seeks to avoid all transactions being aborted when failures occur, and also performs identically when failures do not occur. Our performance study identifies workload factors that make it more effective in preventing transaction aborts and concludes that the analytical models can be equally useful in predicting its performance as well.

Summary The two‐phase commit (2PC) protocol has long been known to have a provably inevitable vulnerability to blocking or non‐progress amidst server crashes, even when the distributed database system guarantees the most demanding timing‐related or “synchrony” requirements. Our aim here is to eliminate this vulnerability by using a blockchain for coordinating 2PC execution. We present the impossibilities, the possibilities, the cost, and the trade‐offs in this blockchain‐based approach to blocking‐free management of distributed transactions. We prove that a non‐blocking and blockchain‐coordinated 2PC protocol can exist only if both the blockchain and distributed database systems meet synchrony requirements; otherwise, although blocking remains eliminated, transactions can unnecessarily abort. We present a blockchain‐coordinated 2PC protocol and provide rigorous arguments for its correctness under the synchrony requirements. We then implement this protocol on the Ethereum Testnet and demonstrate, through our experiments, that the monetary cost of executing smart contracts is quite small, that the protocol performance slows down when using a public blockchain like Ethereum, and that even major violations of synchrony requirements lead only to relatively small increases in unnecessary aborts. We thus identify a trade‐off between improving protocol performance and admitting a risk that transactions could occasionally abort unnecessarily.

A stream-processing server model consisting of an external queue and an internal queue, with instantaneous or non-instantaneous transfers between the two, is analysed in the steady state. Jobs are collected into batches of fixed size prior to being transferred, but there is also a timer mechanism that may preempt such a collection. Exact and approximate solutions are obtained. These are used in order to evaluate the trade-offs between holding costs and transfer costs. The results of several numerical experiments are presented.

We examine a system where servers can host several virtual machines in parallel and where some of the users are malicious. Arrivals and departures of both normal and malicious users are governed by random processes. The aim is to estimate the probability that a possible target will find itself sharing a server with an attacker. Two allocation policies for assigning virtual machines to servers are studied. In both cases, as well as attacks forming part of the arrival process, multiple simultaneous attacks are considered. Closed-form expressions for the desired estimates are obtained. Comparisons with simulations for purposes of validation are presented and the effect of increasing the number of available servers is illustrated.

Services of different types are provided to paying customers on servers hired from a cloud. Different virtual machines can share a server, subject to one or more resource constraints. Incoming jobs whose resource requirements cannot be satisfied are lost. The objective is to maximize the long-term average profit per unit time. A single-server model is analyzed exactly and the results provide approximations for the system with n servers. The latter is also solved exactly when the servers are dedicated and when the VMs can migrate instantaneously. Numerical examples and comparisons with simulations are presented.

Atomic Multicasts are central to the management of replicated data in distributed systems. Previous work has proven the effectiveness of utilising atomic multicasts, opposed to the classic two-phase commit, to coordinate transactions in in-memory databases. However, the current family of protocols utilised by such systems do not scale as the number of destinations increases. We propose that atomic multicasts should not occur between database nodes, instead transaction ordering should be exposed as a service that is provided by a dedicated set of nodes. Our performance study shows a clear improvement in transaction throughput and latency as the number of participants in a transaction increases.

Amazon's recent service disruption and investigations into the underlying causes of similar major outages indicate that cloud outages in future cannot be ruled out with certainty. This paper investigates the idea of tolerating outages by inter-cloud replication, i.e., through service replication on multiple, fail-independent clouds. A challenge in realizing this idea is to minimize performance degradation that inevitably arises when replicas on multiple clouds have to be kept in a mutually consistent state over the Internet. It is addressed by developing a new order protocol that makes the most use of the high bandwidth communication within a cloud and uses the Internet communication to minimum necessary. The protocol also deals with cloud outages and widely differing rates with which service requests can arrive at replicas in different clouds. Experiments performed confirm that the protocol reduces the ordering latencies considerably and also improves throughput.

Primary-backup replication enables arbitrary services, which need not be built as deterministic state machines, to be reliable against server crashes. Further, when the primary does not crash, the performance can be close to that of an un-replicated, 1-server system and is arguably far better than what state machine replication can offer. These advantages have made primary-backup replication a widely used technique in commercial provisioning of services, even though the technique assumes that residual software bugs in a server system can lead only to crashes and cannot result in state corruption. This assumption cannot hold against an attacker intent on exploiting vulnerabilities and corrupting the service state when attacks lead to intrusions. This paper presents a system, called FORTRESS, which can encapsulate a primary-backup system and safeguard it from being intruded. At its core, FORTRESS applies proactive obfuscation techniques in a manner appropriate to primary-backup replication and deploys proxy servers for additional defence. Gain in intrusion resilience is shown to be substantial when assessed through analytical evaluations and simulations for a range of attacker scenarios. Further, by implementing two web-based applications, the average performance drop is demonstrated to be in the order of tens of milliseconds even when obfuscation intervals are as small as tens of seconds.

Consensus is central to several applications including collaborative ones which a wireless ad hoc network can facilitate for mobile users in terrains with no infrastructure support for communication. We solve the consensus problem in a sparse network in which a node can at times have no other node in its wireless range and useful end-to-end connectivity between nodes can just be a temporary feature that emerges at arbitrary intervals of time for any given node pair. Efficient one-to-many dissemination, essential for consensus, now becomes a challenge; enough number of destinations cannot deliver a multicast unless nodes retain the multicast message for exercising opportunistic forwarding. Seeking to keep storage and bandwidth costs low, we propose two protocols. An eventually relinquishing (◇ RC) protocol that does not store messages for long is used for attempting at consensus, and an eventually quiescent (◇ QC) one that stops forwarding messages after a while is used for concluding consensus. Use of the ◇ RC protocol poses additional challenges for consensus, when the fraction, f/n, of nodes that can crash is 1/4 ≤ f/n <; 1/2. Consensus latency and packet overhead are measured through simulations and both decrease considerably even for a modest increase in network density.

Scaling in hardware integration process results in IC-process geometry reductions, lower operating voltages and increased clock speeds. This paper first surveys the reliability obstacles these developments give rise to and then points out that computing systems can no longer be safely assumed to fail only by crashing. Yet this assumption is at the core of primary-backup replication which the literature presents as the appropriate, and hence the most widely used, strategy for time-critical fault-tolerant applications. The paper then observes that building computing nodes with announced crash failure mode is a promising way forward to deal with the emerging reliability challenges. Work carried out to assure such a failure mode has also been briefly surveyed.

This report gives an overview of the workshop on "Dependable Application Support for Self-Organising Networks" held in conjunction with DSN 2007. The principal objective of the workshop is to facilitate a forum for researchers to explore, examine and address dependability related challenges in hosting distributed applications in self-organised networks such as MANETs, sensor and P2P networks.

As e-businesses are becoming ubiquitous, enhancing the performance and scalability of e-business systems has become an increasingly important topic of investigation. As Vitruvius (70-25 BC) put it succinctly 'function follows form', the ability of a system to perform well and scale easily is influenced by how the system itself is formed or implemented. A common approach to implement e-business systems is to make use of off-the-shelf enterprise middleware systems, such as a J2EE-compliant application server. Such middleware systems handle several, often complex, issues and thus simplify application development. They however allow developers the freedom not to use particular forms of support they offer and build their own mechanisms instead. This flexibility gives rise to many implementation methods. The work reported here evaluates these methods for Response Time and Throughput under various environments related to both client side (external to the system) and application execution (internal). To this end, one of the most widespread technologies used by the industry, the Enterprise Java Beans (EJB), is chosen; we have considered six commonly used implementation methods for an e-auction application and five different client-side and execution environments. The resulting study, which involves 78 experimental runs, identifies the strengths and the weaknesses of each implementation method under 13 different scenarios. It thus offers reliable guidelines for developers and valuable insights to researchers.

Any asynchronous total-order protocol must somehow circumvent the well-known FLP impossibility result. This paper exposes the performance gains obtained when this impossibility is dealt with through the use of abstract processes built to have some special failure semantics. Specifically, we build processes with signal-on-fail semantics by (i) having a subset of Byzantine-prone processes paired to check each other’s computational outputs, and (ii) assuming that paired processes do not fail simultaneously. By dynamically invoking the construction of signal-on-fail processes, coordinatorbased total-order protocols which allow less than onethird of processes to fail in a Byzantine manner are developed. Using a LAN-based implementation, failure-free order latencies and fail-over latencies are measured; the former are shown to be smaller compared to the protocol of Castro and Liskov which is generally regarded to perform exceedingly well in the best-case scenarios.

E-business organizations commonly trade services together with quality of service (QoS) guarantees that are often dynamically agreed upon prior to service provisioning. Violating agreed QoS levels incurs penalties and hence service providers agree to QoS requests only after assessing the resource availability. Thus the system should, in addition to providing the services: (i) monitor resource availability, (ii) assess the affordability of a requested QoS level, and (iii) adapt autonomically to QoS perturbations which might undermine any assumptions made during assessment. This paper will focus on building such a system for reliably multicasting messages of arbitrary size over a loss-prone network of arbitrary topology such as the Internet. The QoS metrics of interest will be reliability, latency and relative latency. We meet the objectives (i)-(iii) by describing a network monitoring scheme, developing two multicast protocols, and by analytically estimating the achievable latencies and reliability in terms of controllable protocol parameters. Protocol development involves extending in two distinct ways an existing QoS-adaptive protocol designed for a single packet. Analytical estimation makes use of experimentally justified approximations and their impact is evaluated through simulations. As the protocol extension approaches are complementary in nature, so are the application contexts they are found best suited to; e.g., one is suited to small messages while the other to large messages

This paper is on the construction of a fault-tolerant and responsive server subsystem in an application context where the subsystem is accessed through an asynchronous network by a large number of clients. The server is made fault-tolerant by the triple modular redundancy (TMR) technique: at least two server processes behave correctly, while the third one can behave arbitrarily. An essential requirement for process replication is that the client inputs be delivered to server replicas for processing in an identical order. Moreover, in order to cope with process' memory requirement, a time bound constraint is imposed: no client input can stay in the local memory of server process more than /spl Sigma/ units of time. Based on known technologies, two assumptions are made: (1) the network delivers a given client input to any two server processes within a known bounded time (D), and (2), there is an Ordered Timed Atomic Broadcast protocol built on top of the TMR system with timeliness A. The paper presents two results. The first is a protocol that delivers an ordered stream of client inputs, such that every client input is delivered exactly once to each correct server, thus eliminating redundant verification. It works under the assumption /spl Sigma/>D+/spl Delta/. The second is an impossibility result, namely there can be no ordering protocol when /spl Sigma/<D+/spl Delta/_, where /spl Delta/_ is the minimum timeliness of any reliable broadcast protocol that can be implemented on top of the TMR server (/spl Delta/_ < /spl Delta/).

Encounter Gossip is a family of message propagation protocols for mobile ad-hoc networks. The coverage of propagation (the fraction of nodes that receive the message) can be made arbitrarily close to 1 at the cost of increased bandwidth overhead. This paper proposes several schemes to minimize this overhead without compromising the achieved coverage. The schemes can be timer based or history based. Their effectiveness is assessed through simulations performed in the context of two mobility models.

The mobile ad-hoc networking (MANET) technology offers an ideal medium for hosting self-organized collaborative applications in terrains with no infrastructure support for untethered communication. Collaboration involves users with potentially different initial opinions deciding identically, i.e., reaching consensus. Efficient consensus solutions require efficient broadcast support. This paper presents four crash-tolerant broadcast protocols which are designed (i) to provide the maximum broadcast coverage that can ever be guaranteed, and (ii) to suit a wide range of MANET types: from a connected MANET (no partitions) to intermittently disconnected one (partitions occurring rarely and healing swiftly) to an intermittently connected one (partitions taking longer to heal and re-appearing swiftly). The resulting design challenges are addressed systematically, presenting two foundational results that would guide the protocol design. The protocols' performance is then studied through simulations for a range of node speeds and network densities. The best-performing one is used to host a consensus protocol as its 'application'. The overhead and the latency for reaching consensus are measured; surprisingly, they are hardly affected as the number of nodes with distinct initial opinions increases beyond one.

Fair exchange protocols play an important role in application areas such as e-commerce where protocol participants require mutual guarantees that a transaction involving exchange of items has taken place in a specific manner. A protocol is fair if no protocol participant can gain any advantage over an honest participant by misbehaving. In addition, such a protocol is fault-tolerant if the protocol can ensure that an honest participant does not suffer any loss of fairness despite any failures of the participant's node. This paper presents a family of fair exchange protocols for two participants which make use of the presence of a trusted third party, under a variety of assumptions concerning participant misbehavior, message, delays, and node reliability. The development is systematic, beginning with the strongest set of the assumptions and gradually weakening the assumptions to the weakest set. The resulting protocol family exposes the impact of a given set of assumptions on solving the problem of fair exchange. Specifically, it highlights the relationships that exist between fairness and assumptions on the nature of participant misbehavior, communication delays, and node crashes. The paper also shows that the restrictions assumed on a dishonest participant's misbehavior can be realized through the use of smartcards and smartcard-based protocols.

The problem of efficient message delivery to multiple nodes has been widely studied for ad-hoc networks, be it sensor or mobile ad-hoc networks (MANETs). We are concerned with deterministic delivery guarantees, i.e. the delivery ratio for any given message is guaranteed not to fall below a threshold specified for that message. This threshold can be considered to take the special case value of 100% when message delivery is guaranteed to all nodes. Guaranteeing delivery to all in a group, however desirable, often involves making a simplifying assumption that no destination node leaves the group nor crashes during an execution of the protocol. However, the nodes that make up an ad-hoc network are prone to failures and are likely to leave the group, something which undermines the validity of the assumption. We outline a protocol, called Scribble, which permits the minimum threshold to be less than 100%, thus allowing a few nodes to depart or crash during the execution. More precisely, it provides what we call k-deterministic delivery guarantees, where delivery is guaranteed to at least k destination nodes. The value of k is chosen by the application is required to be a realistic value. Such a service is called manycast.

Measuring progress in terms of industrial take-up, the paper takes a view that the progress in FT RT computing has been significant, and that the new dominant application domains do not allow the 'ingredients' attributable to past success to be re-used at the same level they were once used. Consequently, FT RT computing is acquiring new faces in the form of adaptive and autonomic computing.

This paper presents and studies a reliable multicast protocol whose objective is to deliver a message to all intended destinations, despite possible crashes of the sender and other processes, and communication failures. The protocol enables QoS metrics such as absolute and relative latencies and the probability of reliable delivery, to be negotiated prior to service provisioning. Moreover, it adapts certain parameters dynamically in order to minimize the message traffic required to achieve the negotiated QoS metrics. The performance of the protocol is analyzed mathematically under simplifying assumptions. The accuracy of the approximations is evaluated by simulations.

Replicated processing with majority voting is a well-known method for achieving reliability and availability. Triple modular redundant (TMR) processing is the most commonly used version of that method. Replicated processing requires that the replicas reach agreement on the order in which input requests are to be processed. Almost all synchronous and deterministic ordering protocols published in the literature are time-based in the sense that they require replicas' clocks to be kept synchronized within some known bound. We present a protocol for TMR systems that is based on timeouts and does not require clocks to be kept in bounded synchronism. Our design efforts focus on keeping the ordering delays small, without an unnecessary increase in message overhead. Consequently, we are able to show that no symmetric protocol that works only with unsynchronized clocks can provide a smaller worst-case delay. We also demonstrate through analysis and experiments that our protocol is faster than a time-based one of identical message complexity in certain situations which can prevail in many application settings.

Many fault-tolerant group communication middleware systems have been implemented assuming crash failure semantics. While this assumption is not unreasonable, it becomes hard to justify when applications are required to meet high reliability requirements and are built using commercial off the shelf (COTS) components. This paper presents a structured approach to extend a crash-tolerant middleware system into an authenticated Byzantine tolerant one with small modifications to the original system. The proposed approach is based on state machine replication (SMR) and is motivated by the composability features of standard distributed object technologies such as CORBA. SMR is used to assure signal-onfailure (fail-signal) semantics at a level where existing crash-tolerant services can be seamlessly deployed. The resulting system can provide total ordering that has no liveness requirement for termination. We demonstrate the effectiveness of our approach by porting a crash-tolerant CORBA group communication service – the NewTOP system. We also measure the performance of the resulting system.

No abstract available.

The paper develops a distributed system architecture and proposes an implementation framework for conducting dependable Internet based online auctions, meeting the requirements of scalability and service integrity. Current auction services essentially rely on a central auction server. Given the increasing popularity and usage of electronic auctions, such a centralised approach is fundamentally restrictive with respect to scalability. Further, different national markets have different monetary regulations and may employ different procedures for payment settlements. Catering for local market autonomy means that decentralisation is an essential and practical requirement. With these design goals in mind, the paper presents a system architecture and a framework for a fault-tolerant implementation of the architecture.

The consensus problem is a fundamental problem one has to solve to implement reliable services or applications on top of asynchronous distributed systems prone to failures. Unfortunately, this problem cannot be solved in those systems as soon as one process crashes (Fischer-Lynch-Paterson's impossibility result). Two approaches have been investigated to circumvent this impossibility result. Both consist in enriching the underlying system with appropriate "oracles". The unreliable failure detector concept proposed by Chandra and Toueg (1996) constitutes one family of such oracles. Since it has been proposed the failure detector based approach has given rise to several failure detector-based consensus protocols. The other family of oracles consists in allowing each process to use a random number generator. In that case, protocol termination is only probabilistic. A few randomized consensus protocols for message-passing asynchronous distributed systems have been proposed. Moreover, they consider that processes can only propose values from a binary set. This paper proposes a new randomized consensus protocol that allows processes to propose arbitrary values. Contrary to other randomized consensus protocols, the proposed protocol does not require the a priori knowledge of the set of values that can be proposed by processes. It relies on a relatively simple combination of randomization and reliable broadcast.

The paper develops a distributed systems architecture for dependable Internet based online auctions, meeting the requirements of data integrity, responsiveness, fairness and scalability. It is well known that a tree-based, recursive design approach caters well for scalability requirements. With this observation in mind, the paper develops an approach that permits an auction service to be mapped on to globally distributed auction servers. The paper selects a suitable auction model that treats sellers and buyers symmetrically. This symmetry enables a computational node to play at one level of the tree the role of a seller by dealing with a group of potential buyers as well as to play the role of a potential buyer at the next higher level. Such a symmetric auction (also known as a double auction) is used for supporting a standard auction to be carried out in a hierarchic manner. An architecture is developed and basic algorithms and protocols are presented.

This paper investigates the effects of relaxing the synchronisation embedded in "classical" conversation schemes. Look-ahead conversation scheme enables the synchronisation mandated by conversations to be performed concurrently to other normal system activities, and thereby provides scope for enhancing system performance. In this paper, we take the view that permitting look-ahead must guarantee that executions with and without look-ahead be equivalent for identical inputs and run-time conditions. We identify and formulate the necessary condition for meeting this objective. We then present two schemes for realising this condition. The first scheme is based on piggybacking extra information onto ongoing messages, and the second one is a message passing protocol requiring each participant to send one message to every other participant of the conversation. These schemes do not require a conversation participant to know a priori all participants of the conversation, bur only those it is designed to interact with, during the conversation.

In a distributed system, replication of components, such as objects, is a well known way of achieving availability. For increased availability, crashed and disconnected components must be replaced by new components on available spare nodes. In this context, we address the problem of reconfiguring a group after the group as an entity has failed. Such a failure is termed a group failure which, for example, can be the crash of every component in the group or the group being partitioned into minority islands. The solution assumes crash-proof storage, and eventual recovery of crashed nodes and healing of partitions. It guarantees that: (i) the number of groups reconfigured after a group failure is never more than one, and (ii) the reconfigured group contains a majority of the components which were members just before the group failed, so that the loss of state information due to group failure is minimal. The protocol is efficient in terms of communication rounds and use of stable store, during both normal operations and reconfiguration after a group failure.

Reliable sender-based one-to-many protocols do not scale well due mainly to implosion caused by the excessive rate of feedback packets arriving from receivers. We show that this problem can be circumvented by making the sender poll the receivers at carefully planned timing instants, so that the arrival rate of feedback packets is not large enough to cause implosion. We describe a generic end-to-end protocol which incorporates this polling scheme together with error and flow control mechanisms. We analyse the behaviour of our protocol using simulations which indicate that our scheme can be effective in minimising losses due to implosion, achieving high throughput with low network cost.

A fail-silent node is a self-checking node that either functions correctly or stops functioning after an internal failure is detected. Such a node can be constructed from a number of conventional processors. In a software-implemented fail-silent node, the nonfaulty processors of the node need to execute message order and comparison protocols to "keep in step" and check each other, respectively. In this paper, the design and implementation of efficient protocols for a two processor fail-silent node are described in detail. The performance figures obtained indicate that in a wide class of applications requiring a high degree of fault tolerance, software-implemented fail-silent nodes constructed simply by utilizing standard "off-the-shelf" components are an attractive alternative to their hardware-implemented counterparts that do require special-purpose hardware components, such as fault-tolerant clocks, comparator, and bus interface circuits.

A general purpose group communication protocol suite called Newtop is described. It is assumed that processes can simultaneously belong to many groups, group size could be large, and processes could be communicating over the Internet. Asynchronous communication environment is therefore assumed where message transmission times cannot be accurately estimated, and the underlying network may well get partitioned, preventing functioning processes from communicating with each other. Newtop can provide causality preserving total order delivery to members of a group, ensuring that total order delivery is preserved for multi-group processes. Both symmetric and asymmetric order protocols are supported, permitting a process to use say symmetric version in one group and asymmetric version in other.

Replicated processing with majority voting is a well known method for achieving fault tolerance. Triple Modular Redundant (TMR) processing is the most commonly used version of that method. Replicated processing requires that the replicas reach agreement on the order in which messages are to be processed. Synchronous and deterministic ordering protocols published in the literature require that the replicas maintain an abstraction of clocks that are kept in known and bounded synchronism. We present a protocol for TMR systems that does not require this abstraction of synchronised clocks. We analyse the protocol performance and show that this protocol in practice can be at least as fast as any synchronised clock based ordering protocol. We also derive a faster protocol that has an improved performance in the absence of processor failures. We then build a TMR node and measure its performance to illustrate that the protocols developed here provide faster ordering and are easier to implement.

A VOLTAN node is composed of a number of conventional processors on which application-level processes are replicated to achieve fault-tolerance. The architecture of a family of such nodes with differing functionalities is presented. These include failure-masking, fail-signal, and fail-silent nodes. The software architectures of a three-processor failure-masking and a two-processor fail-silent node are discussed in detail. The suitability of VOLTAN nodes as building blocks of reliable distributed systems is also discussed.>

An algorithm is presented by which nonfaulty processors of a group of fixed size will be able to maintain a consistent and timely knowledge of the group membership. The authors assume an architecture in which the broadcast network is accessed by some time domain multiplexing techniques where the exclusive right to transmit messages is granted to each processor once in every 'cycle'. In an execution of the proposed algorithm, every nonfaulty processor knows of any processor failure within at most two cycles following the cycle in which the failure occurred, and a restarted processor can join the group in two cycles. At most less than half the number of processors are assumed to fail in any three consecutive cycles.>

A distributed system in which a job can be broken into a number of subjobs which are processed sequentially at various processors is considered. The performance of such a system is then compared to the replicated (triple modular redundant, or TMR) version of the system in which each subjob will require concurrent replicated processing with majority voting. The effect of voting times and processor failure rates on the performance of the system is investigated with analytical approximations and computer simulations. The accuracy of the former is examined. The results indicate the possible existence of a threshold voting time, below which the TMR system performs better than the unreplicated one, and above which the situation is reversed. Such thresholds are observed, where possible, in systems with repairable servers, as well as in those with nonrepairable servers.>

Replicated processing with majority voting is a well known method of achieving fault tolerance. We consider the problem of constructing a distributed system composed of an arbitrarily large number of N-modular redundant (NMR) nodes, where each node itself is composed of N, N = 2m + 1 and m ≥ 1, processing and voting elements. Advanced microprocessors, such as Inmos Transputers, provide fast serial communication links for inter-processor communication, making it possible to construct large networks of processors. We describe how replicated processing with majority voting can be achieved for such processor networks. This paper will present the overall systems architecture, including voting and NMR synchronization algorithms specially developed to exploit fast point to point communication facilities.