Sift is a new consensus protocol for replicating state machines. It disaggregates CPU and memory consumption by creating a novel system architecture enabled by one-sided RDMA operations. We show that this system architecture allows us to develop a consensus protocol which centralizes the replication logic. The result is a simplified protocol design with less complex interactions between the participants of the consensus group compared to traditional protocols. The dissaggregated design also enables Sift to reduce deployment costs by sharing backup computational nodes across consensus groups deployed within the same cloud environment. The required storage resources can be further reduced by integrating erasure codes without making significant changes to our protocol. Evaluation results show that in a cloud environment with 100 groups where each group can support up to 2 simultaneous failures, Sift can reduce the cost by 56% compared to an RDMA-based Raft deployment.

Achieving consensus among a set of distributed entities (or participants) is a fundamental problem at the heart of many distributed systems. A critical problem with most consensus protocols is that they do not scale well. As the number of participants trying to achieve consensus increases, increasing network traffic can quickly overwhelm the network from topology-oblivious broadcasts, or a central coordinator for centralized consensus protocols. Thus, either achieving strong consensus is restricted to a handful of participants, or developers must resort to weaker models of consensus.

In this paper, we introduce Mayflower, a new distributed filesystem that is co-designed from the ground up to work together with a network control plane. In addition to the standard distributed filesystem components, Mayflower has a flow monitor and manager running alongside a software-defined networking controller. This tight coupling with the network controller enables Mayflower to make intelligent replica selection and flow scheduling decisions based on both filesystem and network information. It further enables Mayflower to perform global optimizations that are unavailable to conventional distributed filesystems and network control planes. Our evaluation results from both simulations and a prototype implementation show that Mayflower reduces average read completion time by more than 25% compared to current state-of-the-art distributed filesystems with an independent network flow scheduler, and more than 75% compared to HDFS with ECMP.

Bloom Filter based multicast has been proposed as a source-specific multicast solution to eliminate the multicast state requirements in the routers. However, the inherent limitation, the false positives, in the Bloom filter data structure amplifies the bandwidth wastage when the multicast tree scales to a large number of receivers. In this paper, we propose an algorithm which enhances the performance of the Bloom filter based multicast. It keeps the bandwidth waste below an acceptable upper bound while scaling the multicast tree for a large number of receivers. The large multicast tree is split into multiple smaller ones which are encoded into separate Bloom filters. Our algorithm enables multicast forwarding to be efficient - with careful setting of some parameters - for a hundreds of receivers as compared to the 20-30 receivers per group in the original technique. Furthermore, our algorithm, while slightly increasing the state requirements in the multicast sources, retains the desired property of statelessness in the intermediate routers.