All Pairs problems occur in many research fields. The all-pairs problem requires all data elements to be paired with all other data elements. With the advent of new data intensive big data applications and increase in data size, methods to reduce memory foot print and distribute to work equally across compute nodes are needed. In this paper, we propose cyclic quorum sets for all-pairs algorithm computations to reduce memory foot print. We show that the cyclic quorum sets have a unique all-pairs property that allows for minimal data replication. The cyclic quorums set based computing requires only N/√P size memory, up to 50 percent smaller than the dual N/√P array implementations proposed earlier, and significantly smaller than solutions requiring all data in each node. Computation can be distributed efficiently and more importantly and are communication-less after initial data distribution, which is a huge advantage in minimizing computation time. Scaling from 16 to 512 cores (1 to 32 compute nodes), our application experiments on a real dataset demonstrated scalability with greater than 150x (super-linear) speedup with less than 1/4th the memory usage per node in our experiments.

In this paper, we propose a greedy cycle direction heuristic to improve the generalized R redundancy quorum cycle technique. When applied using only single cycles rather than the standard paired cycles, the generalized R redundancy technique has been shown to almost halve the necessary light-trail resources in the network. Our greedy heuristic improves this cycle-based routing technique's fault-tolerance and dependability. For efficiency and distributed control, it is common in distributed systems and algorithms to group nodes into intersecting sets referred to as quorum sets. Optimal communication quorum sets forming optical cycles based on light-trails have been shown to flexibly and efficiently route both point-to-point and multipoint-to-multipoint traffic requests. Commonly cycle routing techniques will use pairs of cycles to achieve both routing and fault-tolerance, which uses substantial resources and creates the potential for underutilization. Instead, we use a single cycle and intentionally utilize R redundancy within the quorum cycles such that every point-to-point communication pairs occur in at least R cycles. Without the paired cycles the direction of the quorum cycles becomes critical to the fault tolerance performance. For this we developed a greedy cycle direction heuristic and our single fault network simulations show a reduction of missing pairs by greater than 30%, which translates to significant improvements in fault coverage.