In large-scale data-parallel analytics, shuffle, or the cross-network read and aggregation of partitioned data between tasks with data dependencies, usually brings in large overhead. To reduce shuffle overhead, we present SCache, an open source plug-in system that particularly focuses on shuffle optimization. By extracting and analyzing shuffle dependencies prior to the actual task execution, SCache can adopt heuristic pre-scheduling combining with shuffle size prediction to pre-fetch shuffle data and balance load on each node. Meanwhile, SCache takes full advantage of the system memory to accelerate the shuffle process. We have implemented SCache and customized Spark to use it as the external shuffle service and co-scheduler. The performance of SCache is evaluated with both simulations and testbed experiments on a 50-node Amazon EC2 cluster. Those evaluations have demonstrated that, by incorporating SCache, the shuffle overhead of Spark can be reduced by nearly 89%, and the overall completion time of TPC-DS queries improves 40% on average.

Data-parallel applications often generate hundreds of flows at the same time in data centers. Since these flows are always connected with application context, traditional flow-level optimization policies are hard to perform well in such collections. The coflow abstraction brings hope and opportunity to make the scheduling much more efficient. But exsiting schedule systems based on that related concept are either static (such as Varys) or impracticable (such as Baraat). In this paper, we address these limitations by presenting Seagull -- a dynamic precise coflow scheduling system to optimize the average CCT (Coflow Completion Time) and guaranteeing predictable completions within coflow deadlines. It's a centralized system which can share the bandwidth resources with background flows in the data center. Our experiments show that 80% CCT of the coflows is about 1.7× faster than Varys. As for deadline meeting, Seagull can guarantee about 50% of admitted coflows finishing within their deadline, which is 10% more precise than Varys.