MapReduce and similar systems significantly ease the task of writing data-parallel code. However, many real-world computations require a pipeline of MapReduces, and programming and managing such pipelines can be difficult. We present FlumeJava, a Java library that makes it easy to develop, test, and run efficient data-parallel pipelines. At the core of the FlumeJava library are a couple of classes that represent immutable parallel collections, each supporting a modest number of operations for processing them in parallel. Parallel collections and their operations present a simple, high-level, uniform abstraction over different data representations and execution strategies. To enable parallel operations to run efficiently, FlumeJava defers their evaluation, instead internally constructing an execution plan dataflow graph. When the final results of the parallel operations are eventually needed, FlumeJava first optimizes the execution plan, and then executes the optimized operations on appropriate underlying primitives (e.g., MapReduces). The combination of high-level abstractions for parallel data and computation, deferred evaluation and optimization, and efficient parallel primitives yields an easy-to-use system that approaches the efficiency of hand-optimized pipelines. FlumeJava is in active use by hundreds of pipeline developers within Google.

In this paper, we present a new algorithm for tracking the flow of values through a program. Our algorithm represents a substantial improvement over the state of the art. Previously described value flow analyses that are control-flow sensitive do not scale well, nor do they eliminate value flow information from infeasible execution paths (i.e., they are path-insensitive). Our algorithm scales to large programs, and it is path-sensitive.The efficiency of our algorithm arises from three insights: The value flow problem can be "bit-vectorized" by tracking the flow of one value at a time; dataflow facts from different execution paths with the same value flow information can be merged; and information about complex aliasing that affects value flow can be plugged in from a different analysis.We have incorporated our analysis in ESP, a software validation tool. We have used ESP to validate the Windows operating system kernel (a million lines of code) against an important security property. This experience suggests that our algorithm scales to large programs, and is accurate enough to trace the flow of values in real code.

In this paper, we describe PSE (Postmortem Symbolic Evaluation), a static analysis algorithm that can be used by programmers to diagnose software failures. The algorithm requires minimal information about a failure, namely its kind (e.g. NULL dereference), and its location in the program's source code. It produces a set of execution traces along which the program can be driven to the given failure.