Modern data processing workloads, such as machine learning and graph processing, involve iterative computations to converge generated models into higher accuracy. An effective caching mechanism is vital to expedite iterative computations since the intermediate data that needs to be stored in memory grows larger over iterations, often exceeding the memory capacity. However, existing systems handle intermediate data through separate operational layers (e.g., caching, eviction, and recovery), with each layer working independently in a greedy or cost-agnostic manner. These layers typically rely on user annotations and past access patterns, failing to make globally optimal decisions for the workload.

We propose FlowKV, a persistent store tailored for large-scale state management of streaming applications. Unlike existing KV stores, FlowKV leverages information from stream processing engines by taking a principled approach toward exploiting information about how and when the applications access data. FlowKV categorizes data access patterns of window operations according to how window boundaries are set and how tuples inside a window are aggregated, and deploys customized in-memory and on-disk data structures optimized for each pattern. In addition, FlowKV takes window metadata as explicit arguments of read and write methods to predict the moment when a window is read, and then loads the tuples of windows in batches from storage ahead of time. Using the NEXMark benchmark as workload, our experiments show that Apache Flink on FlowKV outperforms Flink on RocksDB or Faster with up to 4.12× throughput gain.

When training a deep learning (DL) model, input data are pre-processed on CPUs and transformed into tensors, which are then fed into GPUs for gradient computations of model training. Expensive GPUs must be fully utilized during training to accelerate the training speed. However, intensive CPU operations for input data preprocessing (input pipeline) often lead to CPU bottlenecks; correspondingly, various DL training jobs suffer from GPU under-utilization. We propose FastFlow, a DL training system that automatically mitigates the CPU bottleneck by offloading (scaling out) input pipelines to remote CPUs. FastFlow carefully decides various offloading decisions based on performance metrics specific to applications and allocated resources, while leveraging both local and remote CPUs to prevent the inefficient use of remote resources and minimize the training time. FastFlow's smart offloading policy and mechanisms are seamlessly integrated with TensorFlow for users to enjoy the smart offloading features without modifying the main logic. Our evaluations on our private DL cloud with diverse workloads on various resource environments show that FastFlow improves the training throughput by 1 ~ 4.34X compared to TensorFlow without offloading, by 1 ~ 4.52X compared to TensorFlow with manual CPU offloading (tf.data.service), and by 0.63 ~ 2.06X compared to GPU offloading (DALI).

Apache Spark is a widely used in-memory processing system due to its high performance. For fast data processing, Spark manages in-memory data such as cached or shuffling (aggregate and sorting) data in its own managed memory pools. However, despite its sophisticated memory management scheme, we found that Spark still suffers from out-of-memory (OOM) exceptions and high garbage collection (GC) overheads when wild memory consumers, who are not tracked by Spark and execute external codes, use a large amount of memory. To resolve the problems, we propose PokéMem, which is an enhanced Spark that incorporates wild memory consumers into the managed ones to prevent them from taking up memory spaces excessively in stealth. Our main idea is to open the black-box of unmanaged memory regions in external codes by providing customized data collections. PokéMem enables fine-grained controls of created objects within running tasks, by spilling and reloading the objects of custom data collections based on the memory pressure and access patterns. To further reduce memory pressures, PokéMem exploits pre-built memory estimation models to predict the external code's memory usage and proactively acquires memory before the execution of external code, and also performs JVM heap-usage monitoring to avoid critical memory pressures. With the help of these techniques, our evaluations show that PokéMem outperforms vanilla Spark with at most 3× faster execution with 3.9× smaller GC overheads, and successfully runs workloads without OOM exception that vanilla Spark has failed to run.

Nowadays, large numbers of small IoT stream queries are created from diverse IoT applications and executed on cloud backend servers. However, existing distributed stream processing systems such as Storm and Flink do not efficiently handle the large numbers of IoT stream queries because of their tightly-coupled query/code submission layer and inefficient query execution layer. In this paper, we propose Pluto, a new IoT-aware stream processing system. As a first step for IoT stream processing, this paper focuses on optimizing the execution of many IoT stream queries on a node. Pluto optimizes the end-to-end query processing with a three-phase execution, harnessing IoT-query characteristics. First, Pluto minimizes bottlenecks in the IoT query submission by decoupling the code registration from the query submission process with new APIs, which eliminates duplicate code registration and enables code sharing across queries. Second, in the execution phase, Pluto shares system resources as much as possible and minimizes resource bottlenecks in a machine by exploiting commonalities among IoT stream queries and information exposed in the API. Our evaluations show that Pluto improves the throughput by an order of magnitude compared to other stream processing systems on a 24-core machine, keeping P99 latency less than one second.

Optimizing scheduling and communication of distributed data processing for resource and data characteristics is crucial for achieving high performance. Existing approaches to such optimizations largely fall into two categories. First, distributed runtimes provide low-level policy interfaces to apply the optimizations, but do not ensure the maintenance of correct application semantics and thus often require significant effort to use. Second, policy interfaces that extend a high-level application programming model ensure correctness, but do not provide sufficient fine control. We describe Apache Nemo, an optimization framework for distributed dataflow processing that provides fine control for high performance and also ensures correctness for ease of use. We combine several techniques to achieve this, including an intermediate representation of dataflow, compiler optimization passes, and runtime extensions. Our evaluation results show that Nemo enables composable and reusable optimizations that bring performance improvements on par with existing specialized runtimes tailored for a specific deployment scenario. Apache Nemo is open-sourced at https://nemo.apache.org as an Apache incubator project.

Resource Managers like YARN and Mesos have emerged as a critical layer in the cloud computing system stack, but the developer abstractions for leasing cluster resources and instantiating application logic are very low level. This flexibility comes at a high cost in terms of developer effort, as each application must repeatedly tackle the same challenges (e.g., fault tolerance, task scheduling and coordination) and reimplement common mechanisms (e.g., caching, bulk-data transfers). This article presents REEF, a development framework that provides a control plane for scheduling and coordinating task-level (data-plane) work on cluster resources obtained from a Resource Manager. REEF provides mechanisms that facilitate resource reuse for data caching and state management abstractions that greatly ease the development of elastic data processing pipelines on cloud platforms that support a Resource Manager service. We illustrate the power of REEF by showing applications built atop: a distributed shell application, a machine-learning framework, a distributed in-memory caching system, and a port of the CORFU system. REEF is currently an Apache top-level project that has attracted contributors from several institutions and it is being used to develop several commercial offerings such as the Azure Stream Analytics service.