Presto is an open-source distributed SQL query engine that supports analytics workloads involving multiple exabyte-scale data sources. Presto is used for low-latency interactive use cases as well as long-running ETL jobs at Meta. It was originally launched at Meta in 2013 and donated to the Linux Foundation in 2019. Over the last ten years, upholding query latency and scalability with the hyper growth of data volume at Meta as well as new SQL analytics requirements have raised impressive challenges for Presto. A top priority has been ensuring query reliability does not regress with the shift towards smaller, more elastic container allocation, which requires queries to run with substantially smaller memory headroom and can be preempted at any time. Additionally, new demands from machine learning, privacy, and graph analytics have driven Presto maintainers to think beyond traditional data analytics. In this paper, we discuss several successful evolutions in recent years that have improved Presto latency as well as scalability by several orders of magnitude in production at Meta. Some of the notable ones are hierarchical caching, native vectorized execution engines, materialized views, and Presto on Spark. With these new capabilities, we have deprecated or are in the process of deprecating various legacy query engines so that Presto becomes the single piece to serve interactive, ad-hoc, ETL, and graph processing workloads for the entire data warehouse.

Presto is an open-source distributed SQL query engine for OLAP, aiming for “SQL on everything”. Since open-sourced in 2013, Presto has been consistently gaining popularity in large-scale data analytics and attracting adoption from a wide range of enterprises. From the development and operation of Presto, we witnessed a significant amount of CPU consumption on parsing column-oriented data files in Presto worker nodes. This blocks some companies, including Meta, from increasing analytical data volumes.In this paper, we present a metadata caching layer, built on top of the Alluxio SDK cache and incorporated in each Presto worker node, to cache the intermediate results in file parsing. The metadata cache provides two caching methods: caching the decompressed metadata bytes from raw data files and caching the deserialized metadata objects. Our evaluation of the TPC-DS benchmark on Presto demonstrates that when the cache is warm, the first method can reduce the query’s CPU consumption by 10%-20%, whereas the second method can minimize the CPU usage by 20%-40%.

Hardware execution throttling mechanisms such as duty cycle modulation and voltage/frequency scaling can effectively control core or chip-level resource consumption and hence have been advocated to manage multicore resource competition. However, finding the right throttle setting is challenging since the configuration space grows exponentially as the number of cores increases, making the naive approach of exhaustive search untenable. This paper proposes a flexible framework for Throttling-Enabled Multicore Management (TEMM) that efficiently finds a high-quality hardware execution throttling configuration for a user-specified resource management objective. In a manner similar to the Newton-Raphson method in numerical analysis, TEMM employs an iterative method to continuously improve the configuration search quality by leveraging the search results from previous iterations. Within each iteration, TEMM extrapolates the effects of throttling from reference configurations, searches for a high-quality throttling configuration based on model predictions (accelerated by hill climbing), sample-runs the selected configuration, and adds the measured performance and recorded execution statistics of interest as a new reference. Our evaluations show TEMM can quickly arrive at the exact or close to optimal throttling configuration.