Presto is an open-source distributed SQL query engine for OLAP, aiming for “SQL on everything”. Since open-sourced in 2013, Presto has been consistently gaining popularity in large-scale data analytics and attracting adoption from a wide range of enterprises. From the development and operation of Presto, we witnessed a significant amount of CPU consumption on parsing column-oriented data files in Presto worker nodes. This blocks some companies, including Meta, from increasing analytical data volumes.In this paper, we present a metadata caching layer, built on top of the Alluxio SDK cache and incorporated in each Presto worker node, to cache the intermediate results in file parsing. The metadata cache provides two caching methods: caching the decompressed metadata bytes from raw data files and caching the deserialized metadata objects. Our evaluation of the TPC-DS benchmark on Presto demonstrates that when the cache is warm, the first method can reduce the query’s CPU consumption by 10%-20%, whereas the second method can minimize the CPU usage by 20%-40%.

Presto is an open source distributed query engine used widely at Facebook, Uber, Twitter, Pinterest, and many other internet companies. Since open sourced in 2013, the Presto community has made several rounds of design and implementations, to support a variety of use cases, including interactive analytics, real time reporting and dashboard, ETL workloads, A/B testing, monitoring and alerts, etc. In this paper, we'd like to introduce some of the most important features and performance improvements the open source Presto community made in recent years, which enables companies running Presto at scale, supporting millions of queries per day, with hundreds of thousands of machines. Specifically, how Presto provides unified SQL on heterogeneous storage systems without data copy; how Presto deals with complex data, including nested columnar data and schema evolution; How Presto supports geospatial queries efficiently, and how file list cache works in Presto. We also talk about cluster federation, and Presto on cloud. Experimental results and our production experience could help others running interactive SQL systems at scale.

Graph analytics demand is emerging rapidly and has become one of the key parts of Twitter machine learning for driving engagement, serving most relevant content, and promoting healthier conversations. However, due to lack of infrastructure support for graph analytics, we are suffering from a long timeline and huge engineering effort for each project to deal with graphs at the Twitter scale, which blocks us from fast iteration. To bring fast and scalable graph analytics capability into Twitter, we adopted Spark GraphFrames and conducted a performance evaluation for a typical graph analytics use case -Connected Component - on one of the largest graphs at Twitter. Compared to our existing Scalding solution, Spark GraphFrames achieved more than 33x speedup in running time.

Twitter is migrating its real-time data analytics infrastructure to the cloud. We propose a complete cloud migration procedure to tackle five identified challenges. Following the proposed solution, we successfully migrated a typical Heron job to Dataflow recently. This use case implements 1) control plane of provisioning and orchestration, 2) data plane of Heron to Beam programming model translation, 3) job IOs through replication and direct access/proxying, and 4) data validation by control test and metrics monitoring. The evaluation demonstrated the new Dataflow job matches the legacy Heron job in terms of performance and SLAs. Finally, we retrospect the migration process and identify future work on automated migration.

Twitter's data platform team is serving a large number of real-time analytics jobs, powering a wide range of data science use cases, from aggregations over time to spam detection. These analytics jobs constitute a crucial step in Twitter's data science infrastructure. As a key part of Twitter's “partly cloudy” strategy, real-time data analytics jobs are being migrated from on-premises into the cloud. We would like to share our migration approach and findings in this paper. The jobs to be migrated vary but follow common patterns, including the “read-modify-write store” and “lambda architecture” patterns. Both patterns can be migrated to the Beam data model in general ways. Besides job patterns, the job IOs are handled by replicating or proxying between on-premises and the cloud. Tests are applied in two phases through monitoring metrics and control tests. A case study demonstrates the business impact of migration. Finally, we discuss lessons learned.