Twitter is migrating its real-time data analytics infrastructure to the cloud. We propose a complete cloud migration procedure to tackle five identified challenges. Following the proposed solution, we successfully migrated a typical Heron job to Dataflow recently. This use case implements 1) control plane of provisioning and orchestration, 2) data plane of Heron to Beam programming model translation, 3) job IOs through replication and direct access/proxying, and 4) data validation by control test and metrics monitoring. The evaluation demonstrated the new Dataflow job matches the legacy Heron job in terms of performance and SLAs. Finally, we retrospect the migration process and identify future work on automated migration.

Twitter's data platform team is serving a large number of real-time analytics jobs, powering a wide range of data science use cases, from aggregations over time to spam detection. These analytics jobs constitute a crucial step in Twitter's data science infrastructure. As a key part of Twitter's “partly cloudy” strategy, real-time data analytics jobs are being migrated from on-premises into the cloud. We would like to share our migration approach and findings in this paper. The jobs to be migrated vary but follow common patterns, including the “read-modify-write store” and “lambda architecture” patterns. Both patterns can be migrated to the Beam data model in general ways. Besides job patterns, the job IOs are handled by replicating or proxying between on-premises and the cloud. Tests are applied in two phases through monitoring metrics and control tests. A case study demonstrates the business impact of migration. Finally, we discuss lessons learned.

Real-time stream processing has become increasingly important in recent years and has led to the development of a multitude of stream processing systems. Given the varying job workloads that characterize stream processing, these systems need to be tuned and adjusted to maintain performance targets in the face of variation in incoming traffic. Current auto-scaling systems adopt a series of trials to approach a job's expected performance due to a lack of performance modelling tools. We find that general traffic trends in most jobs lend themselves well to prediction. Based on this premise, we built a system called Caladrius that forecasts the future traffic load of a stream processing job and predicts its processing performance after a proposed change to the parallelism of its operators. Experimental results show that Caladrius is able to estimate a job's throughput performance and CPU load under a given scaling configuration.