As millions of people interact with the Twitter platform across various different products, their actions generate petabytes of data which is used to make their experience and usage better. The Data Ingestion framework is responsible for providing a platform for aggregating incoming data from services backing Twitter products and generating well defined datasets for the Twitter engineers to build data driven products. At Twitter’s data scale of tens of petabytes incoming data every day, data ingestion is offered in two modes. Batch based ingestion aggregates data optimized for throughput and streaming based ingestion aggregates events as they arrive, with low latency. Each option presents a trade off for cost vs latency. In this paper we describe the architecture behind both the options, break down components and compare them against important metrics. We talk about the importance of different frameworks and further provide details of tools available to the developers, who can then choose the right option for their use case.

Data Analytics at Twitter rely on trillions of events generated daily by micro services backing the Twitter platform. Multiple features on Twitter Mobile Application and Web interface are backed by micro services which emit events triggered by user actions. Events are well defined structured objects with fields containing important information relevant to the feature it represents. These events are aggregated and processed before making it available for internal consumption at various storage systems. Event processing pipelines in the past were designed to support scale in the order of billions of events per minute [1]. Events flowing through various software systems were batched to optimize for throughput in favor of latency. These batched event pipelines observe an end to end latency of a few hours, from the time when the event is emitted to when it is ready to be consumed. Project Sparrow is redesigning this pipeline to reduce the event latency from hours to seconds (or minutes).