Modern Internet services generate massive volumes of observability data, yet identifying opportunities for business performance improvements remains elusive. In many cases, such insights manifest only within sub-populations defined by derived attributes that cannot be predefined, might evolve over time, and often cannot be exhaustively enumerated ahead of time. Unfortunately, existing commercial and research systems fall short in one or more aspects of generating such improvement opportunities: expressiveness, automation, and scalability. We present a vision for automatically surfacing opportunities for improvements to tackle these seemingly conflicting and intractable requirements. We highlight the early promise from a proof-of-concept system, showing evaluation on three real-world services and discuss open challenges for future work.

Across many domains (e.g., media/entertainment, mobile apps, finance, IoT, cybersecurity), there is a growing need for stateful analytics over streams of events to meet key business outcomes. Stateful analytics over event streams entails carefully modeling the sequence, timing, and contextual correlations of events to dynamic attributes. Unfortunately, existing frameworks and languages (e.g., SQL, Flink, Spark) entail significant code complexity and expert effort to express such stateful analytics because of their dynamic and stateful nature. Our overarching goal is to simplify and democratize stateful analytics. Through an iterative design and evaluation process including a foundational user study and two rounds of formative evaluations with 15 industry practitioners, we created SEAM-EZ, a no-code visual programming platform for quickly creating and validating stateful metrics. SEAM-EZ features a node-graph editor, interactive tooltips, embedded data views, and auto-suggestion features to facilitate the creation and validation of stateful analytics. We then conducted three real-world case studies of SEAM-EZ with 20 additional practitioners. Our results suggest that practitioners who previously could not or had to spend significant effort to create stateful metrics using traditional tools such as SQL or Spark can now easily and quickly create and validate such metrics using SEAM-EZ.

Internet video providers rely on alerting workflows to identify and remedy incidents that can impact users (e.g., outages or buggy players). There is growing evidence for the need for viewership-based analytics---detecting and diagnosing incidents that manifest through changes in viewership patterns but not in other (e.g., QoE) metrics. However, both detection and diagnosis of viewership anomalies is challenging due to the contextual nature of anomalies, non-stationarity of viewership, and complex dependencies between the structure of events and how they impact different subpopulations of viewers. We present Proteas, an alerting framework for video viewership anomalies that tackles these challenges. Proteas builds on key spatiotemporal structural insights. First, across different sub-populations of viewers and days of the week, we find that the shape of the viewership curve remains invariant over multiple weeks, thus enabling anomaly detection. Second, we use the hierarchy of viewership groups to produce compact alerts. Finally, we find that common anomalies manifest with spatiotemporal signatures, which enables us to classify anomalies to produce actionable alerts. We evaluate Proteas using 3 months of real viewership data (including the onset of the COVID-19 pandemic) and show that Proteas is accurate with over 80% True Positive Rate, average precision of over 86% (i.e., few false positives) and doesn't miss any major events. In addition, we find that approximately half of Proteas's alerts refer to events not caught by other alerting workflows, thus adding value to operators' existing toolkit.

Modern data analytics applications typically process massive amounts of data on clusters of tens, hundreds, or thousands of machines to support near-real-time decisions.The quantity of data and limitations of disk and memory bandwidth often make it infeasible to deliver answers at interactive speeds. However, it has been widely observed that many applications can tolerate some degree of inaccuracy. This is especially true for exploratory queries on data, where users are satisfied with "close-enough" answers if they can come quickly. A popular technique for speeding up queries at the cost of accuracy is to execute each query on a sample of data, rather than the whole dataset. To ensure that the returned result is not too inaccurate, past work on approximate query processing has used statistical techniques to estimate "error bars" on returned results. However, existing work in the sampling-based approximate query processing (S-AQP) community has not validated whether these techniques actually generate accurate error bars for real query workloads. In fact, we find that error bar estimation often fails on real world production workloads. Fortunately, it is possible to quickly and accurately diagnose the failure of error estimation for a query. In this paper, we show that it is possible to implement a query approximation pipeline that produces approximate answers and reliable error bars at interactive speeds.

In this paper, we present BlinkDB, a massively parallel, approximate query engine for running interactive SQL queries on large volumes of data. BlinkDB allows users to trade-off query accuracy for response time, enabling interactive queries over massive data by running queries on data samples and presenting results annotated with meaningful error bars. To achieve this, BlinkDB uses two key ideas: (1) an adaptive optimization framework that builds and maintains a set of multi-dimensional stratified samples from original data over time, and (2) a dynamic sample selection strategy that selects an appropriately sized sample based on a query's accuracy or response time requirements. We evaluate BlinkDB against the well-known TPC-H benchmarks and a real-world analytic workload derived from Conviva Inc., a company that manages video distribution over the Internet. Our experiments on a 100 node cluster show that BlinkDB can answer queries on up to 17 TBs of data in less than 2 seconds (over 200 x faster than Hive), within an error of 2-10%.

Video traffic already represents a significant fraction of today's traffic and is projected to exceed 90% in the next five years. In parallel, user expectations for a high quality viewing experience (e.g., low startup delays, low buffering, and high bitrates) are continuously increasing. Unlike traditional workloads that either require low latency (e.g., short web transfers) or high average throughput (e.g., large file transfers), a high quality video viewing experience requires sustained performance over extended periods of time (e.g., tens of minutes). This imposes fundamentally different demands on content delivery infrastructures than those envisioned for traditional traffic patterns. Our large-scale measurements over 200 million video sessions show that today's delivery infrastructure fails to meet these requirements: more than 20% of sessions have a rebuffering ratio ≥ 10% and more than 14% of sessions have a video startup delay ≥ 10s. Using measurement-driven insights, we make a case for a video control plane that can use a global view of client and network conditions to dynamically optimize the video delivery in order to provide a high quality viewing experience despite an unreliable delivery infrastructure. Our analysis shows that such a control plane can potentially improve the rebuffering ratio by up to 2× in the average case and by more than one order of magnitude under stress.