Business Intelligence (BI) plays a critical role in empowering modern enterprises to make informed data-driven decisions, and has grown into a billion-dollar business. Self-service BI tools like Power BI and Tableau have democratized the "dashboarding" phase of BI, by offering user-friendly, drag-and-drop interfaces that are tailored to non-technical enterprise users. However, despite these advances, we observe that the "data preparation" phase of BI continues to be a key pain point for BI users today. In this work, we systematically study around 2K real BI projects harvested from public sources, focusing on the data-preparation phase of the BI workflows. We observe that users often have to program both (1) data transformation steps and (2) table joins steps, before their raw data can be ready for dashboarding and analysis. A careful study of the BI workflows reveals that transformation and join steps are often intertwined in the same BI project, such that considering both holistically is crucial to accurately predict these steps. Leveraging this observation, we develop an Auto-Prep system to holistically predict transformations and joins, using a principled graph-based algorithm inspired by Steiner-tree, with provable quality guarantees. Extensive evaluations using real BI projects suggest that Auto-Prep can correctly predict over 70% transformation and join steps, significantly more accurate than existing algorithms as well as language-models such as GPT-4.

Data preparation is an essential step in every data-related effort, from scientific projects in academia to data-driven decision-making in industry. Typically, data preparation is not an interesting piece of a project --- it transforms raw data into a format that enables further innovative work. Because such scripts are never intended to be interesting, are project-specific, and are written in general-purpose languages, they can be tedious to understand and difficult to verify. As a result, data preparation scripts can easily become a breeding ground for poor engineering and statistical practices. Ideally, data preparation scripts are "admirably boring" --- they should serve the project, but otherwise be as simple and as standard as possible. We propose a bottom-up script standardization framework that takes a user's data preparation script and transforms it into a simpler, more standardized version of itself. Our framework takes the user's script not as an unchangeable definition of correctness, but as a sketch of the user's intent. We embedded this framework in a system called LucidScript.

Social media, self-driving cars, and traffic cameras produce video streams at large scales and cheap cost. However, storing and querying video at such scales is prohibitively expensive. We propose to treat large-scale video analytics as a data warehousing problem: Video is a format that is easy to produce but needs to be transformed into an application-specific format that is easy to query. Analogously, we define the problem of Video Extract-Transform-Load ( V-ETL ). V-ETL systems need to reduce the cost of running a user-defined V-ETL job while also giving throughput guarantees to keep up with the rate at which data is produced. We find that no current system sufficiently fulfills both needs and therefore propose Skyscraper , a system tailored to V-ETL. Skyscraper can execute arbitrary video ingestion pipelines and adaptively tunes them to reduce cost at minimal or no quality degradation, e.g., by adjusting sampling rates and resolutions to the ingested content. Skyscraper can hereby be provisioned with cheap on-premises compute and uses a combination of buffering and cloud bursting to deal with peaks in workload caused by expensive processing configurations. In our experiments, we find that Skyscraper significantly reduces the cost of V-ETL ingestion compared to adaptions of current SOTA systems, while at the same time giving robustness guarantees that these systems are lacking.

Data provenance is any information about the origin of a piece of data and the process that led to its creation. Most database provenance work has focused on creating models and semantics to query and generate this provenance information. While comprehensive, provenance information remains large and overwhelming, making it hard for data provenance systems to support data exploration. We present a new approach to provenance exploration that builds on data summarization techniques. We contribute novel summarization schemes for the provenance of aggregation queries and techniques for the fast generation of these summarization schemes. We introduce two types of summaries for aggregate queries.Impact summariestake into account the impact of specific groups of tuples in the provenance of the query on an aggregate result, andcomparative summariesallow users to compare the provenance of two aggregate results. We also present algorithms for efficient computation of these summaries, implement optimizations using data sampling and feature selection, and conduct experiments and a user survey to show the feasibility and relevance of our approaches.

Data provenance is any information about the origin of a piece of data and the process that led to its creation. Most database provenance work has focused on creating models and semantics to query and generate this information. While comprehensive, provenance information remains large and overwhelming, which can make it hard for provenance systems to support data exploration. We present a new approach to provenance exploration that builds on data summarization techniques. We contribute two novel summarization schemes for the provenance of aggregation queries: Impact summaries, and comparative summaries. We show with experiments that our techniques incur little overhead compared to basic summaries. We conduct a survey to show that our approaches are useful to users.

Provenance information can be large and overwhelming to users. We present a set of criteria that any provenance exploration tool must have and introduce Pastwatch, a provenance exploration system that adheres to those criteria. We also address the issues associated with provenance of aggregation queries, including the creation of a summarization method that makes provenance of aggregation queries manageable for users. Finally, we conduct a quantitative user study to show statistically significant results that Pastwatch makes provenance information more efficient and easier to use than standard approaches.