PolyBooks: A System for Interactive Multi-Model Querying with Provenance and Reuse

edbt26-demo-12 · David Lengweiler, Tobias Weber, Heiko Schuldt, Marco Vogt
Abstract

Data exploration, integration, organization, and analysis are critical workflows for data scientists. In recent years, tools like Jupyter Notebooks have gained significant traction by incorporating these steps into a unified repository, allowing users to modify, extend, and document complex analytical processes with ease. However, while these tools streamline analysis, they often leave data integration to the user, frequently resulting in the execution of processes on stale data. Furthermore, standard notebooks lack robust support for persisting large datasets, forcing data scientists to rely on file-based storage or ephemeral memory. Databases, especially multi-model databases, offer a convenient repository for consolidating diverse data formats and providing simplified access. In this paper, we present the combination of these two paradigms. We show how integrating notebooks with multi-model databases leverages established data models to access and persist analytical data, ultimately improving performance for data science use cases. Despite the promise of notebooks, existing solutions to these shortcomings, like integrating specialized data tools or leveraging standard single-model databases, often fail to solve the core problem efficiently. These approaches usually require additional know-how and significant adaption effort, resulting in fragmentation that counters the initial goal of notebooks of simplifying the workflow. While data scientists could benefit from having direct access to databases without manual setup to overcome Persistence Deficiencies, a traditional single-model database could only partially address these shortcomings. This is because real-world data is inherently heterogenous. As data scientists frequently deal with a mix of fully structured relational data, semi-structured documents (e.g. JSON files) and graph structures. Trying to force all these diverse data formats into a single-model structure only worsens the challenge of Manual Data Integration. Multi-model database, however, are uniquely suited to overcome these shortcomings. They are designed to support different data models natively, providing separate optimized engines to efficiently store the most common data models. Crucially, they provide unified tools, often a singular or complementary set of query languages, to seamlessly interact with and formulate complex queries across heterogenous data. To truly unify the data access layer, a new approach is necessary. This approach builds on the polystore concept, a system that acts as a direct middle-layer over existing, distributed data resources without requiring physical migration. This is the foundation of the PolyDBMS: a system that not only supports retrieval access to underlying databases but also supports modification operations. The direct integration of notebooks with a PolyDBMS system can, therefore, effectively and completely overcome all the mentioned critical limitations. In this paper, we propose PolyBooks, an extension of the computational notebook concept that embeds visual notebooks as part of PolyDBMS multi-model databases. We argue that this

Assigned reviewers

No reviewers assigned yet.

Candidates from the panel ranked by taxonomy affinity

#ReviewerMatchLoadWhy