PolyPipe: Merging Data Pipelines and Multi-Model Databases
Abstract
Modern data is characterized by its high volume and inherent heterogeneity, primarily managed by systems tailored to three distinct modeling paradigms: the relational model, which enforces strict schema and high structural integrity; the document model, which offers schema flexibility for semi-structured data; and the graph model, which prioritizes modeling complex relationships between entities. While the database industry is trending toward multi-model systems that incorporate features from all paradigms, data management practices still lag behind. Data scientists rely on manual, multi-stage and labor-intensive workflows to integrate disparate data sources. This process forces users to switch tools, results in high data shipping costs, and forfeits database-level optimizations and structural guarantees, leading to complex, brittle and non-reusable “one-off” solutions. We argue that embedding data pipelines directly into a multimodel database offers significant benefits, including streamlining, simplification, and improved maintainability, by utilizing declarative, database-native operators. This paper presents PolyPipe, an extension to the Polypheny multi-model database system. PolyPipe integrates data pipeline functionality as a first-class citizen, allowing the construction of complex pipelines using a hybrid of database and classical operators within a single system. i. Exploration: The initial assessment of data structure and quality. In heterogeneous environments, this forces the user to switch between disparate tools (e.g., SQL clients, JSON viewers, graph visualizers) just to understand the available datasets, preventing a holistic view of the data landscape. ii. Integration: The cleaning, transformation, and joining of data sources. This is often the most labor-intensive phase; without a unified model, it typically requires extracting data into application-layer scripts (e.g., Python or R), resulting in high data shipping costs and the loss of database-level optimizations. iii. Analysis: The application of complex logic or aggregation. When pipelines are decoupled from the storage engine, analytical queries cannot leverage the structural guarantees or indexes of the underlying data models, significantly reducing performance. iv. Visualization: The presentation of results. In disjointed pipelines, the link between the final visual output and the raw source data is often weakened or severed, making it difficult to trace anomalies or verify data lineage dynamically.
Assigned reviewers
No reviewers assigned yet.
Candidates from the panel ranked by taxonomy affinity
| # | Reviewer | Match | Load | Why |
|---|