IncreQueryFusion: On-demand Data Fusion Framework in Dynamic Data Lakes
Abstract
In data lakes, efficient data fusion is essential yet challenging in dynamic scenarios due to heterogeneous multi-source data. Batch-based methods ensure high accuracy but suffer from high latency, while emerging on-demand fusion approaches, despite offering real-time responsiveness, lack native support for incremental scenarios and often struggle with query evidence constraints. To overcome these issues, we propose IncreQueryFusion, an end-to-end on-demand fusion framework tailored for dynamic data lakes. First, it introduces a Temporal Index Evidence Retrieval module to incrementally retrieve and selectively expand evidence for subsequent fusion. Then, an adaptive aligned hierarchical fusion mechanism organizes data values and performs evidence-aware truth inference. Finally, a dynamic fusion truth maintenance algorithm refreshes fusion results efficiently without full recomputation. A comprehensive evaluation of established benchmarks and our newly constructed dynamic datasets demonstrates the effectiveness of IncreQueryFusion. It outperforms the state-of-the-art baseline in accuracy by 5\%-30\% on static workloads, while achieving speedups of over 10$\times$ against on-demand methods and over 100$\times$ against batch methods in dynamic scenarios.
Assigned reviewers
No reviewers assigned yet.
Candidates from the panel ranked by taxonomy affinity
| # | Reviewer | Match | Load | Why |
|---|