arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1801.07237cs.DB
收藏

Smoke:交互速度下的细粒度血统

Smoke: Fine-grained Lineage at Interactive Speed

Fotis Psallidas, Eugene Wu

摘要:

数据沿袭描述了工作流的各个输入和输出数据项之间的关系,并且已经充当了传统(例如,调试、审计、数据集成和安全性)和紧急(例如,交互式可视化、迭代分析、解释和清理)应用程序。沿袭系统需要解决的核心问题,也是本文的主要焦点,是捕获整个工作流中输入和输出数据项之间的关系,目标是简化对沿袭的查询。不幸的是,目前的世系系统要么招致高世系捕获开销,或世系查询处理成本,或两者兼而有之。因此,原则上可以用沿袭术语声明性地表达其逻辑的应用程序会求助于手工调优的实现。为此,我们引入了Smoke,这是一个内存数据库引擎,既不需要血统捕获开销,也不需要血统查询处理。为此,Smoke引入了将沿袭捕获逻辑紧密集成到物理数据库操作符中;用于存储的高效、写优化的沿袭表示;以及在预先知道未来的沿袭查询时进行优化。我们在微基准测试和实际工作负载上的实验表明,与最先进的替代方案相比,Smoke减少了沿袭捕获开销,并将沿袭查询简化了多个数量级。我们在真实世界应用程序上的实验强调,Smoke可以满足交互式可视化的延迟要求(例如,<150 ms),并优于数据分析原语的手写实现。

英文摘要:

Data lineage describes the relationship between individual input and output data items of a workflow, and has served as an integral ingredient for both traditional (e.g., debugging, auditing, data integration, and security) and emergent (e.g., interactive visualizations, iterative analytics, explanations, and cleaning) applications. The core, long-standing problem that lineage systems need to address---and the main focus of this paper---is to capture the relationships between input and output data items across a workflow with the goal to streamline queries over lineage. Unfortunately, current lineage systems either incur high lineage capture overheads, or lineage query processing costs, or both. As a result, applications, that in principle can express their logic declaratively in lineage terms, resort to hand-tuned implementations. To this end, we introduce Smoke, an in-memory database engine that neither lineage capture overhead nor lineage query processing needs to be compromised. To do so, Smoke introduces tight integration of the lineage capture logic into physical database operators; efficient, write-optimized lineage representations for storage; and optimizations when future lineage queries are known up-front. Our experiments on microbenchmarks and realistic workloads show that Smoke reduces the lineage capture overhead and streamlines lineage queries by multiple orders of magnitude compared to state-of-the-art alternatives. Our experiments on real-world applications highlight that Smoke can meet the latency requirements of interactive visualizations (e.g., <150ms) and outperform hand-written implementations of data profiling primitives.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑