arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.13682cs.DBcs.LG
收藏

数据湖上可复现的数据科学:基于Bauplan和Nessie的可重放数据管道

Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie

Bauplan;NYU Tandon(纽约大学坦登工程学院)

Jacopo Tagliabue, Ciro Greco

AI总结:

本文针对数据湖上数据工作负载难以复现的问题,提出结合Bauplan系统与开源目录Nessie,将计算与数据管理解耦,在对象存储上实现时间旅行和分支语义,仅需几条CLI命令即可实现完整的数据管道可复现性。

摘要:

随着Lakehouse架构日益普及,确保数据湖上数据工作负载的可复现性成为数据工程师的关键关注点。然而,实现可复现性仍具挑战。数据管道的庞大规模导致测试与迭代缓慢,而业务逻辑与数据管理的交织使调试复杂化并增加出错风险。本文重点介绍Bauplan在应对该挑战方面的最新进展。我们引入了一个旨在将计算与数据管理解耦的系统,通过结合云运行时与Nessie(一个具备Git语义的开源目录)来实现。通过展示该系统的能力,我们演示了其在对象存储之上提供时间旅行和分支语义的能力,并仅需几条CLI命令即可提供完整的管道可复现性。

英文摘要:

As the Lakehouse architecture becomes more widespread, ensuring the reproducibility of data workloads over data lakes emerges as a crucial concern for data engineers. However, achieving reproducibility remains challenging. The size of data pipelines contributes to slow testing and iterations, while the intertwining of business logic and data management complicates debugging and increases error susceptibility. In this paper, we highlight recent advancements made at Bauplan in addressing this challenge. We introduce a system designed to decouple compute from data management, by leveraging a cloud runtime alongside Nessie, an open-source catalog with Git semantics. Demonstrating the system's capabilities, we showcase its ability to offer time-travel and branching semantics on top of object storage, and offer full pipeline reproducibility with a few CLI commands.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑