arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02335cs.DCcs.AIcs.DB
收藏

构建正确性由设计保证的湖house。数据合同、版本控制和事务性管道为人和代理

Building a Correct-by-Design Lakehouse. Data Contracts, Versioning, and Transactional Pipelines for Humans and Agents

Columbia University(哥伦比亚大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Bauplan Labs(Bauplan实验室)

Weiming Sheng, Jinlang Wang, Manuel Barros, Aldrin Montana, Jacopo Tagliabue, Luca Bigon

AI总结:

本文提出Bauplan,一种以代码为核心的湖house,通过类型表合同、Git式数据版本控制和事务性运行机制,消除并发变更导致的失败问题,提升数据管道的正确性和可重复性。

摘要:

湖house如今是分析和AI的默认基础层,但在并发和不可信的变更下仍脆弱:模式不匹配常在运行时才显现,开发与生产容易分离,多表管道在失败后可能暴露部分结果。我们提出Bauplan,一种以代码为核心的湖house,旨在通过构建消除广泛类别的失败。Bauplan基于已提供原子单表快照演化的存储基础层,并增加三种管道级正确性机制:类型表合同使转换边界可检查,Git式数据版本控制支持可重复协作和审查,以及事务性运行保证整个管道执行的原子发布。我们描述了系统设计,展示了这些抽象如何整合成统一的编程模型,适用于人类和代理,并报告了轻量级Alloy模型的早期结果,该模型既验证了关键直觉,也暴露了事务分支可见性周围的细微反例。我们的经验表明,湖house的正确性最好通过限制编程模型,使许多非法状态变得不可表示,而不是事后修补失败。

英文摘要:

Lakehouses are now the default substrate for analytics and AI, but they remain fragile under concurrent, untrusted change: schema mismatches often surface only at runtime, development and production easily diverge, and multi-table pipelines can expose partial results after failure. We present Bauplan, a code-first lakehouse that aims to eliminate a broad class of these failures by construction. Bauplan builds on a storage substrate that already provides atomic single-table snapshot evolution, and adds three pipeline-level correctness mechanisms: typed table contracts to make transformation boundaries checkable, Git-like data versioning to support reproducible collaboration and review, and transactional runs that guarantee atomic publication of an entire pipeline execution. We describe the system design, show how these abstractions fit together into a unified programming model for humans and agents, and report early results from a lightweight Alloy model that both validates key intuitions and exposes subtle counterexamples around transactional branch visibility. Our experience suggests that correctness in the lakehouse is best addressed not by patching failures after the fact, but by restricting the programming model so that many illegal states become unrepresentable.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑