证明门控发布：无服务器数据湖屋的提交前内容完整性验证
Proof-Gated Publication: Verify-Before-Commit Content Integrity for Serverless Data-Mesh Lakehouses
浏览论文内容
中文总结 AI 辅助
针对无服务器联邦数据写入的内容完整性问题，提出PVDM协议，通过四阶段机制捕获所有注入故障，验证开销低且可扩展。
中文摘要 AI 辅助
联邦数据网格赋予领域团队对其数据产品的所有权，而无服务器计算是领域写入的理想底层载体。这两种趋势都削弱了发布时的正确性。Apache Iceberg和Delta Lake等开放表格式保证提交是原子性的，且读者能看到隔离的快照，但无法保证持久化的行与作业打算写入的行一致；发布结果由写入者的退出状态决定。无服务器作业若静默删除分区、重试时截断文件或重复数据块，仍会生成有效、原子、隔离但错误的快照。本文提出PVDM，一种包含四个阶段的证明门控发布协议：物理阶段（写入可回滚的暂存区）、验证阶段（密钥化多重集证明，证明写入内容等于声明的意图，由独立公证人存储）、持久阶段（跨无服务器重试重放完成的数据块）和元数据阶段（仅当证明通过时才提交目录）。元数据仅在证明通过时提交，证明失败则不会产生消费者可见的快照。验证原语是基于标识和内容投影的密钥化增量多重集哈希，可区分缺失或重复的行与损坏的值。无依赖的参考门、含30个案例的对抗套件及可复现的基准测试，在最多100万行的注入故障中，8000个故障全部被捕获且无错误阻止。我们还在Apache Spark 4.0和Apache Iceberg 1.11上对PVDM进行端到端运行，数据规模达1亿行，在此过程中所有注入故障均在实际提交路径上被阻止，门控发布成本约为17毫秒（与表规模无关），验证开销约为写入的五分之一。这些原语属于现有技术，本文的贡献是将它们组合成一种故障关闭、经公证的提交前验证协议，用于无服务器联邦写入。
英文摘要
Federated data meshes give domain teams ownership of their data products, and serverless compute is an attractive substrate for domain-owned writes. Both trends weaken correctness at publication. Open table formats such as Apache Iceberg and Delta Lake guarantee that a commit is atomic and that readers see an isolated snapshot, but not that the rows persisted equal the rows the job intended to write; publication is decided from the writer's exit status. A serverless job that silently drops a partition, truncates a file on retry, or duplicates a chunk still produces a valid, atomic, isolated, and wrong snapshot. This paper presents PVDM, a proof-gated publication protocol with four phases: Physical (write to rollbackable staging), Verify (a keyed multiset proof that written content equals declared intent, stored by an independent notary), Durable (replay completed chunks across serverless retries), and Metadata (commit the catalog last, only if the proof passed). Metadata commits only if the proof passes, so a failing proof yields no consumer-visible snapshot. The verification primitive is a keyed, incremental multiset hash over identity and content projections, telling missing or duplicated rows apart from corrupted values. A dependency-free reference gate, a thirty-case adversarial suite, and a reproducible benchmark catch eight thousand of eight thousand injected faults up to one million rows with no false blocks. We also run PVDM end-to-end on Apache Spark 4.0 and Apache Iceberg 1.11 at up to one hundred million rows, where every injected fault is blocked on the real commit path, the gated publish costs about seventeen milliseconds regardless of table size, and verification overhead is about a fifth of the write. The primitives are prior art; the contribution is composing them into a fail-closed, notarized, verify-before-commit protocol for serverless federated writes.
