通过预写式血缘实现流水线查询引擎的高效容错
Efficient Fault Tolerance for Pipelined Query Engines via Write-ahead Lineage
AI总结：
本文提出预写式血缘技术，结合血缘重放与预写日志，实现流水线查询引擎的低开销容错，并在Quokka引擎中验证其比SparkSQL快约2倍且恢复性能相当。
摘要：
现代分布式流水线查询引擎要么不支持查询内故障容错，要么采用高开销的方法，例如持久化中间输出或对状态进行检查点。在这项工作中，我们提出了预写式血缘，一种新颖的故障恢复技术，它结合了Spark基于血缘的重放和预写式日志记录。与Spark中血缘在查询执行前确定不同，预写式血缘在运行时持久化记录血缘，以支持流水线查询引擎中的动态任务依赖。由于仅持久化KB级别的血缘而非MB级别的中间输出，与基于溢出或检查点的方法相比，正常执行开销极小。为确保快速故障恢复时间，任务仅消费具有持久化血缘的中间输出，从而防止故障时的全局回滚。此外，来自不同阶段的丢失任务可以以流水线并行方式恢复。我们在一个名为Quokka的分布式流水线查询引擎中实现了预写式血缘。我们表明，在TPC-H基准测试中，Quokka比SparkSQL快约2倍，同时具有相似的故障恢复性能。
英文摘要：
Modern distributed pipelined query engines either do not support intra-query fault tolerance or employ high-overhead approaches such as persisting intermediate outputs or checkpointing state. In this work, we present write-ahead lineage, a novel fault recovery technique that combines Spark's lineage-based replay and write-ahead logging. Unlike Spark, where the lineage is determined before query execution, write-ahead lineage persistently logs lineage at runtime to support dynamic task dependencies in pipelined query engines. Since only KB-sized lineages are persisted instead of MB-sized intermediate outputs, the normal execution overhead is minimal compared to spooling or checkpointing based approaches. To ensure fast fault recovery times, tasks only consume intermediate outputs with persisted lineage, preventing global rollbacks upon failure. In addition, lost tasks from different stages can be recovered in a pipelined parallel manner. We implement write-ahead lineage in a distributed pipelined query engine called Quokka. We show that Quokka is around 2x faster than SparkSQL on the TPC-H benchmark with similar fault recovery performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
