TreeRedux：在 Spark 分布式树聚合中分离关注点

中文摘要 AI 辅助

默认情况下，Apache Spark 的树聚合原语将树根放置在驱动程序上，要求驱动程序与执行器节点一样参与对中间聚合状态的相同聚合计算。对于大型聚合，这会使单一协调器面临大量的计算和内存需求。最近的 Spark 版本可选地将根移动到执行器，但完成的聚合结果仍必须返回并在驱动程序上物化。我们使用精确分位数计算和重击者识别来演示这一限制，在这些场景中，中间聚合状态可能远大于所需的最终结果。我们提出 TreeRedux，一个最小的扩展，增加了一个在执行器上执行的终端最终化操作，将聚合状态 U 映射为紧凑结果 V，从而在驱动程序上物化 V 而不是可能很大的 U。对于精确分位数计算，将 TreeRedux 应用于 GK Select 消除了驱动程序的 epsilon-n 内存项，将驱动程序内存需求降低到与 Spark 的 GK Sketch 相同的渐近阶。在我们的实验中，默认的 GK Select 实现在 25 亿个元素时遇到驱动程序内存不足错误。Spark 的执行器端最终聚合选项将此限制扩展到约 160-180 亿个元素，但仍需要在驱动程序上物化最终聚合状态。Redux Select 在 280 亿个元素内完成，没有出现驱动程序内存不足错误。TreeRedux 允许 Space-Saving 草图具有比在驱动程序上物化完整草图的最大配置大 32 倍的容量。

英文摘要

By default, Apache Spark's tree aggregation primitives place the tree root on the driver, requiring the driver to participate in the same aggregation computation over intermediate aggregation state as executor nodes. For large aggregates, this can expose the single coordinator to substantial computation and memory requirements. Recent Spark versions optionally move the root to an executor, but the completed aggregate must still be returned to and materialized on the driver. We demonstrate this limitation using exact quantile computation and heavy-hitter identification, where the intermediate aggregation state can be substantially larger than the desired final result. We propose TreeRedux, a minimal extension that adds a terminal finalize operation, executed on an executor, that maps the aggregation state U to a compact result V, so that V rather than the potentially large U is materialized on the driver. For exact quantile computation, applying TreeRedux to GK Select removes the driver's epsilon-n memory term, reducing driver memory requirements to the same asymptotic order as Spark's GK Sketch. In our experiments, the default GK Select implementation encountered a driver out-of-memory error at 2.5 billion elements. Spark's executor-side final aggregation option extended this limit to approximately 16-18 billion elements but still required the final aggregation state to be materialized on the driver. Redux Select completed through 28 billion elements without a driver out-of-memory error. TreeRedux allowed Space-Saving sketches with up to 32x the capacity of the largest configuration that materializes a full sketch on the driver.

发表机构

• University of Mississippi（密西西比大学）

机构由 AI 辅助整理，请以论文原文为准。