追踪相似性：分布感知聚合调度（扩展版）
Chasing Similarity: Distribution-aware Aggregation Scheduling (Extended Version)
摘要：
并行聚合是数据分析中普遍存在的操作，在SQL中表示为GROUP BY，在Hadoop中表示为reduce，在TensorFlow中表示为segment。并行聚合从一个可选的本地预聚合步骤开始，然后跨网络重新分区中间结果。虽然本地预聚合适用于低基数聚合，但即使在本地预聚合之后，高基数聚合的网络通信成本仍然很大。问题是基于重划分的高基数聚合算法没有充分利用网络。在这项工作中，我们首先制定了一个数学模型，捕捉并行聚合的性能。我们证明了从一个已知的数据分布找到最佳的聚合计划是NP-难的，假设小集扩展猜想。我们提出了GRASP，一个GReedy聚合调度协议，分解成阶段的并行聚合。GRASP是分布感知的，因为它在每个阶段中聚合最相似的分区，以减少后续阶段中传输的数据大小。此外，GRASP考虑到可用的网络带宽时，在每个阶段的聚合调度，以最大限度地提高网络利用率。对真实数据的实验结果表明，GRASP算法的性能是基于重划分的聚集算法的3.5倍，LOOM算法的2.0倍。
英文摘要：
Parallel aggregation is a ubiquitous operation in data analytics that is expressed as GROUP BY in SQL, reduce in Hadoop, or segment in TensorFlow. Parallel aggregation starts with an optional local pre-aggregation step and then repartitions the intermediate result across the network. While local pre-aggregation works well for low-cardinality aggregations, the network communication cost remains significant for high-cardinality aggregations even after local pre-aggregation. The problem is that the repartition-based algorithm for high-cardinality aggregation does not fully utilize the network. In this work, we first formulate a mathematical model that captures the performance of parallel aggregation. We prove that finding optimal aggregation plans from a known data distribution is NP-hard, assuming the Small Set Expansion conjecture. We propose GRASP, a GReedy Aggregation Scheduling Protocol that decomposes parallel aggregation into phases. GRASP is distribution-aware as it aggregates the most similar partitions in each phase to reduce the transmitted data size in subsequent phases. In addition, GRASP takes the available network bandwidth into account when scheduling aggregations in each phase to maximize network utilization. The experimental evaluation on real data shows that GRASP outperforms repartition-based aggregation by 3.5x and LOOM by 2.0x.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
