一种兼容 All-Reduce 的 Top-K 压缩器:面向通信高效的分布式学习
An All-Reduce Compatible Top-K Compressor for Communication-Efficient Distributed Learning
AI总结:
提出 ARC-Top-K 压缩器,通过梯度草图对齐稀疏模式实现无索引 All-Reduce,兼顾 Top-K 性能与 Rand-K 鲁棒性,训练时间最多减少 60.7%。
摘要:
通信仍然是大规模分布式机器学习的核心瓶颈,梯度稀疏化已成为缓解这一挑战的有前景策略。然而,现有梯度压缩器面临显著局限:Rand-K 丢弃结构信息且实际表现不佳,而 Top-K 保留有效条目但失去收缩性并需昂贵的 All-Gather 操作。本文提出 ARC-Top-K,一种兼容 All-Reduce 的 Top-K 压缩器,通过梯度的轻量级草图对齐节点间稀疏模式,实现无索引 All-Reduce 同时保留全局重要信息。ARC-Top-K 可证明是收缩的,与动量误差反馈(EF21M)结合时,在标准假设下实现线性加速并比原始 EF21M 获得更尖锐的收敛速率。实验表明,ARC-Top-K 匹配 Top-K 精度,同时将实际训练时间最多减少 60.7%,提供结合 Rand-K 鲁棒性与 Top-K 强性能的高效可扩展方案。
英文摘要:
Communication remains a central bottleneck in large-scale distributed machine learning, and gradient sparsification has emerged as a promising strategy to alleviate this challenge. However, existing gradient compressors face notable limitations: Rand-$K$ discards structural information and performs poorly in practice, while Top-$K$ preserves informative entries but loses the contraction property and requires costly All-Gather operations. In this paper, we propose ARC-Top-$K$, an {All-Reduce}-Compatible Top-$K$ compressor that aligns sparsity patterns across nodes using a lightweight sketch of the gradient, enabling index-free All-Reduce while preserving globally significant information. ARC-Top-$K$ is provably contractive and, when combined with momentum error feedback (EF21M), achieves linear speedup and sharper convergence rates than the original EF21M under standard assumptions. Empirically, ARC-Top-$K$ matches the accuracy of Top-$K$ while reducing wall-clock training time by up to 60.7\%, offering an efficient and scalable solution that combines the robustness of Rand-$K$ with the strong performance of Top-$K$.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。