THC：利用张量同态压缩加速分布式深度学习
THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression
AI总结：
针对分布式深度学习中双向梯度压缩带来的参数服务器计算开销大、压缩误差高的问题，提出支持压缩值直接聚合且兼容网内聚合的THC双向压缩框架，可显著提升训练速度。
摘要：
深度神经网络（DNN）是图像分类、计算机视觉和自然语言处理等核心应用场景的事实标准。随着DNN和数据集规模不断扩大，需要在越来越大的集群上进行分布式训练。主要瓶颈在于由此产生的通信开销——工作节点每轮都要交换模型更新（即梯度）。为解决这一瓶颈并加速训练，一种被广泛采用的方法是压缩。然而，以往的部署方案通常只是在两个方向上分别使用单向梯度压缩方案来实现双向压缩，这会导致参数服务器端产生大量计算开销，且压缩误差增大，最终造成训练时间延长、准确率下降。我们提出了张量同态压缩（THC），这是一种新型双向压缩框架，支持对压缩后的值直接聚合，从而消除了上述计算开销。此外，THC与网内聚合（INA）兼容，可实现进一步加速。我们的评估显示，与当前最优系统相比，使用THC训练代表性视觉和语言模型时，结合INA达到目标准确率的速度快1.40倍至1.47倍，使用软件参数服务器（PS）则快1.28倍至1.33倍。
英文摘要：
Deep neural networks (DNNs) are the de facto standard for essential use cases, such as image classification, computer vision, and natural language processing. As DNNs and datasets get larger, they require distributed training on increasingly larger clusters. A main bottleneck is the resulting communication overhead where workers exchange model updates (i.e., gradients) on a per-round basis. To address this bottleneck and accelerate training, a widely-deployed approach is compression. However, previous deployments often apply bi-directional compression schemes by simply using a uni-directional gradient compression scheme in each direction. This results in significant computational overheads at the parameter server and increased compression error, leading to longer training and lower accuracy. We introduce Tensor Homomorphic Compression (THC), a novel bi-directional compression framework that enables the direct aggregation of compressed values and thus eliminating the aforementioned computational overheads. Moreover, THC is compatible with in-network aggregation (INA), which allows for further acceleration. Our evaluation shows that training representative vision and language models with THC reaches target accuracy by 1.40x to 1.47x faster using INA and 1.28x to 1.33x faster using a software PS compared with state-of-the-art systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
