使用无损同态压缩加速分布式深度学习
Accelerating Distributed Deep Learning using Lossless Homomorphic Compression
AI总结：
提出一种无损同态压缩算法，将worker级压缩与网内聚合相结合，无需CPU/GPU处理即可高效聚合且不损失训练精度，在多种DNN模型上显著提升了聚合吞吐量和训练速度。
摘要：
随着深度神经网络（DNNs）的复杂度和规模不断增长，分布式训练期间通信开销的增加已成为一个重大瓶颈，挑战着分布式训练系统的可扩展性。现有解决方案虽然旨在通过worker级压缩和网内聚合来缓解这一瓶颈，但由于无法有效调和压缩效果与计算开销之间的权衡，阻碍了整体性能和可扩展性，因此存在不足。在本文中，我们引入了一种新颖的压缩算法，有效地将worker级压缩与网内聚合相结合。我们的解决方案既是同态的，允许在没有CPU/GPU处理的情况下进行高效的网内聚合，又是无损的，确保不妥协训练准确性。在压缩和计算效率方面具有理论最优性，我们的方法在NCF、LSTM、VGG19和BERT-base等多种DNN模型上得到了实证验证，显示出聚合吞吐量高达6.33倍的提升，以及每次迭代训练速度3.74倍的增长。
英文摘要：
As deep neural networks (DNNs) grow in complexity and size, the resultant increase in communication overhead during distributed training has become a significant bottleneck, challenging the scalability of distributed training systems. Existing solutions, while aiming to mitigate this bottleneck through worker-level compression and in-network aggregation, fall short due to their inability to efficiently reconcile the trade-offs between compression effectiveness and computational overhead, hindering overall performance and scalability. In this paper, we introduce a novel compression algorithm that effectively merges worker-level compression with in-network aggregation. Our solution is both homomorphic, allowing for efficient in-network aggregation without CPU/GPU processing, and lossless, ensuring no compromise on training accuracy. Theoretically optimal in compression and computational efficiency, our approach is empirically validated across diverse DNN models such as NCF, LSTM, VGG19, and BERT-base, showing up to a 6.33$\times$ improvement in aggregation throughput and a 3.74$\times$ increase in per-iteration training speed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
