arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.02610cs.LGcs.DC
收藏

面向采用8位浮点进行端侧训练与通信的联邦学习

Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point

Texas A&M University(德克萨斯大学);Arm(恩智浦);Lund University(隆德大学)

Bokun Wang, Axel Berg, Durmus Alp Emre Acar, Chuteng Zhou

AI总结:

本研究针对联邦学习场景,提出结合客户端FP8训练与服务器端FP32全局模型的新方法,在保证模型精度的同时实现至少2.9倍的通信量降低,并提供了收敛性分析。

摘要:

近期研究表明,与采用FP32/FP16训练相比,8位浮点(FP8)可用于高效训练神经网络,能降低计算成本。本研究探讨了FP8训练在联邦学习场景中的应用。该方法不仅具备FP8的常规优势——这类优势对边缘端的端侧训练十分理想,还因显著的权重量压缩降低了客户端与服务器间的通信成本。我们提出了一种结合客户端FP8训练、同时保留服务器端FP32全局模型的新方法,并给出了收敛性分析。在多种机器学习模型和数据集上开展的实验表明,为达到相同的训练模型精度,与FP32基线相比,我们的方法在各类任务和模型中始终能实现至少2.9倍的通信量降低。

英文摘要:

Recent work has shown that 8-bit floating point (FP8) can be used for efficiently training neural networks with reduced computational cost compared to training in FP32/FP16. In this work, we investigate the use of FP8 training in a federated learning context. This approach brings not only the usual benefits of FP8 which are desirable for on-device training at the edge, but also reduces client-server communication costs due to significant weight compression. We present a novel method for combining FP8 client training while maintaining a global FP32 server model and provide convergence analysis. Experiments with various machine learning models and datasets show that our method consistently yields communication reductions of at least 2.9x across a variety of tasks and models compared to an FP32 baseline to achieve the same trained model accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑