arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.07477cs.LGcs.CLcs.CR
收藏

使用私有联邦学习训练语言模型的种群扩展方法

Population Expansion for Training Language Models with Private Federated Learning

UC San Diego(加州大学圣地亚哥分校);Apple(苹果公司)

Tatsuki Koga, Congzheng Song, Martin Pelikan, Mona Chitnis

AI总结:

针对私有联邦学习在小设备种群下噪声大、延迟高的问题,提出基于领域自适应的种群扩展方法,将语言模型效用提升13%至30%。

摘要:

联邦学习(FL)与差分隐私(DP)相结合,可利用分布式设备进行机器学习(ML)训练,并提供形式化的隐私保证。当设备种群规模较大时,带DP的FL能够及时产出性能良好的模型。然而,对于种群规模较小的应用,不仅模型效用会下降,因为DP噪声与种群规模成反比,而且训练延迟也会增加,因为从较小的设备池中等待足够数量的客户端变为可用状态会更慢。因此,在本工作中,我们提出基于领域自适应技术扩展种群,以在小种群训练时加快训练速度并提升最终模型质量。我们通过实验证明,所提技术可在真实世界语言建模数据集上将效用提升13%至30%。

英文摘要:

Federated learning (FL) combined with differential privacy (DP) offers machine learning (ML) training with distributed devices and with a formal privacy guarantee. With a large population of devices, FL with DP produces a performant model in a timely manner. However, for applications with a smaller population, not only does the model utility degrade as the DP noise is inversely proportional to population, but also the training latency increases since waiting for enough clients to become available from a smaller pool is slower. In this work, we thus propose expanding the population based on domain adaptation techniques to speed up the training and improves the final model quality when training with small populations. We empirically demonstrate that our techniques can improve the utility by 13% to 30% on real-world language modeling datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑