arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.19945cs.LGcs.AI
收藏

DP-FedAdamW: 一种用于差分隐私联邦大模型的高效优化器

DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models

School of Cyber Engineering, Xidian University(西安电子科技大学电子信息学院);College of Intelligence and Computing, Tianjin University(天津大学智能科学与计算学院)

Jin Liu, Yinbin Miao, Ning Xi, Junkang Liu

AI总结:

本文提出DP-FedAdamW,一种针对差分隐私联邦学习的AdamW优化器,通过稳定二阶矩方差、消除DP偏差和对齐局部更新以缓解客户端漂移,提升了收敛效率和鲁棒性。

摘要:

在差分隐私(DP)下平衡收敛效率和鲁棒性是联邦学习(FL)的核心挑战。尽管AdamW在大规模模型中加速了训练和微调,但直接应用于差分隐私联邦学习(DPFL)时面临三个主要问题:(i)数据异质性和隐私噪声共同放大了二阶矩估计器的方差,(ii)DP扰动偏移了二阶矩估计器,(iii)DP放大了AdamW对局部过拟合的敏感性,加剧了客户端漂移。我们提出DP-FedAdamW,这是首个基于AdamW的DPFL优化器。它通过稳定二阶矩方差、消除DP引起的偏差以及对齐局部更新以遏制客户端漂移来恢复AdamW在DP下的性能。理论上,我们建立了无偏的二阶矩估计器,并证明了在不假设异质性的情况下具有线性加速的收敛速率,同时提供了更紧的(ε,δ)-DP保证。我们的实验证明,DP-FedAdamW在语言和视觉Transformer以及ResNet-18上均有效。在Tiny-ImageNet(Swin-Base,ε=1)上,DP-FedAdamW比现有最先进(SOTA)方法提升了5.83%。代码可在附录中获得。

英文摘要:

Balancing convergence efficiency and robustness under Differential Privacy (DP) is a central challenge in Federated Learning (FL). While AdamW accelerates training and fine-tuning in large-scale models, we find that directly applying it to Differentially Private FL (DPFL) suffers from three major issues: (i) data heterogeneity and privacy noise jointly amplify the variance of second-moment estimator, (ii) DP perturbations bias the second-moment estimator, and (iii) DP amplify AdamW sensitivity to local overfitting, worsening client drift. We propose DP-FedAdamW, the first AdamW-based optimizer for DPFL. It restores AdamW under DP by stabilizing second-moment variance, removing DP-induced bias, and aligning local updates to the global descent to curb client drift. Theoretically, we establish an unbiased second-moment estimator and prove a linearly accelerated convergence rate without any heterogeneity assumption, while providing tighter $(\varepsilon,δ)$-DP guarantees. Our empirical results demonstrate the effectiveness of DP-FedAdamW across language and vision Transformers and ResNet-18. On Tiny-ImageNet (Swin-Base, $\varepsilon=1$), DP-FedAdamW outperforms the state-of-the-art (SOTA) by 5.83\%. The code is available in Appendix.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑