Balancing convergence efficiency and robustness under Differential Privacy (DP) is a central challenge in Federated Learning (FL). While AdamW accelerates training and fine-tuning in large-scale models, we find that directly applying it to Differentially Private FL (DPFL) suffers from three major issues: (i) data heterogeneity and privacy noise jointly amplify the variance of second-moment estimator, (ii) DP perturbations bias the second-moment estimator, and (iii) DP amplify AdamW sensitivity to local overfitting, worsening client drift. We propose DP-FedAdamW, the first AdamW-based optimizer for DPFL. It restores AdamW under DP by stabilizing second-moment variance, removing DP-induced bias, and aligning local updates to the global descent to curb client drift. Theoretically, we establish an unbiased second-moment estimator and prove a linearly accelerated convergence rate without any heterogeneity assumption, while providing tighter $(\varepsilon,δ)$-DP guarantees. Our empirical results demonstrate the effectiveness of DP-FedAdamW across language and vision Transformers and ResNet-18. On Tiny-ImageNet (Swin-Base, $\varepsilon=1$), DP-FedAdamW outperforms the state-of-the-art (SOTA) by 5.83\%. The code is available in Appendix.
FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Models
School of Computer Science and Technology, Tianjin University(天津大学计算机科学与技术学院);Medical College, Tianjin University(天津大学医学院);School of Artificial Intelligence, Xidian University(西安电子科技大学人工智能学院);State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院通用人工智能国家重点实验室);Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China(广州黄埔 Pazhou 实验室)
DC-SGD: Differentially Private SGD with Dynamic Clipping through Gradient Norm Distribution Estimation
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Ant Group(蚂蚁集团);School of Engineering, University of Virginia(弗吉尼亚大学工程学院)
When Gradient Clipping Becomes a Control Mechanism for Differential Privacy in Deep Learning
Computer Science, University of California, Merced, CA 95343, USA(计算机科学,加州大学默塞德分校);Department of Applied Mathematics, University of California, Merced, CA 95343, USA(应用数学系,加州大学默塞德分校);Department of Innovation(创新部门);Research, North Carolina State University, Raleigh, NC, USA(研究,北卡罗来纳州立大学,拉洛城);Automation (MESA) Lab, Department of Mechanical Engineering, School of Engineering, University of California, Merced, CA 95343, USA(自动化(MESA)实验室,机械工程系,工程学院,加州大学默塞德分校)
Towards the Flatter Landscape and Better Generalization in Federated Learning under Client-level Differential Privacy
Tsinghua University(清华大学);Nanjing University of Science and Technology(南京理工大学);JD Explore Academy(京东探索研究院);Shenzhen Wisdom and Strategy Technology Co., Ltd.(深圳智策科技有限公司);University of Sydney(悉尼大学)
Balancing Utility and Privacy: Dynamically Private SGD with Random Projection
Department of Mechanical Engineering(机械工程系);Translational AI Center(转化人工智能中心);Department of Electrical and Computer Engineering(电气与计算机工程系);Iowa State University(爱荷华州立大学);Department of Energy and Power Engineering(能源与动力工程系);Tsinghua University(清华大学)
Private Gradient Descent for Linear Regression: Tighter Error Bounds and Instance-Specific Uncertainty Estimation
Paul G Allen School of Computer Science and Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);Boston University(波士顿大学);Google DeepMind(谷歌DeepMind)