通过软节流增强异步SGD的鲁棒性
Robustifying Asynchronous SGD via Soft Throttling
浏览论文内容
中文总结 AI 辅助
本文提出Throttle算法，通过指数降权快速客户端更新，统一了异步SGD与同步拜占庭鲁棒SGD，理论分析收敛性并实验验证了鲁棒性与性能提升。
中文摘要 AI 辅助
异步SGD是分布式学习中一种流行的算法，其中每个客户端的梯度更新在到达时即被应用。这带来了加速，但也增加了对攻击的脆弱性，因为快速的客户端可能主导总更新。我们引入了Throttle，一种异步SGD的拜占庭鲁棒泛化，其关键思想是以因子$q$指数地降低来自较快客户端的更新的权重。异步SGD（$q=1$）和同步拜占庭鲁棒SGD（$q\to\infty$）都对应于Throttle的特定设置。我们提供了收敛速度的理论分析，并在理论上和经验上验证了对攻击的鲁棒性。值得注意的是，我们的实验表明，即使在非拜占庭设置中，这种降权机制也能提高相对于标准异步SGD的性能。
英文摘要
Asynchronous SGD is a popular algorithm for distributed learning where each client's gradient update is applied on arrival. This leads to a speed-up, but also an increased vulnerability to attacks, as fast clients can dominate the total update. We introduce Throttle, a Byzantine-robust generalization of asynchronous SGD where the key idea is to exponentially down-weight updates from faster clients by a factor $q$. Both asynchronous SGD ($q=1$) and synchronous Byzantine-robust SGD ($q\to\infty$) correspond to specific settings of Throttle. We provide a theoretical analysis of the convergence rate and validate the robustness to attacks both theoretically and empirically. Remarkably, our experiments show that this down-weighting mechanism can also improve performance over standard asynchronous SGD even in the non-Byzantine setting.
发表机构
- Okinawa Institute of Science and Technology（冲绳科学技术大学院大学）
- National University of Singapore（新加坡国立大学）
- Toyota Motor Corporation（丰田汽车公司）
- University of Zurich（苏黎世大学）
机构由 AI 辅助整理，请以论文原文为准。
