发表机构
Faculty of Science & Technology Sophia University; Technical R&D Department Shendian Energy Co., Ltd.（索菲亚大学科学技术学院; 申电能源有限公司技术研发部）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对联邦学习中更新掩码的正则化留存问题，证明了客户端掩码设置与联邦平均对正则化代价的影响，在CIFAR-10上开展实验并分析了相关因素的作用。
AI 中文摘要
针对非独立同分布（non-IID）数据的联邦学习旨在寻找平坦极小值以在各客户端间实现泛化，现有方法从集中式训练中借鉴了锐度感知最小化（sharpness-aware minimization）。还有第二种获取平坦极小值的方式，即正则化来自参数更新添加的噪声，且该方式从未被迁移到联邦设置中。我们揭示了其中的原因：掩码（mask）会因优化器向尖锐方向移动而产生代价。我们证明，当每个客户端抽取自身的掩码时，联邦平均（federated averaging）会将该代价削弱恰好等于客户端队列（cohort）规模的倍数；而若为所有客户端提供相同的掩码，则会将该代价恢复，恢复倍数等于队列的逆梯度多样性（inverse gradient diversity）。在我们针对CIFAR-10的实验设置中，该倍数在最大可能为10的情况下为1.19；关闭小批量采样（minibatch sampling）会将其提升至8.96，而将数据异质性改变100倍时，该倍数保持在1.17至1.50之间。使正则化得以保留的配置训练效果过差，无法投入使用。
英文摘要
Federated learning on non-IID data seeks flat minima to generalize across clients, and existing methods borrow sharpness-aware minimization from centralized training. There is a second way to reach flat minima, in which the regularization comes for free from noise added to the parameter updates, and it has never been carried over to the federated setting as an implicit regularizer. We show the reason. Masking charges the optimizer for moving in sharp directions. We prove that when each client draws its own mask, federated averaging weakens that charge by exactly the cohort size, and that giving every client the same mask brings it back by a factor equal to the inverse gradient diversity of the cohort. In our experiment setting on CIFAR-10, that factor is 1.19 out of a possible 10. Turning off minibatch sampling raises it to 8.96, while changing data heterogeneity a thousandfold leaves it between 1.17 and 1.50. The configurations keeping the regularization train far too poorly to use.
Commentsfixed some grammatical and spelling errors and appended three citations
