拜占庭鲁棒的联邦表示学习

中文摘要 AI 辅助

我们研究存在对抗性客户端的联邦学习（FL），其目标是在不知道诚实（非对抗性）客户端身份的情况下最小化其平均损失。在异质性下，单一共享模型参数在统计上是不合适的：它无法捕捉不同客户端之间不同的数据生成过程，导致不可约的模型异质性偏差，并严重限制了对抗性客户端的鲁棒性（即拜占庭鲁棒性）。我们通过表示学习来解决这个问题，其中每个客户端学习一个个性化的线性头部，同时通过拜占庭鲁棒聚合协作估计一个共享的非线性表示。我们证明了诚实表示梯度之间的异质性由表示误差和统计误差控制，这些误差随着每个客户端的数据样本数量（τ）或迭代次数（T）而衰减。特别是，我们的非渐近参数恢复误差界揭示了三项：（i）随T消失的初始化相关误差，（ii）随τ和诚实客户端数量减少的有限样本噪声项，以及（iii）也随T减少的随机梯度方差项。重要的是，我们的界中没有不可约的模型异质性偏差。我们将回归分析扩展到多类分类，并在CIFAR-10、FEMNIST和学校考试成绩数据集上进行了实证验证。

英文摘要

We study federated learning (FL) with adversarial clients, where the goal is to minimize the average loss of the honest (non-adversarial) clients without knowing their identity. Under heterogeneity, a single shared model parameter is statistically inappropriate: it cannot capture the distinct data-generating processes across clients, incurring an irreducible model-heterogeneity bias and severely limiting robustness to adversarial clients (a.k.a. Byzantine-robustness). We address this problem through representation learning, where each client learns a personalized linear head, while collaboratively estimating a shared nonlinear representation through Byzantine-robust aggregation. We demonstrate that the heterogeneity among honest representation gradients is controlled by the representation error and statistical errors that decay either with the number of data samples per client ($τ$) or the number of iterations ($T$). In particular, our non-asymptotic parameter recovery error bound reveals three terms: (i) an initialization-dependent error that goes away with $T$, (ii) finite-sample noise terms that decreases with $τ$ and the number of honest clients, and (iii) a stochastic gradient variance term that also reduces with $T$. Importantly, with no irreducible model-heterogeneity bias in our bounds. We extend the regression analysis to multiclass classification, and empirically validate it on CIFAR-10, FEMNIST, and School Exam Score datasets.

发表机构

• Columbia University（哥伦比亚大学）
• Sorbonne Université（索邦大学）
• Université Paris Cité（巴黎西岱大学）
• CNRS（法国国家科学研究中心）
• University of Copenhagen（哥本哈根大学）

机构由 AI 辅助整理，请以论文原文为准。