arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2201.01954cs.LGmath.OCmath.STstat.MLstat.TH
收藏

平滑损失函数的联邦优化

Federated Optimization of Smooth Loss Functions

Massachusetts Institute of Technology(麻省理工学院);Purdue University(普渡大学)

Ali Jadbabaie, Anuran Makur, Devavrat Shah

AI总结:

本文研究联邦学习框架下的经验风险最小化问题,提出FedLRGD算法利用数据平滑性诱导的低秩结构来近似梯度,并证明在数据维度较小且平滑度足够时,其联邦预言机复杂度优于FedAve。

摘要:

在这项工作中,我们在联邦学习框架内研究经验风险最小化(ERM),其中中央服务器使用存储在 $m$ 个客户端上的训练数据来最小化 ERM 目标函数。在此设置中,Federated Averaging (FedAve) 算法是确定 ERM 问题 $ε$-近似解的主要方法。与标准优化算法类似,FedAve 的收敛分析仅依赖于损失函数在优化参数上的平滑性。然而,损失函数在训练数据中通常也非常平滑。为了利用这种额外的平滑性,我们提出了 Federated Low Rank Gradient Descent (FedLRGD) 算法。由于数据上的平滑性会在损失函数上诱导出近似的低秩结构,我们的方法首先在服务器和客户端之间执行几轮通信,以学习服务器可用来近似客户端梯度的权重。然后,我们的方法使用不精确梯度下降在服务器上解决 ERM 问题。为了展示 FedLRGD 相较于 FedAve 的优越性能,我们提出了一种联邦预言机复杂度的概念,作为标准预言机复杂度的对应物。在对损失函数的一些假设下(例如参数的强凸性、数据的 $η$-Hölder 平滑性等),我们证明了 FedLRGD 的联邦预言机复杂度规模约为 $ϕm(p/ε)^{Θ(d/η)}$,而 FedAve 的规模约为 $ϕm(p/ε)^{3/4}$(忽略次主导因素),其中 $ϕ\gg 1$ 是“通信与计算比”,$p$ 是参数维度,$d$ 是数据维度。然后,我们展示了当 $d$ 较小且损失函数在数据上足够平滑时,FedLRGD 在联邦预言机复杂度上优于 FedAve。最后,在分析 FedLRGD 的过程中,我们还建立了一个关于潜变量模型低秩近似的结果。

英文摘要:

In this work, we study empirical risk minimization (ERM) within a federated learning framework, where a central server minimizes an ERM objective function using training data that is stored across $m$ clients. In this setting, the Federated Averaging (FedAve) algorithm is the staple for determining $ε$-approximate solutions to the ERM problem. Similar to standard optimization algorithms, the convergence analysis of FedAve only relies on smoothness of the loss function in the optimization parameter. However, loss functions are often very smooth in the training data too. To exploit this additional smoothness, we propose the Federated Low Rank Gradient Descent (FedLRGD) algorithm. Since smoothness in data induces an approximate low rank structure on the loss function, our method first performs a few rounds of communication between the server and clients to learn weights that the server can use to approximate clients' gradients. Then, our method solves the ERM problem at the server using inexact gradient descent. To show that FedLRGD can have superior performance to FedAve, we present a notion of federated oracle complexity as a counterpart to canonical oracle complexity. Under some assumptions on the loss function, e.g., strong convexity in parameter, $η$-Hölder smoothness in data, etc., we prove that the federated oracle complexity of FedLRGD scales like $ϕm(p/ε)^{Θ(d/η)}$ and that of FedAve scales like $ϕm(p/ε)^{3/4}$ (neglecting sub-dominant factors), where $ϕ\gg 1$ is a "communication-to-computation ratio," $p$ is the parameter dimension, and $d$ is the data dimension. Then, we show that when $d$ is small and the loss function is sufficiently smooth in the data, FedLRGD beats FedAve in federated oracle complexity. Finally, in the course of analyzing FedLRGD, we also establish a result on low rank approximation of latent variable models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑