时变有向网络下的分布式超梯度计算
Decentralized Hyper-Gradient Computation over Time-Varying Directed Networks
AI总结:
针对去中心化联邦学习中超梯度估计通信成本高、无法适配时变有向网络的问题,提出基于Push-Sum平均的最优性条件,推导得到低通信成本、适配时变有向网络的超梯度估计器,验证其收敛性并拓展了新应用场景。
摘要:
本文针对去中心化联邦学习(FL)中超梯度估计时的通信问题展开研究。去中心化FL中的超梯度用于量化客户端超参数的扰动对全局共享最优模型性能的影响。现有研究中,客户端通过在静态无向网络上传输Hessian矩阵来追踪该影响,存在两大缺陷:(i)通信成本过高;(ii)无法利用更高效鲁棒的网络,即时变有向网络。为解决上述问题,本文提出一种基于模型参数与梯度平均操作的FL替代最优性条件,并采用适用于时变有向网络的共识优化技术Push-Sum作为平均操作。由此推导得到的超梯度估计器具备两大优势:(i)仅需进行向量的Push-Sum通信;(ii)可在时变有向网络上运行。本文从理论和实验两方面验证了该估计器可收敛至真实超梯度,并进一步展示了其可支撑两项新应用:时变网络下的去中心化影响估计与个性化建模。
英文摘要:
This paper addresses the communication issues when estimating hyper-gradients in decentralized federated learning (FL). Hyper-gradients in decentralized FL quantifies how the performance of globally shared optimal model is influenced by the perturbations in clients' hyper-parameters. In prior work, clients trace this influence through the communication of Hessian matrices over a static undirected network, resulting in (i) excessive communication costs and (ii) inability to make use of more efficient and robust networks, namely, time-varying directed networks. To solve these issues, we introduce an alternative optimality condition for FL using an averaging operation on model parameters and gradients. We then employ Push-Sum as the averaging operation, which is a consensus optimization technique for time-varying directed networks. As a result, the hyper-gradient estimator derived from our optimality condition enjoys two desirable properties; (i) it only requires Push-Sum communication of vectors and (ii) it can operate over time-varying directed networks. We confirm the convergence of our estimator to the true hyper-gradient both theoretically and empirically, and we further demonstrate that it enables two novel applications: decentralized influence estimation and personalization over time-varying networks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。