联邦学习中间接多终端信源编码的和率失真函数
Sum-Rate-Distortion Function for Indirect Multiterminal Source Coding in Federated Learning
摘要：
联邦学习（FL）的主要焦点之一是通信效率，因为在每一轮模型训练中，大量参与的边缘设备将其更新发送到边缘服务器。现有的作品从边缘设备重建每个模型更新，并隐含地假设本地模型更新是独立的边缘设备。然而，在FL中，模型更新是间接的多终端信源编码问题，也称为CEO问题，其中每个边缘设备不能直接观察到要在解码器处重构的梯度，而是仅提供有噪声的版本。现有的工作不利用冗余的信息传输的不同边缘。本文研究FL中间接多端信源编码问题的码率域，目标是在一个特定的梯度方差上界下获得最小可达码率。在无偏估计下得到了二次向量高斯CEO问题的速率域，并在边缘设备和维数上的梯度相同的特殊情况下，得到了和速率失真函数的显式表达式.最后，我们分析了凸Minibatched SGD和非凸Minibatched SGD的通信效率的基础上的和率失真函数，分别。
英文摘要：
One of the main focus in federated learning (FL) is the communication efficiency since a large number of participating edge devices send their updates to the edge server at each round of the model training. Existing works reconstruct each model update from edge devices and implicitly assume that the local model updates are independent over edge devices. In FL, however, the model update is an indirect multi-terminal source coding problem, also called as the CEO problem where each edge device cannot observe directly the gradient that is to be reconstructed at the decoder, but is rather provided only with a noisy version. The existing works do not leverage the redundancy in the information transmitted by different edges. This paper studies the rate region for the indirect multiterminal source coding problem in FL. The goal is to obtain the minimum achievable rate at a particular upper bound of gradient variance. We obtain the rate region for the quadratic vector Gaussian CEO problem under unbiased estimator and derive an explicit formula of the sum-rate-distortion function in the special case where gradient are identical over edge device and dimension. Finally, we analyse communication efficiency of convex Minibatched SGD and non-convex Minibatched SGD based on the sum-rate-distortion function, respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
