FedUV：面向异构联邦学习的均匀性与方差方法
FedUV: Uniformity and Variance for Heterogeneous Federated Learning
AI总结：
针对联邦学习中数据异构导致的性能下降问题，提出基于分类器维度分布方差和编码器表征超球面均匀性的正则化方法FedUV，在高度非IID场景下性能提升显著且可扩展至更大模型与数据集。
摘要：
联邦学习是一种利用广泛分布的数据训练神经网络的极具潜力的框架。然而，当数据呈异构分布时，模型性能会大幅下降。近期研究表明，这是由于网络的最后一层最容易受到局部偏差的影响，部分研究通过将最后一层冻结为正交分类器取得了一定成效。受“冻结权重会使奇异值保持恒定”这一观察的启发，我们通过对权重应用奇异值分解（SVD）来研究分类器的训练动态。我们发现，在独立同分布（IID）和非独立同分布（non-IID）设置下训练时存在差异。基于这一发现，我们为局部训练引入了两个正则化项，以持续模拟IID设置：（1）分类器各维度概率分布的方差；（2）编码器表征的超球面均匀性。这些正则化项促使局部模型无论局部数据分布如何，都能像在IID设置中一样运行，从而在灵活适配数据的同时，抵消对偏差的易感性。在标签偏移和特征偏移两种设置下开展的大量实验中，我们验证了所提方法能以大幅优势取得最优性能，尤其是在高度non-IID的场景中，同时还能扩展至更大的模型和数据集。
英文摘要：
Federated learning is a promising framework to train neural networks with widely distributed data. However, performance degrades heavily with heterogeneously distributed data. Recent work has shown this is due to the final layer of the network being most prone to local bias, some finding success freezing the final layer as an orthogonal classifier. We investigate the training dynamics of the classifier by applying SVD to the weights motivated by the observation that freezing weights results in constant singular values. We find that there are differences when training in IID and non-IID settings. Based on this finding, we introduce two regularization terms for local training to continuously emulate IID settings: (1) variance in the dimension-wise probability distribution of the classifier and (2) hyperspherical uniformity of representations of the encoder. These regularizations promote local models to act as if it were in an IID setting regardless of the local data distribution, thus offsetting proneness to bias while being flexible to the data. On extensive experiments in both label-shift and feature-shift settings, we verify that our method achieves highest performance by a large margin especially in highly non-IID cases in addition to being scalable to larger models and datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
