分布式与联邦学习的改进信息论泛化界
Improved Information Theoretic Generalization Bounds for Distributed and Federated Learning
AI总结：
本文为分布式与联邦学习提出改进的信息论泛化界，在 Bregman 散度或 Lipschitz 连续损失下给出逐节点互信息上界，展现对节点数 K 的 1/K 改进依赖。
摘要：
我们考虑网络化环境下统计学习问题期望泛化误差的信息论界。在该环境中，存在 K 个节点，每个节点拥有各自独立的数据集，各节点的模型必须聚合为最终的集中式模型。我们既考虑模型的简单平均，也考虑更复杂的多轮算法。我们针对多种问题给出期望泛化误差的上界，例如具有 Bregman 散度或 Lipschitz 连续损失的问题，这些上界展示了对节点数 K 的改进依赖关系，即 1/K 的依赖。这些“逐节点”界以每个节点上训练数据集与训练后权重之间的互信息来表示，因此有助于刻画每个节点存在通信或隐私约束时所固有的泛化性质。
英文摘要：
We consider information-theoretic bounds on expected generalization error for statistical learning problems in a networked setting. In this setting, there are $K$ nodes, each with its own independent dataset, and the models from each node have to be aggregated into a final centralized model. We consider both simple averaging of the models as well as more complicated multi-round algorithms. We give upper bounds on the expected generalization error for a variety of problems, such as those with Bregman divergence or Lipschitz continuous losses, that demonstrate an improved dependence of $1/K$ on the number of nodes. These "per node" bounds are in terms of the mutual information between the training dataset and the trained weights at each node, and are therefore useful in describing the generalization properties inherent to having communication or privacy constraints at each node.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
