We generalize the notion of average Lipschitz smoothness proposed by Ashlagi et al. (COLT 2021) by extending it to Hölder smoothness. This measure of the "effective smoothness" of a function is sensitive to the underlying distribution and can be dramatically smaller than its classic "worst-case" Hölder constant. We consider both the realizable and the agnostic (noisy) regression settings, proving upper and lower risk bounds in terms of the average Hölder smoothness; these rates improve upon both previously known rates even in the special case of average Lipschitz smoothness. Moreover, our lower bound is tight in the realizable setting up to log factors, thus we establish the minimax rate. From an algorithmic perspective, since our notion of average smoothness is defined with respect to the unknown underlying distribution, the learner does not have an explicit representation of the function class, hence is unable to execute ERM. Nevertheless, we provide distinct learning algorithms that achieve both (nearly) optimal learning rates. Our results hold in any totally bounded metric space, and are stated in terms of its intrinsic geometry. Overall, our results show that the classic worst-case notion of Hölder smoothness can be essentially replaced by its average, yielding considerably sharper guarantees.
A Convergence Theory for Federated Average: Beyond Smoothness
University of British Columbia（不列颠哥伦比亚大学）；Adobe Research（奥多比研究院）；Columbia University（哥伦比亚大学）；McGill University（麦吉尔大学）
联邦学习使得大量边缘计算设备能够学习模型，而无需联合进行数据共享。作为该设置中的领先算法，Federated Average FedAvg在本地设备上并行运行随机梯度下降（SGD）并且仅偶尔对序列进行平均，由于其简单性和低通信成本而被广泛使用。然而，尽管最近的研究工作，它缺乏理论分析的假设下超越平滑。本文分析了FedAvg的收敛性。与已有工作不同的是，我们放宽了强光滑性的假设。更具体地说，我们假设半光滑性和半Lipschitz性质的损失函数，其中有一个额外的一阶项的假设定义。此外，我们还假设梯度上有界，这比收敛性分析格式中常用的有界梯度假设要弱。作为一种解决方案，本文对联邦学习的收敛性进行了理论研究。
Adaptive Variance Reduction without Knowing Oracle Smoothness
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）
