通过批量更新实现SGD的高概率收敛
High-Probability Convergence of SGD via Batched Updates
- North Carolina State University（北卡罗来纳州立大学）
- University of California, San Diego（加州大学圣地亚哥分校）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本文提出批量SGD变体，通过分时期低方差梯度更新，在标准假设下为强凸和非凸目标建立近乎最优的高概率收敛速率，并首次为联邦学习提供高概率保证，实现对数通信复杂度和线性加速。
AI中文摘要：
随机梯度下降（SGD）是大规模优化的主要工具。虽然其迭代的平均行为（通常以均方误差界表征）已被充分理解，但为最后一次迭代获得高概率保证仍然具有挑战性。先前解决此问题的方法要么施加了限制性假设（如有界域或有界梯度），要么依赖于涉及辅助序列的复杂证明。在这项工作中，我们提出了批量SGD（Batched SGD），这是一种简单的变体，它将在线样本划分为多个时期，并使用精细化的低方差梯度估计在每个时期执行一次更新。我们的主要贡献在于证明了这种批量机制能够实现一种令人惊讶的简单高概率分析，从而避免了限制性假设和辅助序列。在标准光滑性和范数-次高斯噪声假设下，我们为强凸和非凸目标建立了近乎最优的速率。此外，我们表明我们的批量思想自然地扩展到联邦学习（FL）。我们为FL提供了首个高概率保证，实现了对数通信复杂度、智能体数量的线性加速以及对数据异构性的鲁棒性。
英文摘要：
Stochastic gradient descent (SGD) is the primary workhorse for large-scale optimization. While the average behavior of its iterates, typically characterized by mean-squared error bounds, is well-understood, obtaining high-probability guarantees for the last iterate remains challenging. Prior approaches to this problem have either imposed restrictive assumptions (such as bounded domains or gradients) or relied on complex proofs involving auxiliary sequences. In this work, we propose Batched SGD, a simple variant that partitions online samples into epochs and performs a single update per epoch using a refined, low-variance gradient estimate. Our main contribution demonstrates that this batching mechanism enables a surprisingly simple high-probability analysis that avoids both restrictive assumptions and auxiliary sequences. Under standard smoothness and norm-sub-Gaussian noise assumptions, we establish near-optimal rates for both strongly convex and non-convex objectives. Furthermore, we show that our batching idea extends naturally to federated learning (FL). We provide the first high-probability guarantees for FL, achieving logarithmic communication complexity, linear speedup in the number of agents, and resilience to data heterogeneity.
