arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.01732cs.LGmath.OC
收藏

超越架设:一种统一的时空梯度跟踪方法

Beyond Scaffold: A Unified Spatio-Temporal Gradient Tracking Method

Division of Decision and Control Systems, School of EECS, KTH Royal Institute of Technology(决策与控制系统系,皇家理工学院);College of Control Science and Engineering, Zhejiang University(控制科学与工程学院,浙江大学)

Yan Huang, Jinming Xu, Jiming Chen, Karl Henrik Johansson

AI总结:

ST-GT通过统一时空梯度跟踪方法,有效缓解数据异质性和局部梯度噪声,实现强凸问题的线性收敛和非凸问题的次线性收敛,提升通信效率。

摘要:

在分布式和联邦学习算法中,通过在通信轮次之间执行多次本地更新通常可以减少通信开销。然而,由于节点间的数据异质性和每个节点内部的局部梯度噪声,这种策略可能导致局部模型偏离全局最优。为了解决这个问题,我们从梯度跟踪的角度重新审视已知的联邦学习方法Scaffold(Karimireddy等人,2020),并提出一种统一的时空梯度跟踪算法,称为ST-GT,用于时间变化图上的分布式随机优化。ST-GT通过跟踪相邻节点的全局梯度来缓解数据异质性,同时维护局部梯度的运行平均值以显著抑制噪声,尽管存储开销略有增加。不假设数据异质性有界,我们证明ST-GT在强凸问题中达到线性收敛速度,在非凸情况下达到次线性速度。值得注意的是,ST-GT在强凸情况下实现了通信复杂度关于每个轮次本地更新次数τ的首次线性加速。与传统梯度跟踪方法相比,ST-GT将拓扑依赖的噪声项从σ²降至σ²/τ,其中σ²表示噪声水平,从而提高了通信效率。

英文摘要:

In distributed and federated learning algorithms, communication overhead is often reduced by performing multiple local updates between communication rounds. However, due to data heterogeneity across nodes and the local gradient noise within each node, this strategy can lead to the drift of local models away from the global optimum. To address this issue, we revisit the well-known federated learning method Scaffold (Karimireddy et al., 2020) under a gradient tracking perspective, and propose a unified spatio-temporal gradient tracking algorithm, termed ST-GT, for distributed stochastic optimization over time-varying graphs. ST-GT tracks the global gradient across neighboring nodes to mitigate data heterogeneity, while maintaining a running average of local gradients to substantially suppress noise, with slightly more storage overhead. Without assuming bounded data heterogeneity, we prove that ST-GT attains a linear convergence rate for strongly convex problems and a sublinear rate for nonconvex cases. Notably, ST-GT achieves the first linear speed-up in communication complexity with respect to the number of local updates per round $τ$ for the strongly-convex setting. Compared to traditional gradient tracking methods, ST-GT reduces the topology-dependent noise term from $σ^2$ to $σ^2/τ$, where $σ^2$ denotes the noise level, thereby improving communication efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑