通过交叉曲率预测块坐标性能

中文摘要 AI 辅助

同步和顺序块更新是机器学习中使用的两种基本优化策略，例如神经网络训练、联邦学习和低秩自适应。在它们之间进行选择是困难的，因为它们的相对优势取决于目标几何和迭代次数。我们开发了一个统一理论，用于比较雅可比（JC）、高斯-赛德尔（GS）和部分顺序确定性块梯度更新。我们的分析通过交叉块曲率表达了一步损失差异，带有$O(\eta^3)$的余项，其中$\eta$是学习率。我们在正则性条件和$\eta K \le T$（对于固定$T$）下，推导出$K$次迭代后的有符号损失比较，误差为$O(K\eta^3)$，当预测差异超过此误差时，识别出更好的方法。我们在不同机器学习设置中沿观察到的训练轨迹评估这些公式。在500次迭代中，我们的理论在神经网络中正确识别出较低损失方法的迭代占98.0%，联邦学习占83.4%，LoRA占97.6%。使用测量的参数差异在每一步应用损失递归，将这些比率分别提高到100.0%、93.2%和99.6%。

英文摘要

Simultaneous and sequential block updates are two basic optimization strategies used across machine learning, such as neural-network training, federated learning, and low-rank adaptation. Choosing between them is difficult because their relative advantage depends on both the objective geometry and the number of iterations. We develop a unified theory for comparing Jacobi (JC), Gauss--Seidel (GS), and partially sequential deterministic block-gradient updates. Our analysis expresses the one-step loss difference through cross-block curvature, with an $O(η^3)$ remainder, where $η$ is the learning rate. We derive a signed loss comparison after $K$ iterations with $O(Kη^3)$ error under regularity conditions and $ηK\le T$ for fixed $T$, identifying the better method when the predicted difference exceeds this error. We evaluate these formulas along observed training trajectories across different machine learning settings. Over 500 iterations, our theory correctly identifies the lower-loss method in 98.0\% of iterations for the neural network, 83.4\% for federated learning, and 97.6\% for LoRA. Applying the loss recursion at each step using the measured parameter difference raises these rates to 100.0\%, 93.2\%, and 99.6\%, respectively.

发表机构

• The Hong Kong Polytechnic University（香港理工大学）
• Xi’an Jiaotong University（西安交通大学）
• Beihang University（北京航空航天大学）

机构由 AI 辅助整理，请以论文原文为准。