理解模型不一致性对随机重排增量SGD收敛性的影响
Understanding the Impact of Model Incoherence on Convergence of Incremental SGD with Random Reshuffle
摘要：
虽然随机重排的SGD在机器学习应用中得到了广泛的应用，但对模型特征如何影响算法收敛性的理解有限。在本文中，我们引入模型不相干性来刻画模型特征的多样性，并研究了弱强凸性下模型不相干性对随机重排SGD收敛性的影响。具体而言，极小化不相干性度量了样本损失的全局极小化与总损失的全局极小化之间的差异，并影响随机重排的SGD的收敛误差。特别地，我们证明了随机重排SGD产生的变量序列在完全极小相干下收敛到总损失的某个全局极小。另一个曲率不相干度量了样本损失的条件数的质量，并决定了SGD的收敛速度。在模型不相干的情况下，我们的结果表明随机重排下的SGD比随机采样下的SGD具有更快的收敛速度和更小的收敛误差，从而证明了随机重排下的SGD具有更好的实际性能.
英文摘要：
Although SGD with random reshuffle has been widely-used in machine learning applications, there is a limited understanding of how model characteristics affect the convergence of the algorithm. In this work, we introduce model incoherence to characterize the diversity of model characteristics and study its impact on convergence of SGD with random reshuffle under weak strong convexity. Specifically, minimizer incoherence measures the discrepancy between the global minimizers of a sample loss and those of the total loss and affects the convergence error of SGD with random reshuffle. In particular, we show that the variable sequence generated by SGD with random reshuffle converges to a certain global minimizer of the total loss under full minimizer coherence. The other curvature incoherence measures the quality of condition numbers of the sample losses and determines the convergence rate of SGD. With model incoherence, our results show that SGD has a faster convergence rate and smaller convergence error under random reshuffle than those under random sampling, and hence provide justifications to the superior practical performance of SGD with random reshuffle.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
