SGB：优化配分函数的随机梯度界方法
SGB: Stochastic Gradient Bound Method for Optimizing Partition Functions
摘要：
本文讨论了随机学习环境中优化配分函数的问题。我们提出了一个随机变量的约束优化算法，依赖于上界的分区函数与二次代理。所提出的方法的更新，我们称之为随机配分函数界（SPFB），类似于缩放的随机梯度下降，其中缩放因子依赖于二阶项，然而不同于Hessian。类似于拟牛顿格式，这一项是使用函数值及其梯度的随机近似来构造的。我们证明了所提出的方法的次线性收敛速度，并显示其低秩变体（LSPFB）的建设。逻辑回归实验表明，所提出的计划显着优于SGD。我们还讨论了如何使用二次配分函数界来有效训练深度学习模型和非凸优化。
英文摘要：
This paper addresses the problem of optimizing partition functions in a stochastic learning setting. We propose a stochastic variant of the bound majorization algorithm that relies on upper-bounding the partition function with a quadratic surrogate. The update of the proposed method, that we refer to as Stochastic Partition Function Bound (SPFB), resembles scaled stochastic gradient descent where the scaling factor relies on a second order term that is however different from the Hessian. Similarly to quasi-Newton schemes, this term is constructed using the stochastic approximation of the value of the function and its gradient. We prove sub-linear convergence rate of the proposed method and show the construction of its low-rank variant (LSPFB). Experiments on logistic regression demonstrate that the proposed schemes significantly outperform SGD. We also discuss how to use quadratic partition function bound for efficient training of deep learning models and in non-convex optimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
