arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.00683stat.MEmath.OCstat.ML
收藏

投影随机自然梯度变分推断的收敛性:各种步长和样本或批量大小调度

Convergence of projected stochastic natural gradient variational inference for various step size and sample or batch size schedules

Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK(格勒诺布尔阿尔卑斯大学、Inria、法国国家科学研究中心、格勒诺布尔理工学院、LJK实验室)

Thomas Guilmeau, Hadrien Hendrikx, Florence Forbes

AI总结:

本文研究了变分分布属于指数族时投影随机自然梯度变分推断的收敛性,证明了在不同步长和样本/批量大小调度下,算法收敛到最优解的速率。

摘要:

随机自然梯度变分推断(NGVI)是一种用于贝叶斯推断的流行且高效的算法。尽管在实践中取得了成功,但该方法的收敛性仍不完全清楚。本文定义并研究了当变分分布形成指数族时的投影随机NGVI。随机性出现在梯度不可行期望或大和的情况下。我们为常数或递减步长与常数或递增样本/批量大小的组合证明了新的非渐近收敛结果。当所有超参数固定时,NGVI被证明能够几何收敛到最优解的邻域,而我们为其他步长和样本/批量大小调度组合建立了收敛到最优解的速率形式为O(1/T^ρ),其中ρ≥1。这些速率适用于目标后验分布在某种意义上接近所考虑的指数族。我们的理论结果扩展了现有的NGVI和随机优化结果,并提供了更灵活的方法来调整步长和样本/批量大小,以满足速度、资源或准确性约束。

英文摘要:

Stochastic natural gradient variational inference (NGVI) is a popular and efficient algorithm for Bayesian inference. Despite empirical success, the convergence of this method is still not fully understood. In this work, we define and study a projected stochastic NGVI when variational distributions form an exponential family. Stochasticity arises when either gradients are intractable expectations or large sums. We prove new non-asymptotic convergence results for combinations of constant or decreasing step sizes and constant or increasing sample/batch sizes. When all hyperparameters are fixed, NGVI is shown to converge geometrically to a neighborhood of the optimum, while we establish convergence to the optimum with rates of the form $\mathcal{O}\left(\frac{1}{T^ρ} \right)$, possibly with $ρ\geq 1$, for all other combinations of step size and sample/batch size schedules. These rates apply when the target posterior distribution is close in some sense to the considered exponential family. Our theoretical results extend existing NGVI and stochastic optimization results and provide more flexibility to adjust, in a principled way, step sizes and sample/batch sizes in order to meet speed, resources, or accuracy constraints.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑