arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.11249cs.LGmath.STstat.MLstat.TH
收藏

关于证据下界的自然梯度

On the Natural Gradient of the Evidence Lower Bound

Hamburg University of Technology(汉堡工业大学);Santa Fe Institute(圣菲研究所);Leipzig University(莱比锡大学)

Nihat Ay, Jesse van Oostrum, Adwait Datar

AI总结:

本文研究证据下界(ELBO)的自然梯度,发现无约束优化下证据与ELBO间隙的自然梯度本质为零,ELBO最大化等价于最小化与目标分布的KL散度,还推导了约束在模型内时该等价性成立的条件并通过柱形模型形式化其几何刻画。

摘要:

本文研究了在生成式机器学习中占据核心地位的证据下界(ELBO)的Fisher-Rao梯度,也被称为自然梯度。研究揭示,在无约束优化场景下,证据与其下界ELBO之间的间隙的自然梯度本质上为零。因此,最大化ELBO等价于最小化与目标分布的Kullback-Leibler散度,而这正是学习的主要目标函数。基于这一洞见,我们推导得出一个条件,即便优化被约束在某一模型内,该等价关系依然成立。这一条件给出了一种几何刻画,我们通过柱形模型(cylindrical model)的概念将其形式化。

英文摘要:

This article studies the Fisher-Rao gradient, also referred to as the natural gradient, of the evidence lower bound (ELBO) which plays a central role in generative machine learning. It reveals that the gap between the evidence and its lower bound, the ELBO, has essentially a vanishing natural gradient within unconstrained optimization. As a result, maximization of the ELBO is equivalent to minimization of the Kullback-Leibler divergence from a target distribution, the primary objective function of learning. Building on this insight, we derive a condition under which this equivalence persists even when optimization is constrained to a model. This condition yields a geometric characterization, which we formalize through the notion of a cylindrical model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑