arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.06162stat.MLcs.LG
收藏

Wasserstein 近端算子描述基于分数的生成模型并解决记忆问题

Wasserstein proximal operators describe score-based generative models and resolve memorization

University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of South Carolina, Columbia(南卡罗来纳大学哥伦比亚分校)

Benjamin J. Zhang, Siting Liu, Wuchen Li, Markos A. Katsoulakis, Stanley J. Osher

AI总结:

本文通过Wasserstein近端算子和平均场博弈揭示基于分数的生成模型的数学结构,提出可解释的核模型以提升训练效率并解决记忆效应,同时为流形学习和泛化提供新解释。

摘要:

我们关注基于分数的生成模型(SGMs)的基本数学结构。我们首先用Wasserstein近端算子(WPO)来表述SGMs,并通过平均场博弈(MFGs)证明,WPO表述揭示了描述扩散和基于分数模型的归纳偏置的数学结构。特别地,MFGs以一对耦合偏微分方程的形式给出最优性条件:一个前向受控的Fokker-Planck(FP)方程和一个后向的Hamilton-Jacobi-Bellman(HJB)方程。通过Cole-Hopf变换,并利用交叉熵可以与密度的线性泛函相关联这一事实,我们证明HJB方程是一个非受控的FP方程。其次,有了手头的数学结构,我们提出一个可解释的基于核的分数函数模型,该模型在训练样本和训练时间方面显著提高了SGMs的性能。此外,WPO信息核模型被明确构建以避免最近研究的基于分数的生成模型的记忆效应。新基于核模型的数学形式与MFG终端条件的使用相结合,揭示了SGMs的流形学习和泛化性质的新解释,并为其记忆效应提供了解决方案。最后,我们数学上信息丰富、可解释的基于核模型为高维应用提出了新的可扩展定制神经网络架构。

英文摘要:

We focus on the fundamental mathematical structure of score-based generative models (SGMs). We first formulate SGMs in terms of the Wasserstein proximal operator (WPO) and demonstrate that, via mean-field games (MFGs), the WPO formulation reveals mathematical structure that describes the inductive bias of diffusion and score-based models. In particular, MFGs yield optimality conditions in the form of a pair of coupled partial differential equations: a forward-controlled Fokker-Planck (FP) equation, and a backward Hamilton-Jacobi-Bellman (HJB) equation. Via a Cole-Hopf transformation and taking advantage of the fact that the cross-entropy can be related to a linear functional of the density, we show that the HJB equation is an uncontrolled FP equation. Second, with the mathematical structure at hand, we present an interpretable kernel-based model for the score function which dramatically improves the performance of SGMs in terms of training samples and training time. In addition, the WPO-informed kernel model is explicitly constructed to avoid the recently studied memorization effects of score-based generative models. The mathematical form of the new kernel-based models in combination with the use of the terminal condition of the MFG reveals new explanations for the manifold learning and generalization properties of SGMs, and provides a resolution to their memorization effects. Finally, our mathematically informed, interpretable kernel-based model suggests new scalable bespoke neural network architectures for high-dimensional applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑