arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.20971cs.LGcs.CV
收藏

在视觉、语言与控制任务的扩散模型中分摊难解推理

Amortizing intractable inference in diffusion models for vision, language, and control

Mila(米拉研究所);Université de Montréal(蒙特利尔大学);KAIST(韩国科学技术院);Jagiellonian University(雅盖隆大学);Ciela Institute(谢拉研究所);Dreamfold(Dreamfold公司);Recursion(Recursion公司);CIFAR(加拿大高等研究院);University of Edinburgh(爱丁堡大学)
Siddarth Venkatraman, Moksh Jain, Luca Scimeca, Minsu Kim, Marcin Sendera, Mohsin Hasan, Luke Rowe, Sarthak Mittal, Pablo Lemos, Emmanuel Bengio, Alexandre Adam… 展开作者

Siddarth Venkatraman, Moksh Jain, Luca Scimeca, Minsu Kim, Marcin Sendera, Mohsin Hasan, Luke Rowe, Sarthak Mittal, Pablo Lemos, Emmanuel Bengio, Alexandre Adam, Jarrid Rector-Brooks, Yoshua Bengio, Glen Berseth, Esmeralda S. Whitammer

AI总结:

本文针对扩散模型用作下游任务先验时的难解后验推理问题,提出相对轨迹平衡方法,在视觉、语言、多模态及连续控制等任务上取得良好效果

摘要:

扩散模型已成为视觉、语言及强化学习领域中有效的分布估计器,但将其用作下游任务的先验时会产生难解的后验推理问题。本文研究了数据相关的后验分布$\boldsymbol{x}\tilde{p}^{\rm post}(\boldsymbol{x})\tilde{p}(\boldsymbol{x})r(\boldsymbol{x})$的分摊采样,该模型由扩散生成模型先验$p(\boldsymbol{x})$与黑盒约束或似然函数$r(\boldsymbol{x})$构成。我们提出并证明了一种无数据学习目标——相对轨迹平衡的渐近正确性,该目标用于训练能从该后验采样的扩散模型,而现有方法仅能近似或在受限情形下解决该问题。相对轨迹平衡源于扩散模型的生成流网络视角,该视角允许使用深度强化学习技术提升模式覆盖度。实验展示了扩散先验下任意后验的无偏推理的广泛潜力:在视觉任务(分类器引导)、语言任务(离散扩散大语言模型下的补全)、多模态数据任务(文本到图像生成)中均有应用。除生成建模外,我们还将相对轨迹平衡应用于基于得分的行为先验的连续控制问题,在离线强化学习基准上取得了最优结果。

英文摘要:

Diffusion models have emerged as effective distribution estimators in vision, language, and reinforcement learning, but their use as priors in downstream tasks poses an intractable posterior inference problem. This paper studies amortized sampling of the posterior over data, $\mathbf{x}\sim p^{\rm post}(\mathbf{x})\propto p(\mathbf{x})r(\mathbf{x})$, in a model that consists of a diffusion generative model prior $p(\mathbf{x})$ and a black-box constraint or likelihood function $r(\mathbf{x})$. We state and prove the asymptotic correctness of a data-free learning objective, relative trajectory balance, for training a diffusion model that samples from this posterior, a problem that existing methods solve only approximately or in restricted cases. Relative trajectory balance arises from the generative flow network perspective on diffusion models, which allows the use of deep reinforcement learning techniques to improve mode coverage. Experiments illustrate the broad potential of unbiased inference of arbitrary posteriors under diffusion priors: in vision (classifier guidance), language (infilling under a discrete diffusion LLM), and multimodal data (text-to-image generation). Beyond generative modeling, we apply relative trajectory balance to the problem of continuous control with a score-based behavior prior, achieving state-of-the-art results on benchmarks in offline reinforcement learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑