arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.00626cs.LGcs.AI
收藏

基于能量模型的扩散模型最大熵逆强化学习

Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models

Korea Institute for Advanced Study(韩国高等研究院);Seoul National University(首尔大学);University of Seoul(首尔市立大学);Hanyang University(汉阳大学);Saige Research(赛奇研究所)

Sangwoong Yoon, Himchan Hwang, Dohyun Kwon, Yung-Kyun Noh, Frank C. Park

AI总结:

本研究提出DxMI最大熵逆强化学习方法,联合训练扩散模型与EBM,搭配DxDP子例程提升效率,实现少步高质量生成,还能稳定EBM训练并提升异常检测性能。

摘要:

我们提出一种最大熵逆强化学习(IRL)方法,用于提升扩散生成模型的样本质量,尤其是在生成时间步数量较少的场景下。与IRL基于从专家演示中学习到的奖励函数训练策略的思路类似,我们利用从训练数据估计得到的对数概率密度来训练(或微调)扩散模型。由于我们采用能量基模型(EBM,Energy-Based Model)来表示对数密度,本方法可归结为扩散模型与EBM的联合训练。我们提出的IRL公式命名为Diffusion by Maximum Entropy IRL(DxMI,基于最大熵逆强化学习的扩散方法),是一个极小极大问题,当两个模型都收敛到数据分布时达到平衡。熵最大化在DxMI中起到关键作用,既促进扩散模型的探索,又保证EBM的收敛。我们还提出了一种面向扩散模型的新型强化学习算法——Diffusion by Dynamic Programming(DxDP,基于动态规划的扩散方法),作为DxMI中的子例程。DxDP通过将原问题转化为最优控制公式,用价值函数替代时间反向传播,从而实现DxMI中扩散模型更新的高效性。我们的实证研究表明,使用DxMI微调的扩散模型仅需4步和10步就能生成高质量样本。此外,DxMI无需马尔可夫链蒙特卡洛(MCMC)即可训练EBM,稳定了EBM的训练动态并提升了异常检测性能。

英文摘要:

We present a maximum entropy inverse reinforcement learning (IRL) approach for improving the sample quality of diffusion generative models, especially when the number of generation time steps is small. Similar to how IRL trains a policy based on the reward function learned from expert demonstrations, we train (or fine-tune) a diffusion model using the log probability density estimated from training data. Since we employ an energy-based model (EBM) to represent the log density, our approach boils down to the joint training of a diffusion model and an EBM. Our IRL formulation, named Diffusion by Maximum Entropy IRL (DxMI), is a minimax problem that reaches equilibrium when both models converge to the data distribution. The entropy maximization plays a key role in DxMI, facilitating the exploration of the diffusion model and ensuring the convergence of the EBM. We also propose Diffusion by Dynamic Programming (DxDP), a novel reinforcement learning algorithm for diffusion models, as a subroutine in DxMI. DxDP makes the diffusion model update in DxMI efficient by transforming the original problem into an optimal control formulation where value functions replace back-propagation in time. Our empirical studies show that diffusion models fine-tuned using DxMI can generate high-quality samples in as few as 4 and 10 steps. Additionally, DxMI enables the training of an EBM without MCMC, stabilizing EBM training dynamics and enhancing anomaly detection performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑