arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.01804cs.LGcs.AI
收藏

从扩散模型中提取奖励函数

Extracting Reward Functions from Diffusion Models

University of Oxford(牛津大学)

Felipe Nuti, Tim Franzmeyer, João F. Henriques

AI总结:

本文提出通过比较低奖励与高奖励行为扩散模型提取相对奖励函数,给出存在唯一性条件及梯度对齐学习算法,并在导航、运动和图像生成任务中验证其有效性。

摘要:

扩散模型在图像生成方面取得了显著成果,同样也被用于在序贯决策任务中学习高性能策略。决策扩散模型可以在较低质量的数据上训练,然后通过奖励函数进行引导,以生成接近最优的轨迹。我们研究通过比较一个建模低奖励行为的决策扩散模型和一个建模高奖励行为的决策扩散模型来提取奖励函数的问题;这一设定与逆强化学习相关。我们首先定义了两个扩散模型的相对奖励函数概念,并给出了其存在且唯一的条件。随后,我们设计了一种实用的学习算法来提取该函数:将由神经网络参数化的奖励函数的梯度与两个扩散模型输出之间的差异对齐。我们的方法在导航环境中找到了正确的奖励函数,并且证明了用学习到的奖励函数引导基础模型,可在标准运动基准上显著提升性能。最后,我们通过从两个大规模图像生成扩散模型中学习类奖励函数,证明该方法可推广到序贯决策之外;提取出的奖励函数能够成功为有害图像赋予较低奖励。

英文摘要:

Diffusion models have achieved remarkable results in image generation, and have similarly been used to learn high-performing policies in sequential decision-making tasks. Decision-making diffusion models can be trained on lower-quality data, and then be steered with a reward function to generate near-optimal trajectories. We consider the problem of extracting a reward function by comparing a decision-making diffusion model that models low-reward behavior and one that models high-reward behavior; a setting related to inverse reinforcement learning. We first define the notion of a relative reward function of two diffusion models and show conditions under which it exists and is unique. We then devise a practical learning algorithm for extracting it by aligning the gradients of a reward function -- parametrized by a neural network -- to the difference in outputs of both diffusion models. Our method finds correct reward functions in navigation environments, and we demonstrate that steering the base model with the learned reward functions results in significantly increased performance in standard locomotion benchmarks. Finally, we demonstrate that our approach generalizes beyond sequential decision-making by learning a reward-like function from two large-scale image generation diffusion models. The extracted reward function successfully assigns lower rewards to harmful images.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑