arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.08714cs.LGcs.AI
收藏

PRDP:面向扩散模型大规模奖励微调的近端奖励差异预测

PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models

Google(谷歌公司);Rutgers University(罗格斯大学);Accenture(埃森哲公司)

Fei Deng, Qifei Wang, Wei Wei, Matthias Grundmann, Tingbo Hou

AI总结:

本文提出PRDP方法,通过奖励差异预测目标替代RL目标,首次在超100K提示的大规模数据集上实现扩散模型的稳定黑盒奖励微调,并在复杂未见提示上取得优于RL方法的生成质量。

摘要:

奖励微调已成为将基础模型与下游目标对齐的一种有前景的方法。在语言领域,通过使用强化学习(RL)最大化反映人类偏好的奖励,已经取得了显著成功。然而,在视觉领域,现有基于RL的奖励微调方法受限于其在大规模训练中的不稳定性,导致它们无法泛化到复杂的、未见过的提示。在本文中,我们提出了近端奖励差异预测(PRDP),首次在超过100K提示的大规模提示数据集上实现了扩散模型的稳定黑盒奖励微调。我们的关键创新是奖励差异预测(RDP)目标,该目标与RL目标具有相同的最优解,同时具有更好的训练稳定性。具体而言,RDP目标是一个监督回归目标,要求扩散模型从其去噪轨迹中预测生成图像对的奖励差异。我们从理论上证明,获得完美奖励差异预测的扩散模型恰好是RL目标的最大化者。我们进一步开发了一种具有近端更新的在线算法,以稳定地优化RDP目标。在实验中,我们证明PRDP在小规模训练中能够匹配成熟的基于RL方法的奖励最大化能力。此外,通过在Human Preference Dataset v2和Pick-a-Pic v1数据集中的文本提示上进行大规模训练,PRDP在一组多样化的复杂未见提示上取得了优越的生成质量,而基于RL的方法则完全失败。

英文摘要:

Reward finetuning has emerged as a promising approach to aligning foundation models with downstream objectives. Remarkable success has been achieved in the language domain by using reinforcement learning (RL) to maximize rewards that reflect human preference. However, in the vision domain, existing RL-based reward finetuning methods are limited by their instability in large-scale training, rendering them incapable of generalizing to complex, unseen prompts. In this paper, we propose Proximal Reward Difference Prediction (PRDP), enabling stable black-box reward finetuning for diffusion models for the first time on large-scale prompt datasets with over 100K prompts. Our key innovation is the Reward Difference Prediction (RDP) objective that has the same optimal solution as the RL objective while enjoying better training stability. Specifically, the RDP objective is a supervised regression objective that tasks the diffusion model with predicting the reward difference of generated image pairs from their denoising trajectories. We theoretically prove that the diffusion model that obtains perfect reward difference prediction is exactly the maximizer of the RL objective. We further develop an online algorithm with proximal updates to stably optimize the RDP objective. In experiments, we demonstrate that PRDP can match the reward maximization ability of well-established RL-based methods in small-scale training. Furthermore, through large-scale training on text prompts from the Human Preference Dataset v2 and the Pick-a-Pic v1 dataset, PRDP achieves superior generation quality on a diverse set of complex, unseen prompts whereas RL-based methods completely fail.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑