面向扩散模型中列表式偏好的更优优化
Towards Better Optimization For Listwise Preference in Diffusion Models
AI总结：
针对扩散模型中列表式偏好优化不足的问题，提出Diffusion-LPO框架，基于Plackett-Luce模型扩展DPO目标，在多任务中优于成对DPO基线。
摘要：
基于人类反馈的强化学习（RLHF）已被证明能有效让文本生成图像（T2I）扩散模型对齐人类偏好。尽管直接偏好优化（DPO）因计算高效且无需显式奖励建模而被广泛采用，但其在扩散模型中的应用主要依赖成对偏好，列表式偏好的精准优化仍基本未被解决。实际中，人类对图像偏好的反馈常包含隐含排序信息，能比成对比较传递更精准的人类偏好。本研究提出Diffusion-LPO，这是一个简单有效的框架，用于利用列表式数据实现扩散模型的列表式偏好优化。给定一个文本描述，我们将用户反馈聚合为图像排序列表，并在Plackett-Luce模型下推导DPO目标的列表式扩展。Diffusion-LPO通过鼓励每个样本都优于所有排名更低的备选样本，来确保整个排序的一致性。我们通过实验验证了Diffusion-LPO在文本生成图像、图像编辑、个性化偏好对齐等多种任务上的有效性，其在视觉质量和偏好对齐效果上始终优于成对DPO基线。
英文摘要：
Reinforcement learning from human feedback (RLHF) has proven effectiveness for aligning text-to-image (T2I) diffusion models with human preferences. Although Direct Preference Optimization (DPO) is widely adopted for its computational efficiency and avoidance of explicit reward modeling, its applications to diffusion models have primarily relied on pairwise preferences. The precise optimization of listwise preferences remains largely unaddressed. In practice, human feedback on image preferences often contains implicit ranked information, which conveys more precise human preferences than pairwise comparisons. In this work, we propose Diffusion-LPO, a simple and effective framework for Listwise Preference Optimization in diffusion models with listwise data. Given a caption, we aggregate user feedback into a ranked list of images and derive a listwise extension of the DPO objective under the Plackett-Luce model. Diffusion-LPO enforces consistency across the entire ranking by encouraging each sample to be preferred over all of its lower-ranked alternatives. We empirically demonstrate the effectiveness of Diffusion-LPO across various tasks, including text-to-image generation, image editing, and personalized preference alignment. Diffusion-LPO consistently outperforms pairwise DPO baselines on visual quality and preference alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
