超越成对偏好：扩散模型的列表级奖励感知对齐
Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models
AI总结：
提出Diffusion LAIR方法，通过列表级奖励感知优化，利用连续奖励分数和所有候选图像同时优化扩散模型，在文本到图像生成等任务上超越成对偏好基线。
摘要：
偏好优化已成为从人类反馈中进行在线强化学习（RLHF）的一种高效替代方案，用于对齐文本到图像扩散模型。然而，现有方法大多将监督简化为二元成对比较。当训练数据自然包含同一提示的多个候选图像，并且连续奖励分数能提供比单一赢家-输家标签更丰富的信息时，这种成对简化具有局限性。为解决这些局限性，我们提出了Diffusion LAIR，一种用于扩散模型的奖励感知列表级偏好优化方法。对于每个提示，LAIR将一组候选图像的奖励分数转换为居中优势权重，然后在隐式奖励上优化优势加权回归目标，隐式奖励定义为当前模型相对于固定参考模型的去噪损失改进，并带有二次惩罚以正则化隐式奖励的幅度。所得目标同时使用所有候选图像而非选择成对，并通过显式控制隐式奖励的幅度保持保守性。LAIR目标在隐式奖励空间中具有有界闭式最优解，阐明了正则化强度如何控制偏好更新的幅度。实验表明，Diffusion LAIR在SD1.5和SDXL上，在文本到图像生成、组合生成和图像编辑基准测试中均优于强偏好优化基线。
英文摘要：
Preference optimization has emerged as an efficient alternative to online reinforcement learning from human feedback (RLHF) for aligning text-to-image diffusion models. However, existing methods largely reduce supervision to binary pairwise comparisons. This pairwise reduction is limiting when training data naturally contains multiple candidate images for the same prompt, and when continuous reward scores can provide richer information than a single winner-loser label. To address these limitations, we propose Diffusion LAIR, a reward-aware listwise preference optimization method for diffusion models. For each prompt, LAIR converts reward scores across a group of candidate images into centered advantage weights, then optimizes an advantage-weighted regression objective on the implicit reward, defined as the denoising-loss improvement of the current model over a fixed reference model, with a quadratic penalty that regularizes the magnitude of the implicit reward. The resulting objective uses all candidates simultaneously rather than selecting pairs, and remains conservative by explicitly controlling the magnitude of the implicit reward. The LAIR objective admits a bounded closed-form optimum in implicit-reward space, clarifying how the regularization strength controls the magnitude of the preference update. Experiments show that Diffusion LAIR outperforms strong preference optimization baselines on SD1.5 and SDXL across text-to-image generation, compositional generation, and image editing benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
