Diffusion-RPO：通过相对偏好优化对齐扩散模型
Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimization
AI总结：
本文提出Diffusion-RPO方法，通过相对偏好优化并利用同提示及语义相关的跨模态提示-图像对来对齐扩散模型与人类偏好，同时引入风格对齐评估指标，在Stable Diffusion 1.5和XL-1.0上优于监督微调和Diffusion-DPO。
摘要：
将大语言模型与人类偏好对齐已成为语言建模研究中的关键焦点。然而，将偏好学习整合到文本到图像（T2I）生成模型中仍相对处于未探索领域。Diffusion-DPO技术通过针对特定文本提示的扩散模型采用成对偏好学习迈出了初步步伐。我们提出Diffusion-RPO，这是一种旨在更有效地将基于扩散的T2I模型与人类偏好对齐的新方法。该方法同时利用具有相同提示的提示-图像对以及跨多种模态具有语义相关内容的提示-图像对。此外，我们开发了一种新的评估指标——风格对齐（style alignment），旨在克服当前人类偏好对齐评估中普遍存在的高成本、低可复现性和有限可解释性等挑战。我们的研究结果表明，在微调Stable Diffusion 1.5和XL-1.0版本时，Diffusion-RPO优于监督微调（Supervised Fine-Tuning）和Diffusion-DPO等既有方法，在人类偏好自动评估和风格对齐方面均取得了更优的结果。我们的代码可在https://github.com/yigu1008/Diffusion-RPO获取。
英文摘要：
Aligning large language models with human preferences has emerged as a critical focus in language modeling research. Yet, integrating preference learning into Text-to-Image (T2I) generative models is still relatively uncharted territory. The Diffusion-DPO technique made initial strides by employing pairwise preference learning in diffusion models tailored for specific text prompts. We introduce Diffusion-RPO, a new method designed to align diffusion-based T2I models with human preferences more effectively. This approach leverages both prompt-image pairs with identical prompts and those with semantically related content across various modalities. Furthermore, we have developed a new evaluation metric, style alignment, aimed at overcoming the challenges of high costs, low reproducibility, and limited interpretability prevalent in current evaluations of human preference alignment. Our findings demonstrate that Diffusion-RPO outperforms established methods such as Supervised Fine-Tuning and Diffusion-DPO in tuning Stable Diffusion versions 1.5 and XL-1.0, achieving superior results in both automated evaluations of human preferences and style alignment. Our code is available at https://github.com/yigu1008/Diffusion-RPO
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
