RPO：通过丰富的视觉-语言偏好进行视觉生成模型微调
RPO: Fine-Tuning Visual Generative Models via Rich Vision-Language Preferences
AI总结：
RPO通过利用视觉语言模型的丰富反馈信号，改进偏好对的编纂，从而提升视觉生成模型的微调效果。
摘要：
传统上，针对大语言模型/视觉生成模型的偏好微调方法往往仅依赖奖励模型的标注，这可能导致不透明，难以解释偏好背后的原因，并容易出现奖励黑客或过拟合等问题。我们引入了丰富的偏好优化（RPO），这是一种新的流程，利用来自视觉语言模型（VLMs）的丰富反馈信号来改进用于微调视觉生成模型（如文本到图像扩散模型）的偏好对的编纂。我们的方法首先通过提示VLMs生成合成图像的详细批评，进而提示VLMs提取可靠且可操作的图像编辑指令。通过实施这些指令，我们创建了精细化的图像，从而生成合成且信息丰富的偏好对，作为增强的微调数据集。我们展示了我们流程和所生成数据集在微调最新扩散模型中的有效性。
英文摘要：
Traditional preference tuning methods for LLMs/Visual Generative Models often rely solely on reward model labeling, which can be opaque, offer limited insights into the rationale behind preferences, and are prone to issues such as reward hacking or overfitting. We introduce Rich Preference Optimization (RPO), a novel pipeline that leverages rich feedback signals from Vision Language Models (VLMs) to improve the curation of preference pairs for fine-tuning visual generative models like text-to-image diffusion models. Our approach begins with prompting VLMs to generate detailed critiques of synthesized images, from which we further prompt VLMs to extract reliable and actionable image editing instructions. By implementing these instructions, we create refined images, resulting in synthetic, informative preference pairs that serve as enhanced tuning datasets. We demonstrate the effectiveness of our pipeline and the resulting datasets in fine-tuning state-of-the-art diffusion models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
