反射偏好优化（RPO）：通过提示引导的反思增强策略对齐
Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflection
AI总结：
RPO通过提示引导的反思增强策略对齐，减少幻觉并提升多模态基准性能。
摘要：
直接偏好优化（DPO）已经作为一种轻量且有效的替代方法出现，用于对齐大语言和视觉-语言模型，替代强化学习从人类反馈（RLHF）和强化学习与AI反馈（RLAIF）。然而，标准的DPO公式，其中选择和拒绝的响应由同一策略生成，由于两个响应经常有相似的错误并表现出小的Kullback-Leibler（KL）散度，导致学习信号较弱。这导致了缓慢且不稳定的收敛。为了解决这一限制，我们引入了反射偏好优化（RPO），一种新的框架，将提示引导的反思纳入DPO范式中。RPO使用外部模型来识别幻觉来源并生成简洁的反思提示，使能够构建具有更强对比度和更清晰偏好信号的策略对。我们理论上表明，通过提示增加预期偏好边际通过互信息，并提高样本效率，同时保持在策略分布族内。经验上，RPO在更少的训练样本和迭代中实现了更优的对齐，显著降低了幻觉率，并在多模态基准上实现了最先进的性能。
英文摘要：
Direct Preference Optimization (DPO) has emerged as a lightweight and effective alternative to Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with AI Feedback (RLAIF) for aligning large language and vision-language models. However, the standard DPO formulation, in which both the chosen and rejected responses are generated by the same policy, suffers from a weak learning signal because the two responses often share similar errors and exhibit small Kullback-Leibler (KL) divergence. This leads to slow and unstable convergence. To address this limitation, we introduce Reflective Preference Optimization (RPO), a new framework that incorporates hint-guided reflection into the DPO paradigm. RPO uses external models to identify hallucination sources and generate concise reflective hints, enabling the construction of on-policy preference pairs with stronger contrastiveness and clearer preference signals. We theoretically show that conditioning on hints increases the expected preference margin through mutual information and improves sample efficiency while remaining within the policy distribution family. Empirically, RPO achieves superior alignment with fewer training samples and iterations, substantially reducing hallucination rates and delivering state-of-the-art performance across multimodal benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
