arXiv 2406.118392024-10-07cs.CVcs.AIcs.CLcs.LG

mDPO：面向多模态大语言模型的条件偏好优化

mDPO: Conditional Preference Optimization for Multimodal Large Language Models

University of Southern California（南加州大学）；Microsoft Research（微软研究院）；University of California Davis（加州大学戴维斯分校）

Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, Muhao Chen

备注：: Accepted to EMNLP 2024 Main Conference. Project website: https://feiwang96.github.io/mDPO

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态偏好优化中模型忽略图像条件的无条件偏好问题，提出同时优化图像偏好、引入奖励锚点的mDPO方法，有效提升多模态大模型性能、减少幻觉。

摘要：

直接偏好优化（DPO）已被证明是大语言模型（LLM）对齐的有效方法。近期研究尝试将DPO应用于多模态场景，但发现难以实现稳定的性能提升。通过对比实验，我们发现多模态偏好优化中存在无条件偏好问题，即模型会忽略图像条件。为解决该问题，我们提出mDPO——一种多模态DPO目标函数，通过同时优化图像偏好来避免模型过度优先考虑纯语言偏好。此外，我们引入奖励锚点机制，强制选中响应的奖励为正，从而避免其生成概率下降——这是相对偏好优化的固有问题。在两个不同规模的多模态大语言模型以及三个广泛使用的基准上的实验表明，mDPO有效解决了多模态偏好优化中的无条件偏好问题，显著提升了模型性能，尤其在减少幻觉方面效果突出。

英文摘要：

Direct preference optimization (DPO) has shown to be an effective method for large language model (LLM) alignment. Recent works have attempted to apply DPO to multimodal scenarios but have found it challenging to achieve consistent improvement. Through a comparative experiment, we identify the unconditional preference problem in multimodal preference optimization, where the model overlooks the image condition. To address this problem, we propose mDPO, a multimodal DPO objective that prevents the over-prioritization of language-only preferences by also optimizing image preference. Moreover, we introduce a reward anchor that forces the reward to be positive for chosen responses, thereby avoiding the decrease in their likelihood -- an intrinsic problem of relative preference optimization. Experiments on two multimodal LLMs of different sizes and three widely used benchmarks demonstrate that mDPO effectively addresses the unconditional preference problem in multimodal preference optimization and significantly improves model performance, particularly in reducing hallucination.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。