通过对抗负挖掘实现大型多模态模型的模态平衡偏好优化
Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Mining
AI总结：
提出模态平衡偏好优化（MBPO）框架，通过对抗负挖掘和在线验证奖励结合GRPO，解决大型多模态模型的模态不平衡问题，提升视觉-语言任务性能并减少幻觉。
摘要：
大型多模态模型（LMMs）的任务适应和对齐已通过指令微调得到显著推进，并进一步通过最近的偏好优化得到加强。然而，大多数LMMs在推理过程中仍遭受严重的模态不平衡问题，即语言先验偏差压倒视觉输入，这阻碍了它们对下游任务的泛化并导致幻觉。现有针对LMMs的偏好优化方法在构建训练数据时并未关注抑制其大型语言模型（LLM）后端的内部偏差。此外，它们严重依赖离线数据，缺乏探索适应训练中动态分布变化的各种响应的能力。同时，组相对策略优化（GRPO）——一种利用在线生成数据和验证奖励来提升推理能力的最新方法——在LMM对齐中仍未得到充分探索。在本文中，我们提出了一种新颖的偏好学习框架——模态平衡偏好优化（MBPO），以解决LMMs中的模态不平衡问题。MBPO通过对抗性扰动输入图像来生成硬负样本，即因视觉信息使用有限而被LLM偏差误导的拒绝响应，从而构建更有效的离线偏好数据集。此外，MBPO利用封闭式任务易于验证的特性，生成带有验证奖励的在线响应。然后采用GRPO使用离线-在线混合数据训练模型。大量实验表明，MBPO能够提升LMMs在具有挑战性的视觉-语言任务上的性能，并有效减少幻觉。
英文摘要：
The task adaptation and alignment of Large Multimodal Models (LMMs) have been significantly advanced by instruction tuning and further strengthened by recent preference optimization. Yet, most LMMs still suffer from severe modality imbalance during reasoning, i.e., outweighing language prior biases over visual inputs, which bottlenecks their generalization to downstream tasks and causes hallucinations. However, existing preference optimization approaches for LMMs do not focus on restraining the internal biases of their Large Language Model (LLM) backbones when curating the training data. Moreover, they heavily rely on offline data and lack the capacity to explore diverse responses adaptive to dynamic distributional shifts during training. Meanwhile, Group Relative Policy Optimization (GRPO), a recent method using online-generated data and verified rewards to improve reasoning capabilities, remains largely underexplored in LMM alignment. In this paper, we propose a novel preference learning framework, Modality-Balancing Preference Optimization (MBPO), to address the modality imbalance in LMMs. MBPO constructs a more effective offline preference dataset by generating hard negatives, i.e., rejected responses misled by LLM biases due to limited usage of visual information, through adversarial perturbation of input images. Moreover, MBPO leverages the easy-to-verify nature of close-ended tasks to generate online responses with verified rewards. GRPO is then employed to train the model with offline-online hybrid data. Extensive experiments demonstrate that MBPO can enhance LMM performance on challenging vision-language tasks and effectively reduce hallucinations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
