利用自举偏好优化增强多模态大语言模型
Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
AI总结：
针对多模态大语言模型偏重预训练语料、忽视视觉信息的问题，提出BPO，通过失真图像和文本LLM构造自举负面回答进行偏好优化，显著提升多基准性能。
摘要：
多模态大语言模型（MLLMs）擅长基于视觉输入生成回答。然而，它们往往存在一种偏差，即倾向于生成与其预训练语料库相似的回答，从而掩盖了视觉信息的重要性。我们将这种偏差视为对预训练统计数据的“偏好”，它会阻碍模型基于视觉输入进行 grounding。为缓解该问题，我们提出了 Bootstrapped Preference Optimization（BPO，自举偏好优化），该方法利用包含从模型自身自举得到的负面回答的数据集进行偏好学习。具体而言，我们提出以下两种策略：1）向 MLLM 使用失真的图像输入，以诱发包含显著预训练偏差的回答；2）利用基于文本的 LLM，将错误但常见的元素显式注入原始回答中。这些不理想的回答与数据集中原始标注回答配对，以构建偏好数据集，随后用于执行偏好学习。我们的方法有效抑制了预训练 LLM 偏差，增强了基于视觉输入的 grounding。大量实验表明，其在多个基准测试上均取得显著性能提升，推动了多模态对话系统的最先进水平。
英文摘要：
Multimodal Large Language Models (MLLMs) excel in generating responses based on visual inputs. However, they often suffer from a bias towards generating responses similar to their pretraining corpus, overshadowing the importance of visual information. We treat this bias as a "preference" for pretraining statistics, which hinders the model's grounding in visual input. To mitigate this issue, we propose Bootstrapped Preference Optimization (BPO), which conducts preference learning with datasets containing negative responses bootstrapped from the model itself. Specifically, we propose the following two strategies: 1) using distorted image inputs to the MLLM for eliciting responses that contain signified pretraining bias; 2) leveraging text-based LLM to explicitly inject erroneous but common elements into the original response. Those undesirable responses are paired with original annotated responses from the datasets to construct the preference dataset, which is subsequently utilized to perform preference learning. Our approach effectively suppresses pretrained LLM bias, enabling enhanced grounding in visual inputs. Extensive experimentation demonstrates significant performance improvements across multiple benchmarks, advancing the state-of-the-art in multimodal conversational systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
