通过偏好微调对齐视觉大型语言模型中的模态
Aligning Modalities in Vision Large Language Models via Preference Fine-tuning
AI总结：
本文将视觉大型语言模型（VLLMs）的幻觉问题视为模态对齐问题，提出POVID框架，利用AI模型自动生成偏好与非偏好反馈数据，通过直接偏好优化进行RLHF微调，有效减少幻觉并提升模型在标准基准上的性能。
摘要：
遵循指令的视觉大型语言模型（VLLMs）最近在各种任务上取得了显著进展。这些方法融合了强大的预训练视觉模型和大型语言模型（LLMs）。由于这些组件是单独训练的，因此需要通过对额外的图像-语言对进行联合训练来对齐所学到的表示。这一过程并不完美，可能会导致模型产生幻觉——即使核心LLM高度符合事实且视觉主干具有足够完整的表示，模型也会提供不能准确反映图像的答案。在这项工作中，我们将幻觉问题构建为一个对齐问题，并通过偏好调整来解决它。具体而言，我们提出了POVID，利用AI模型生成反馈数据。我们使用真实指令作为偏好响应，并采用两阶段方法生成非偏好数据。首先，我们提示GPT-4V将看似合理的幻觉注入正确答案中。其次，我们对图像进行扭曲以触发VLLM固有的幻觉行为。这是一种自动化方法，不依赖人工数据生成，也不需要完美的专家，这使得它易于扩展。最后，这两种生成策略都通过直接偏好优化集成到RLHF管道中。在广泛的基准测试中，我们表明我们不仅能减少幻觉，还能提升模型在标准基准上的性能，优于先前的方法。我们的数据和代码可在https://github.com/YiyangZhou/POVID获取。
英文摘要：
Instruction-following Vision Large Language Models (VLLMs) have achieved significant progress recently on a variety of tasks. These approaches merge strong pre-trained vision models and large language models (LLMs). Since these components are trained separately, the learned representations need to be aligned with joint training on additional image-language pairs. This procedure is not perfect and can cause the model to hallucinate - provide answers that do not accurately reflect the image, even when the core LLM is highly factual and the vision backbone has sufficiently complete representations. In this work, we frame the hallucination problem as an alignment issue, tackle it with preference tuning. Specifically, we propose POVID to generate feedback data with AI models. We use ground-truth instructions as the preferred response and a two-stage approach to generate dispreferred data. First, we prompt GPT-4V to inject plausible hallucinations into the correct answer. Second, we distort the image to trigger the inherent hallucination behavior of the VLLM. This is an automated approach, which does not rely on human data generation or require a perfect expert, which makes it easily scalable. Finally, both of these generation strategies are integrated into an RLHF pipeline via Direct Preference Optimization. In experiments across broad benchmarks, we show that we can not only reduce hallucinations, but improve model performance across standard benchmarks, outperforming prior approaches. Our data and code are available at https://github.com/YiyangZhou/POVID.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
