缓解多模态大语言模型中的对象幻觉：基于数据增强的短语级对齐
Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment
AI总结：
本文提出数据增强的短语级对齐（DPA）损失，通过生成幻觉与正确响应对并微调现成MLLMs，有效缓解对象幻觉，同时保持通用性能，在幻觉VQA和图像描述任务上分别提升F1达13.4%和降低幻觉率4.2%。
摘要：
尽管多模态大语言模型（MLLMs）取得了显著进展，但它们常常生成事实不准确的信息，即所谓的幻觉。在本工作中，我们解决MLLMs中的对象幻觉问题，即生成关于输入图像中不存在的对象的信息。我们引入了数据增强的短语级对齐（DPA），这是一种新颖的损失函数，可应用于经过指令微调的现成MLLMs，以缓解幻觉，同时保持其通用的视觉-语言能力。为了使用DPA对MLLMs进行微调，我们首先通过生成式数据增强，在短语级别选择性地修改正确响应的地面真值信息，生成一组“幻觉”和“正确”的响应对。然后使用DPA损失训练MLLMs，以降低幻觉短语相对于正确短语的可能性。我们在各种基准上的全面评估证实了DPA在缓解幻觉方面的有效性，同时保留了MLLMs在通用任务上的开箱即用性能。例如，使用DPA微调的MLLMs，我们称之为幻觉衰减语言与视觉助手（HALVA），在幻觉视觉问答上将F1分数提高了高达13.4%，在图像描述任务上将幻觉率降低了高达4.2%。
英文摘要：
Despite their significant advancements, Multimodal Large Language Models (MLLMs) often generate factually inaccurate information, referred to as hallucination. In this work, we address object hallucinations in MLLMs, where information is generated about an object not present in the input image. We introduce Data-augmented Phrase-level Alignment (DPA), a novel loss which can be applied to instruction-tuned off-the-shelf MLLMs to mitigate hallucinations, while preserving their general vision-language capabilities. To fine-tune MLLMs with DPA, we first generate a set of `hallucinated' and `correct' response pairs through generative data augmentation by selectively altering the ground-truth information of the correct responses at a phrase level. The DPA loss is then used to train MLLMs to reduce the likelihood of hallucinated phrases compared to the correct ones. Our thorough evaluation on various benchmarks confirms the effectiveness of DPA in mitigating hallucination while retaining the out-of-the-box performance of the MLLMs on general tasks. For instance, MLLMs finetuned with DPA, which we refer to as Hallucination Attenuated Language and Vision Assistant (HALVA), improve F1 by up to 13.4% on hallucination visual question-answering and reduce the hallucination rate by up to 4.2% on image description tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
