通过对象感知偏好优化缓解多模态大语言模型中的幻觉
Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization
AI总结：
本文提出 CHAIR-DPO，利用 CHAIR 指标从生成答案对中构造非幻觉与幻觉偏好样本，并通过 DPO 微调 MLLM，在多个基准上显著减少幻觉。
摘要：
多模态大语言模型（MLLMs）已成为处理从 NLP 到计算机视觉等众多任务的统一接口。尽管 MLLMs 在许多基准测试中展现出最先进的结果，但一个长期存在的问题是它们倾向于产生幻觉，即生成用户查询的答案，而这些答案并未在视觉输入中得到体现。本文将幻觉问题作为一个对齐问题来处理，旨在引导 MLLM 使其更偏好生成不含幻觉的内容。近期方法通常需要复杂流程来构建用于对齐训练的合成偏好数据，并且往往依赖专有模型；与之不同，我们利用了著名的 CHAIR 指标，该指标最初被提出用于衡量图像描述中的幻觉程度。给定一对生成答案，我们利用 CHAIR 区分胜者和败者选项（即非幻觉样本和幻觉样本），并通过直接偏好优化（DPO）微调现成的 MLLMs。所得方法称为 CHAIR-DPO，它在多个幻觉基准测试上有效减少了幻觉答案数量，证明了使用基于 CHAIR 的奖励微调 MLLM 的有效性。源代码和训练后的模型公开发布于 https://github.com/aimagelab/CHAIR-DPO。
英文摘要：
Multimodal Large Language Models (MLLMs) emerge as a unified interface to address a multitude of tasks, ranging from NLP to computer vision. Despite showcasing state-of-the-art results in many benchmarks, a long-standing issue is the tendency of MLLMs to hallucinate, that is to generate answers to the user's query that are not reflected in the visual input. In this paper, we address the problem of hallucinations as an alignment problem, seeking to steer the MLLM so that it prefers generating content without hallucinations. In contrast to recent approaches that require complicated pipelines to build synthetic preference data for alignment training, often relying on proprietary models, we capitalize on the well-known CHAIR metric, originally proposed to gauge the degree of hallucinations in image captioning. Given a pair of generated answers, we leverage CHAIR to distinguish winner and loser options (i.e., non-hallucinated and hallucinated samples) and fine-tune off-the-shelf MLLMs via Direct Preference Optimization (DPO). The resulting method, which we refer to as CHAIR-DPO, effectively diminishes the amount of hallucinated answers on several hallucination benchmarks, demonstrating the effectiveness of fine-tuning the MLLM with a CHAIR-based reward. Source code and trained models are publicly available at https://github.com/aimagelab/CHAIR-DPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
