Antidote：一个用于缓解LVLM在反事实预设与物体感知中幻觉的统一框架
Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception
AI总结：
本文提出统一框架Antidote，通过合成数据将事实先验融入问题并建模为偏好优化，以缓解LVLM在反事实预设和物体感知中的幻觉，同时构建CP-Bench基准，在LLaVA系列上显著提升多项指标且无需外部监督。
摘要：
大型视觉-语言模型（LVLMs）在各种跨模态任务中取得了令人瞩目的成果。然而，幻觉问题，即模型生成反事实响应，仍然是一个挑战。尽管近期研究试图缓解物体感知幻觉，但它们侧重于模型的响应生成，而忽视了任务问题本身。本文讨论了LVLMs在解决反事实预设问题（CPQs）时的脆弱性，即模型容易接受反事实物体的预设并产生严重的幻觉响应。为此，我们引入了“Antidote”，一个统一的、由合成数据驱动的后训练框架，用于缓解上述两类幻觉。它利用合成数据将事实先验融入问题中以实现自我校正，并将缓解过程解耦为一个偏好优化问题。此外，我们构建了“CP-Bench”，一个用于评估LVLMs正确处理CPQs并生成事实响应的新型基准。应用于LLaVA系列模型后，Antidote能够同时将CP-Bench上的性能提升超过50%，将POPE提升1.8-3.3%，并将CHAIR和SHR提升30-50%，且无需依赖来自更强LVLMs的外部监督或人类反馈，也未引入明显的灾难性遗忘问题。
英文摘要：
Large Vision-Language Models (LVLMs) have achieved impressive results across various cross-modal tasks. However, hallucinations, i.e., the models generating counterfactual responses, remain a challenge. Though recent studies have attempted to alleviate object perception hallucinations, they focus on the models' response generation, and overlooking the task question itself. This paper discusses the vulnerability of LVLMs in solving counterfactual presupposition questions (CPQs), where the models are prone to accept the presuppositions of counterfactual objects and produce severe hallucinatory responses. To this end, we introduce "Antidote", a unified, synthetic data-driven post-training framework for mitigating both types of hallucination above. It leverages synthetic data to incorporate factual priors into questions to achieve self-correction, and decouple the mitigation process into a preference optimization problem. Furthermore, we construct "CP-Bench", a novel benchmark to evaluate LVLMs' ability to correctly handle CPQs and produce factual responses. Applied to the LLaVA series, Antidote can simultaneously enhance performance on CP-Bench by over 50%, POPE by 1.8-3.3%, and CHAIR & SHR by 30-50%, all without relying on external supervision from stronger LVLMs or human feedback and introducing noticeable catastrophic forgetting issues.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
