Woodpecker：多模态大语言模型的幻觉校正方法
Woodpecker: Hallucination Correction for Multimodal Large Language Models
AI总结：
针对多模态大语言模型的幻觉问题，提出免训练的事后校正方法Woodpecker，通过五阶段流程校正生成文本中的幻觉，适配多种模型且具备可解释性，在POPE基准上准确率较基线最高提升30.66%。
摘要：
幻觉是笼罩在快速发展的多模态大语言模型（MLLMs）之上的一大阴影，指生成文本与图像内容不一致的现象。为缓解幻觉问题，现有研究主要采用指令微调方式，需用特定数据重新训练模型。本文提出了一条不同的路径，引入一种名为Woodpecker的免训练方法。如同啄木鸟为树木治病一样，该方法会从生成文本中找出并校正幻觉内容。具体而言，Woodpecker包含五个阶段：关键概念提取、问题构建、视觉知识验证、视觉主张生成以及幻觉校正。Woodpecker以事后补救的方式实现，可轻松适配不同的MLLMs，同时可通过访问五个阶段的中间输出实现可解释性。我们从定量和定性两方面对Woodpecker进行了评估，展现了这一新范式的巨大潜力。在POPE基准测试上，我们的方法相较于基线模型MiniGPT-4/mPLUG-Owl，准确率分别提升了30.66%和24.33%。源代码已发布于https://github.com/BradyFU/Woodpecker。
英文摘要：
Hallucination is a big shadow hanging over the rapidly evolving Multimodal Large Language Models (MLLMs), referring to the phenomenon that the generated text is inconsistent with the image content. In order to mitigate hallucinations, existing studies mainly resort to an instruction-tuning manner that requires retraining the models with specific data. In this paper, we pave a different way, introducing a training-free method named Woodpecker. Like a woodpecker heals trees, it picks out and corrects hallucinations from the generated text. Concretely, Woodpecker consists of five stages: key concept extraction, question formulation, visual knowledge validation, visual claim generation, and hallucination correction. Implemented in a post-remedy manner, Woodpecker can easily serve different MLLMs, while being interpretable by accessing intermediate outputs of the five stages. We evaluate Woodpecker both quantitatively and qualitatively and show the huge potential of this new paradigm. On the POPE benchmark, our method obtains a 30.66%/24.33% improvement in accuracy over the baseline MiniGPT-4/mPLUG-Owl. The source code is released at https://github.com/BradyFU/Woodpecker.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
