大型视觉语言模型的统一幻觉缓解框架
A Unified Hallucination Mitigation Framework for Large Vision-Language Models
AI总结：
针对大型视觉语言模型难以处理多种查询类型幻觉的问题，提出统一框架Dentist，通过先分类查询再执行不同缓解策略，在MMbench上显著提升了准确率。
摘要：
幻觉是大型视觉语言模型（LVLMs）在长文本生成中常见的问题，难以根除。含幻觉的生成内容与图像内容部分不一致。为缓解幻觉，当前研究主要关注模型推理过程或模型生成结果，但其设计的方案有时无法妥善处理各种类型的查询以及生成内容中关于这些查询的幻觉。为精准应对各类幻觉，我们提出了一个用于缓解幻觉的统一框架Dentist。其核心步骤是先对查询进行分类，然后根据分类结果执行不同的幻觉缓解流程，就像牙医先观察牙齿再制定方案一样。在简单的部署中，Dentist可将查询分类为感知或推理，并轻松缓解答案中潜在的幻觉，这已在我们的实验中得到证明。在MMbench上，我们在Image Quality（一项粗粒度感知视觉问答（VQA）任务）上的准确率相比基线InstructBLIP/LLaVA/VisualGLM分别提升了13.44%/10.2%/15.8%。
英文摘要：
Hallucination is a common problem for Large Vision-Language Models (LVLMs) with long generations which is difficult to eradicate. The generation with hallucinations is partially inconsistent with the image content. To mitigate hallucination, current studies either focus on the process of model inference or the results of model generation, but the solutions they design sometimes do not deal appropriately with various types of queries and the hallucinations of the generations about these queries. To accurately deal with various hallucinations, we present a unified framework, Dentist, for hallucination mitigation. The core step is to first classify the queries, then perform different processes of hallucination mitigation based on the classification result, just like a dentist first observes the teeth and then makes a plan. In a simple deployment, Dentist can classify queries as perception or reasoning and easily mitigate potential hallucinations in answers which has been demonstrated in our experiments. On MMbench, we achieve a 13.44%/10.2%/15.8% improvement in accuracy on Image Quality, a Coarse Perception visual question answering (VQA) task, over the baseline InstructBLIP/LLaVA/VisualGLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
