arXiv 2507.046802025-08-19cs.LGcs.AIcs.CV

识别、隔离与清除：通过自进化蒸馏缓解大视觉语言模型中的幻觉

Identify, Isolate, and Purge: Mitigating Hallucinations in LVLMs via Self-Evolving Distillation

Wenhao Li, Xiu Su, Jingyi Wu, Feng Yang, Yang Liu, Yi Chen, Shan You, Chang Xu

备注：: In Figure 2, the correlation coefficient and the scatter plot do not match. I calculated this correlation using two sets of settings. I used the scatter plot from setting A, but accidentally wrote the correlation coefficient, r, from setting B

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出自进化蒸馏方法SEED，通过识别、隔离并清除LVLM内部知识中的幻觉，结合模式寻求进化和幻觉消除适配器，在LLaVA-1.5等模型上显著缓解幻觉。

摘要：

大视觉语言模型（LVLMs）在多媒体等众多领域已展现出显著进展。然而，幻觉问题严重限制了其可信度与应用潜力。现有缓解方法通常依赖外部工具或多轮推理比较，这显著增加了推理时间。本文提出自进化蒸馏（SEED），该方法在LVLMs内部知识中识别幻觉，将其隔离并清除，然后将净化后的知识蒸馏回模型，从而实现自进化。此外，我们发现传统蒸馏方法容易在LVLMs的输出空间中引入空洞区域。为解决这一问题，我们提出了一种模式寻求进化方法，通过蒸馏捕获净化后知识分布的主导模式，从而避免空洞区域可能产生的混乱结果。进一步，我们引入幻觉消除适配器，通过学习净化后的知识来校正原始模型的暗知识。在多个基准上的大量实验验证了SEED的优越性，表明其在缓解代表性LVLM模型（如LLaVA-1.5和InternVL2）的幻觉方面有显著改进。值得注意的是，LLaVA-1.5在幻觉评估指标POPE-Random上的F1分数从81.3提升至88.3。

英文摘要：

Large Vision-Language Models (LVLMs) have demonstrated remarkable advancements in numerous areas such as multimedia. However, hallucination issues significantly limit their credibility and application potential. Existing mitigation methods typically rely on external tools or the comparison of multi-round inference, which significantly increase inference time. In this paper, we propose \textbf{SE}lf-\textbf{E}volving \textbf{D}istillation (\textbf{SEED}), which identifies hallucinations within the inner knowledge of LVLMs, isolates and purges them, and then distills the purified knowledge back into the model, enabling self-evolution. Furthermore, we identified that traditional distillation methods are prone to inducing void spaces in the output space of LVLMs. To address this issue, we propose a Mode-Seeking Evolving approach, which performs distillation to capture the dominant modes of the purified knowledge distribution, thereby avoiding the chaotic results that could emerge from void spaces. Moreover, we introduce a Hallucination Elimination Adapter, which corrects the dark knowledge of the original model by learning purified knowledge. Extensive experiments on multiple benchmarks validate the superiority of our SEED, demonstrating substantial improvements in mitigating hallucinations for representative LVLM models such as LLaVA-1.5 and InternVL2. Remarkably, the F1 score of LLaVA-1.5 on the hallucination evaluation metric POPE-Random improved from 81.3 to 88.3.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。