arXiv 2411.127132024-11-19cs.CVcs.AI

CATCH：互补自适应令牌级对比解码以缓解大型视觉语言模型中的幻觉

CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs

Tsinghua University（清华大学）；Pengcheng Laboratory（鹏城实验室）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）；Chang’an University（长安大学）

Zhehan Kan, Ce Zhang, Zihan Liao, Yapeng Tian, Wenming Yang, Junyuan Xiao, Xu Li, Dongmei Jiang, Yaowei Wang, Qingmin Liao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出CATCH方法，通过互补视觉解耦、非视觉筛选和自适应令牌级对比解码，缓解LVLM中因视觉-语言错位导致的幻觉问题，无需特定数据或额外训练即可泛化到新任务。

摘要：

大型视觉语言模型（LVLM）系统展示了令人印象深刻的视觉-语言推理能力，但普遍存在严重幻觉问题，在医疗保健和自主系统等关键领域构成重大风险。尽管此前已有多种缓解幻觉的努力，一个持续存在的问题仍然存在：视觉-语言错位导致的视觉缺陷，形成了视觉处理能力的瓶颈。为应对这一挑战，我们基于信息瓶颈理论开发了互补自适应令牌级对比解码以缓解LVLM中的幻觉（CATCH）。CATCH引入了用于视觉信息分离的互补视觉解耦（CVD）、用于幻觉检测的非视觉筛选（NVS），以及用于幻觉缓解的自适应令牌级对比解码（ATCD）。CATCH解决了与视觉缺陷相关的问题，这些问题会导致细粒度特征感知能力下降和开放式场景中的累积幻觉。该方法适用于各种视觉问答任务，无需任何特定数据或先验知识，并且无需额外训练即可稳健地泛化到新任务，为在各种具有挑战性的应用中推进LVLM开辟了新的可能性。

英文摘要：

Large Vision-Language Model (LVLM) systems have demonstrated impressive vision-language reasoning capabilities but suffer from pervasive and severe hallucination issues, posing significant risks in critical domains such as healthcare and autonomous systems. Despite previous efforts to mitigate hallucinations, a persistent issue remains: visual defect from vision-language misalignment, creating a bottleneck in visual processing capacity. To address this challenge, we develop Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs (CATCH), based on the Information Bottleneck theory. CATCH introduces Complementary Visual Decoupling (CVD) for visual information separation, Non-Visual Screening (NVS) for hallucination detection, and Adaptive Token-level Contrastive Decoding (ATCD) for hallucination mitigation. CATCH addresses issues related to visual defects that cause diminished fine-grained feature perception and cumulative hallucinations in open-ended scenarios. It is applicable to various visual question-answering tasks without requiring any specific data or prior knowledge, and generalizes robustly to new tasks without additional training, opening new possibilities for advancing LVLM in various challenging applications.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。