VASparse：通过视觉感知Token稀疏化实现高效的视觉幻觉缓解
VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsification
AI总结：
针对大视觉语言模型的视觉幻觉问题，提出即插即用的VASparse解码算法，通过视觉感知token稀疏化与稀疏视觉对比解码，在保持解码速度的同时实现了当前最优的幻觉缓解效果。
摘要：
大视觉语言模型（LVLM）可能生成与现实不符的输出，即视觉幻觉（VH），这严重阻碍了其实际应用。为缓解视觉幻觉，已有多种解码策略被提出以增强视觉信息，但这类方法大多需要二次解码与回滚，大幅降低了推理速度。本文从token稀疏性角度出发，提出了一种高效的即插即用解码算法——基于视觉感知稀疏化的VASparse，用于缓解视觉幻觉。VASparse的灵感源于两项实证观察：（1）大视觉语言模型中注意力的稀疏激活特性；（2）与视觉无关的token稀疏化会加剧视觉幻觉。基于这些洞察，我们提出了一种兼顾效率与可信度的新型token稀疏化策略。具体而言，VASparse在解码过程中执行视觉感知的token选择策略，在有效保留视觉上下文的同时减少冗余token。此外，我们创新性地引入了基于稀疏的视觉对比解码方法，无需二次解码带来的时间开销即可重新校准幻觉输出的分布。随后，VASparse重新校准注意力分数，以惩罚大视觉语言模型对文本token的注意力下沉现象。在四个主流基准上的大量实验证实，VASparse无需额外训练或后处理，即可在不同大视觉语言模型系列中有效缓解视觉幻觉。值得注意的是，VASparse在缓解视觉幻觉方面达到了当前最优性能，同时保持了具有竞争力的解码速度。代码可在https://github.com/mengchuang123/VASparse-github获取。
英文摘要：
Large Vision-Language Models (LVLMs) may produce outputs that are unfaithful to reality, also known as visual hallucinations (VH), which significantly impedes their real-world usage. To alleviate VH, various decoding strategies have been proposed to enhance visual information. However, many of these methods may require secondary decoding and rollback, which significantly reduces inference speed. In this work, we propose an efficient plug-and-play decoding algorithm via Visual-Aware Sparsification (VASparse) from the perspective of token sparsity for mitigating VH. VASparse is inspired by empirical observations: (1) the sparse activation of attention in LVLMs, and (2) visual-agnostic tokens sparsification exacerbates VH. Based on these insights, we propose a novel token sparsification strategy that balances efficiency and trustworthiness. Specifically, VASparse implements a visual-aware token selection strategy during decoding to reduce redundant tokens while preserving visual context effectively. Additionally, we innovatively introduce a sparse-based visual contrastive decoding method to recalibrate the distribution of hallucinated outputs without the time overhead associated with secondary decoding. Subsequently, VASparse recalibrates attention scores to penalize attention sinking of LVLMs towards text tokens. Extensive experiments across four popular benchmarks confirm the effectiveness of VASparse in mitigating VH across different LVLM families without requiring additional training or post-processing. Impressively, VASparse achieves state-of-the-art performance for mitigating VH while maintaining competitive decoding speed. Code is available at https://github.com/mengchuang123/VASparse-github.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
