图像 Token 很重要：通过隐空间编辑缓解基于离散 Tokenizer 的大型视觉语言模型中的幻觉
Image Tokens Matter: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing
AI总结：
针对离散图像tokenizer的大型视觉语言模型因视觉先验导致的幻觉问题，提出通过GNN和对比学习聚类识别视觉缺失token，并在生成时修改隐空间图像嵌入以抑制其影响，有效缓解幻觉且保留表达能力。
摘要：
带有离散图像 tokenizer 的大型视觉语言模型（LVLMs）通过将视觉输入编码为一组有限的 token 来统一多模态表示。尽管它们很有效，我们发现这些模型仍然会幻觉出不存在的物体。我们假设这可能是由于训练期间引入的视觉先验所致：当某些图像 token 频繁在同一空间区域共同出现并代表共享物体时，它们会与这些物体的言语描述产生强烈的关联。结果，模型可能会通过唤起经常与当前 token 共同出现但在视觉上不存在的 token 来产生幻觉。为了验证这一假设，我们使用分割数据集构建了图像 token 的共现图，并采用带有对比学习的图神经网络（GNN），随后使用聚类方法将经常在相似视觉语境中共同出现的 token 进行分组。我们发现幻觉主要对应于其 token 占据输入主导地位的聚类，更具体地说，与图像中存在的 token 相比，这些聚类中视觉上不存在的 token 与幻觉物体显示出高得多的相关性。基于这一观察，我们提出了一种幻觉缓解方法，该方法通过在生成过程中修改隐空间图像嵌入来抑制视觉上不存在的 token 的影响。实验表明，我们的方法在保留表达能力的同时减少了幻觉。代码可在 https://github.com/weixingW/CGC-VTD/tree/main 获取。
英文摘要：
Large Vision-Language Models (LVLMs) with discrete image tokenizers unify multimodal representations by encoding visual inputs into a finite set of tokens. Despite their effectiveness, we find that these models still hallucinate non-existent objects. We hypothesize that this may be due to visual priors induced during training: When certain image tokens frequently co-occur in the same spatial regions and represent shared objects, they become strongly associated with the verbalizations of those objects. As a result, the model may hallucinate by evoking visually absent tokens that often co-occur with present ones. To test this assumption, we construct a co-occurrence graph of image tokens using a segmentation dataset and employ a Graph Neural Network (GNN) with contrastive learning followed by a clustering method to group tokens that frequently co-occur in similar visual contexts. We find that hallucinations predominantly correspond to clusters whose tokens dominate the input, and more specifically, that the visually absent tokens in those clusters show much higher correlation with hallucinated objects compared to tokens present in the image. Based on this observation, we propose a hallucination mitigation method that suppresses the influence of visually absent tokens by modifying latent image embeddings during generation. Experiments show our method reduces hallucinations while preserving expressivity. Code is available at https://github.com/weixingW/CGC-VTD/tree/main
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
