通过再平衡对比解码缓解视觉-语言模型中的幻觉
Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding
AI总结：
针对视觉-语言模型偏重文本token而产生幻觉的问题，本文提出双分支RBD方法，通过噪声激发与显著视觉token选择再平衡注意力，并在CHAIR和POPE上取得更优效果。
摘要：
尽管视觉-语言模型（VLM）在视觉问答和图像描述等任务中展现出令人瞩目的能力，但它们仍受到幻觉问题的困扰。对这些模型中注意力分布的分析表明，VLM倾向于处理文本token而非视觉token。这种注意力分布的不平衡导致VLM在多模态知识冲突情况下偏向文本知识，从而产生与图像信息的差异。本文提出再平衡对比解码（RBD）方法，该方法利用文本分支和视觉分支重新校准VLM中的注意力分布。具体而言，文本分支注入图像噪声以激发模型对文本的依赖，从而减少文本偏差。同时，视觉分支专注于选择显著token，改进注意力机制以突出主要主体。这种双分支策略使RBD方法能够在增强视觉信息的同时减少文本偏差。实验结果表明，我们的方法RBD在CHAIR和POPE指标上优于现有方法，可在不降低模型通用能力的情况下缓解幻觉。
英文摘要：
Although Visual-Language Models (VLMs) have shown impressive capabilities in tasks like visual question answering and image captioning, they still struggle with hallucinations. Analysis of attention distribution in these models shows that VLMs tend to processing textual tokens rather than visual tokens. This imbalance of attention distribution causes VLMs to favor textual knowledge in the case of multimodal knowledge conflicts, resulting in differences from the image information. In this paper, we propose Re-Balancing Contrastive Decoding (RBD) method, which employs textual and visual branches to recalibrate attention distribution in VLMs. Specifically, the textual branch injects image noise to stimulate the model's dependency on text, thereby reducing textual bias. Concurrently, the visual branch focuses on the selection of significant tokens, refining the attention mechanism to highlight the primary subject. This dual-branch strategy enables the RBD method to diminish textual bias while enhancing visual information. Experimental results demonstrate that our method, RBD, outperforms the existing methods by the CHAIR and POPE metrics, mitigate hallucinations without reducing the model's general capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
