揭示大型视觉-语言模型中的跨模态知识冲突
Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models
AI总结：
本文定义了大型视觉-语言模型中的跨模态参数化知识冲突问题，提出动态对比解码和提示策略来检测并缓解冲突，在ViQuAE和InfoSeek数据集上显著提升了模型准确率。
摘要：
大型视觉-语言模型（LVLMs）在捕获和推理多模态输入方面展现了令人印象深刻的能力。然而，这些模型容易出现参数化知识冲突，这源于其视觉和语言组件之间所表示知识的不一致性。在本文中，我们正式定义了跨模态参数化知识冲突问题，并提出了一种系统方法来检测、解释和缓解它们。我们引入了一个流水线来识别视觉和文本答案之间的冲突，显示最近的LVLMs中跨模态的冲突率持续偏高，且与模型规模无关。我们进一步调查了这些冲突如何干扰推理过程，并提出了一种对比度量来将冲突样本与其他样本区分开来。基于这些见解，我们开发了一种新颖的动态对比解码方法，该方法根据答案置信度移除从较低置信度模态组件推断出的不良logits。对于不提供logits的模型，我们还引入了两种基于提示的策略来缓解冲突。我们的方法在ViQuAE和InfoSeek数据集上实现了令人满意的准确率提升。具体而言，使用LLaVA-34B，我们提出的动态对比解码将平均准确率提高了2.24%。
英文摘要：
Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities for capturing and reasoning over multimodal inputs. However, these models are prone to parametric knowledge conflicts, which arise from inconsistencies of represented knowledge between their vision and language components. In this paper, we formally define the problem of $\textbf{cross-modality parametric knowledge conflict}$ and present a systematic approach to detect, interpret, and mitigate them. We introduce a pipeline that identifies conflicts between visual and textual answers, showing a persistently high conflict rate across modalities in recent LVLMs regardless of the model size. We further investigate how these conflicts interfere with the inference process and propose a contrastive metric to discern the conflicting samples from the others. Building on these insights, we develop a novel dynamic contrastive decoding method that removes undesirable logits inferred from the less confident modality components based on answer confidence. For models that do not provide logits, we also introduce two prompt-based strategies to mitigate the conflicts. Our methods achieve promising improvements in accuracy on both the ViQuAE and InfoSeek datasets. Specifically, using LLaVA-34B, our proposed dynamic contrastive decoding improves an average accuracy of 2.24%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
