自描述多模态交互调优：放大可利用冗余以实现鲁棒的视觉语言模型
Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models
AI总结：
针对视觉语言模型中的幻觉和鲁棒性问题，提出自描述多模态交互调优方法，通过放大模态间冗余信息来补偿受损模态，并设计多模态交互门机制将独特交互转化为冗余交互，实验表明该方法可减少38.3%的视觉诱导错误并提升16.8%的一致性。
摘要：
当前的视觉语言模型在面对模糊或受损模态时存在幻觉和鲁棒性问题。我们假设这些问题可以通过利用模态间的共享信息来补偿受损模态得到解决。为此，我们分析了多模态交互——模态提供的冗余（共享）、独特（排他）和协同（涌现）任务相关信息——以确定它们对模型可靠性的影响。具体来说，放大冗余交互将增加这种可利用的共享信息以解决这些问题；然而，现代指令数据集通常消除冗余以优先考虑视觉定位。我们通过一个自描述工作流弥合这一差距，该工作流包含一个 extsc{多模态交互门}：一种将独特交互转化为冗余交互的机制。我们的发现表明，增加冗余可以减少38.3%的视觉诱导错误，并提高16.8%的一致性。
英文摘要：
Current vision language models face hallucination and robustness issues against ambiguous or corrupted modalities. We hypothesize that these issues can be addressed by exploiting the shared information between modalities to compensate for the impaired one. To this end, we analyze multimodal interactions -- redundant (shared), unique (exclusive), and synergistic (emergent) task-relevant information provided by the modalities -- to determine their impacts on model reliability. Specifically, amplifying redundant interactions would increase this exploitable shared information to resolve these issues; yet, modern instruction datasets often eliminate redundancies to prioritize visual grounding. We bridge this gap through a self-captioning workflow featuring a \textsc{Multimodal Interaction Gate}: a mechanism to convert unique interactions into redundant interactions. Our findings suggest that increasing redundancy can reduce visual induced errors by 38.3\% and improve consistency by 16.8\%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
