语言模型能看得更清楚：面向 LLM 多模态推理的视觉对比解码
Language Models Can See Better: Visual Contrastive Decoding For LLM Multimodal Reasoning
AI总结：
本文提出免训练的 MVCD 框架，通过视觉信号文本化与对比示例解码增强 LLM 的视觉感知和多模态推理，并在五个问答数据集、四个 LLM 上验证了准确率提升。
摘要：
尽管大型语言模型（LLM）在语言任务的推理和生成方面表现出色，但它们并非专门为多模态挑战而设计。然而，训练多模态大型语言模型（MLLM）需要大量资源，并受到各种训练限制的约束。在本文中，我们提出了基于模块化的视觉对比解码（Modular-based Visual Contrastive Decoding，MVCD）框架，以消除这一障碍。我们的框架利用 LLM 的上下文学习（In-Context Learning，ICL）能力以及所提出的、专门为该框架定制的视觉对比示例解码（contrastive-example decoding，CED），无需任何额外训练。通过将视觉信号转换为文本，并在解码过程中关注对比输出分布，我们可以突出上下文示例引入的新信息，探索它们之间的联系，并避免过度依赖先前编码的知识。MVCD 增强了 LLM 的视觉感知能力，使其能够看见输入视觉内容并对其进行推理。为证明 MVCD 的有效性，我们在五个问答数据集上使用四个 LLM 进行了实验。结果不仅显示模型准确率获得持续提升，也很好地解释了解码策略内部的有效组成部分。代码将在 https://github.com/Pbhgit/MVCD 提供。
英文摘要：
Although Large Language Models (LLMs) excel in reasoning and generation for language tasks, they are not specifically designed for multimodal challenges. Training Multimodal Large Language Models (MLLMs), however, is resource-intensive and constrained by various training limitations. In this paper, we propose the Modular-based Visual Contrastive Decoding (MVCD) framework to move this obstacle. Our framework leverages LLMs' In-Context Learning (ICL) capability and the proposed visual contrastive-example decoding (CED), specifically tailored for this framework, without requiring any additional training. By converting visual signals into text and focusing on contrastive output distributions during decoding, we can highlight the new information introduced by contextual examples, explore their connections, and avoid over-reliance on prior encoded knowledge. MVCD enhances LLMs' visual perception to make it see and reason over the input visuals. To demonstrate MVCD's effectiveness, we conduct experiments with four LLMs across five question answering datasets. Our results not only show consistent improvement in model accuracy but well explain the effective components inside our decoding strategy. Our code will be available at https://github.com/Pbhgit/MVCD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
