交错模态思维链
Interleaved-Modal Chain-of-Thought
AI总结：
针对视觉语言模型纯文本推理难以关联图像细节的问题，本文提出ICoT并用即插即用的ADS依据注意力图选择图像区域，最高提升14%性能。
摘要：
思维链（Chain-of-Thought，CoT）提示可促使大型语言模型（LLM）在得出最终答案之前生成一系列中间推理步骤。然而，当迁移到视觉语言模型（VLM）时，其纯文本推理依据难以表达与原始图像的细粒度关联。本文提出一种融入图像的多模态思维链，名为 Interleaved-modal Chain-of-Thought（ICoT，交错模态思维链），它生成由成对视觉和文本推理依据组成的顺序推理步骤，以推断最终答案。直观而言，新颖的 ICoT 要求 VLM 能够生成细粒度的交错模态内容，而当前 VLM 很难实现这一点。考虑到所需的视觉信息通常是输入图像的一部分，我们提出 Attention-driven Selection（ADS，注意力驱动选择），以在现有 VLM 上实现 ICoT。ADS 智能地插入输入图像的区域，以生成交错模态推理步骤，且额外延迟可忽略。ADS 仅依赖 VLM 的注意力图，无需参数化，因此它是一种即插即用策略，可推广到一系列 VLM。我们将 ADS 应用于两种具有不同架构的流行 VLM，以实现 ICoT。在三个基准上的大量评估表明，与现有多模态 CoT 提示方法相比，ICoT 提示实现了显著的性能（最高达14%）和可解释性提升。
英文摘要：
Chain-of-Thought (CoT) prompting elicits large language models (LLMs) to produce a series of intermediate reasoning steps before arriving at the final answer. However, when transitioning to vision-language models (VLMs), their text-only rationales struggle to express the fine-grained associations with the original image. In this paper, we propose an image-incorporated multimodal Chain-of-Thought, named \textbf{Interleaved-modal Chain-of-Thought (ICoT)}, which generates sequential reasoning steps consisting of paired visual and textual rationales to infer the final answer. Intuitively, the novel ICoT requires VLMs to enable the generation of fine-grained interleaved-modal content, which is hard for current VLMs to fulfill. Considering that the required visual information is usually part of the input image, we propose \textbf{Attention-driven Selection (ADS)} to realize ICoT over existing VLMs. ADS intelligently inserts regions of the input image to generate the interleaved-modal reasoning steps with ignorable additional latency. ADS relies solely on the attention map of VLMs without the need for parameterization, and therefore it is a plug-and-play strategy that can be generalized to a spectrum of VLMs. We apply ADS to realize ICoT on two popular VLMs of different architectures. Extensive evaluations of three benchmarks have shown that ICoT prompting achieves substantial performance (up to 14\%) and interpretability improvements compared to existing multimodal CoT prompting methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
