用于多模态大语言模型中视觉推理精炼的 Image-of-Thought 提示
Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
AI总结：
提出 Image-of-Thought (IoT) 提示方法，通过自动设计视觉信息提取操作逐步生成视觉推理路径，结合文本与视觉推理路径，提升了多模态大语言模型在零样本视觉推理任务中的性能并增强了过程可解释性。
摘要：
近期，Chain-of-Thought (CoT) 及相关基于推理路径的工作显著提升了 Large Language Models (LLMs) 在复杂推理任务中的性能。随着 Multimodal Large Language Models (MLLMs) 的发展，增强其处理复杂多模态推理问题的能力成为一个关键前沿。然而，在 CoT 中融入多模态推理路径尚未得到充分研究。我们提出了 Image-of-Thought (IoT) 提示方法，该方法帮助 MLLMs 逐步提取视觉推理路径。具体而言，IoT 提示能够根据输入图像和问题，自动设计关键的视觉信息提取操作。每一步视觉信息精炼都会识别支持复杂视觉推理问题答案的特定视觉推理路径。除文本 CoT 之外，IoT 同时利用视觉和文本推理路径，以帮助 MLLMs 理解复杂的多模态信息。IoT 提示在不同 MLLMs 的多种视觉理解任务中，均提升了零样本视觉推理性能。此外，IoT 提示生成的逐步视觉特征解释阐明了视觉推理过程，有助于分析大型多模态模型的认知过程。
英文摘要：
Recent advancements in Chain-of-Thought (CoT) and related rationale-based works have significantly improved the performance of Large Language Models (LLMs) in complex reasoning tasks. With the evolution of Multimodal Large Language Models (MLLMs), enhancing their capability to tackle complex multimodal reasoning problems is a crucial frontier. However, incorporating multimodal rationales in CoT has yet to be thoroughly investigated. We propose the Image-of-Thought (IoT) prompting method, which helps MLLMs to extract visual rationales step-by-step. Specifically, IoT prompting can automatically design critical visual information extraction operations based on the input images and questions. Each step of visual information refinement identifies specific visual rationales that support answers to complex visual reasoning questions. Beyond the textual CoT, IoT simultaneously utilizes visual and textual rationales to help MLLMs understand complex multimodal information. IoT prompting has improved zero-shot visual reasoning performance across various visual understanding tasks in different MLLMs. Moreover, the step-by-step visual feature explanations generated by IoT prompting elucidate the visual reasoning process, aiding in analyzing the cognitive processes of large multimodal models
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
