CIMR：面向大视觉语言模型鲁棒指令遵循的上下文感知迭代多模态推理
CIMR: Contextualized Iterative Multimodal Reasoning for Robust Instruction Following in LVLMs
AI总结：
针对大视觉语言模型难处理复杂多步多模态指令的问题，提出带上下文感知迭代推理与自校正模块的CIMR框架，在新MAP数据集上准确率达91.5%，优于多款主流模型。
摘要：
大语言模型（LLM）和大视觉语言模型（LVLM）的快速发展提升了我们处理与生成人类语言及视觉信息的能力。然而，这类模型往往难以应对需要逻辑推理、动态反馈整合与迭代自校正的复杂多步多模态指令。为解决该问题，我们提出CIMR（上下文感知迭代多模态推理）——一种引入上下文感知迭代推理与自校正模块的新型框架。CIMR分为两个阶段运行：首先是初始推理与响应生成，随后利用解析后的多模态反馈进行迭代优化。动态融合模块会在每一步深度整合文本、视觉与上下文特征。我们在Visual Instruction Tuning（VIT，视觉指令调优）数据集上微调LLaVA-1.5-7B，并在新提出的Multi-modal Action Planning（MAP，多模态动作规划）数据集上评估CIMR。CIMR的准确率达到91.5%，优于GPT-4V（89.2%）、LLaVA-1.5（78.5%）、MiniGPT-4（75.3%）与InstructBLIP（72.8%）等当前最优模型，证明了其迭代推理与自校正能力在复杂任务中的有效性。
英文摘要：
The rapid advancement of Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) has enhanced our ability to process and generate human language and visual information. However, these models often struggle with complex, multi-step multi-modal instructions that require logical reasoning, dynamic feedback integration, and iterative self-correction. To address this, we propose CIMR: Contextualized Iterative Multimodal Reasoning, a novel framework that introduces a context-aware iterative reasoning and self-correction module. CIMR operates in two stages: initial reasoning and response generation, followed by iterative refinement using parsed multi-modal feedback. A dynamic fusion module deeply integrates textual, visual, and contextual features at each step. We fine-tune LLaVA-1.5-7B on the Visual Instruction Tuning (VIT) dataset and evaluate CIMR on the newly introduced Multi-modal Action Planning (MAP) dataset. CIMR achieves 91.5% accuracy, outperforming state-of-the-art models such as GPT-4V (89.2%), LLaVA-1.5 (78.5%), MiniGPT-4 (75.3%), and InstructBLIP (72.8%), demonstrating the efficacy of its iterative reasoning and self-correction capabilities in complex tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
