多模态大语言模型中的免训练推理与反思
Training-Free Reasoning and Reflection in MLLMs
AI总结：
提出FRANK模型，通过分层权重合并与泰勒闭式融合，免训练赋予多模态大模型推理反思能力，在MMMU上以69.2%超越基线。
摘要：
近期，推理大语言模型（如DeepSeek-R1和OpenAI-o1）的进展通过强化学习展示了令人印象深刻的推理能力。然而，将这些能力扩展到多模态大语言模型（MLLMs）受到再训练的高昂成本和高质量、可验证的多模态推理数据集稀缺性的阻碍。本文介绍了FRANK模型，一种免训练且类似R1的多模态大语言模型，它赋予现成的多模态大语言模型推理和反思能力，无需任何梯度更新或额外监督。我们的关键见解是在多模态大语言模型解码器层之间解耦感知和推理。具体来说，我们观察到与较深的解码器层相比，较浅的解码器层将更多注意力分配给视觉标记，而较深的解码器层则集中于文本语义。这一观察启发了一种分层权重合并方法，将视觉预训练的多模态大语言模型与推理专用的语言模型相结合。为此，我们提出了一种逐层的、基于泰勒级数的闭式融合机制，将推理能力整合到深层解码器层，同时保持浅层解码器层中的视觉基础。在具有挑战性的多模态推理基准上的大量实验证明了我们方法的有效性。在MMMU基准上，我们的模型FRANK-38B达到了69.2的准确率，比最强基线InternVL2.5-38B高出5.3个百分点，甚至超过了专有的GPT-4o模型。我们的项目主页位于：http://iip.whu.edu.cn/frank/index.html
英文摘要：
Recent advances in Reasoning LLMs (e.g., DeepSeek-R1 and OpenAI-o1) have showcased impressive reasoning capabilities via reinforcement learning. However, extending these capabilities to Multimodal LLMs (MLLMs) is hampered by the prohibitive costs of retraining and the scarcity of high-quality, verifiable multimodal reasoning datasets. This paper introduces FRANK Model, a training-FRee ANd r1-liKe MLLM that imbues off-the-shelf MLLMs with reasoning and reflection abilities, without any gradient updates or extra supervision. Our key insight is to decouple perception and reasoning across MLLM decoder layers. Specifically, we observe that compared to the deeper decoder layers, the shallow decoder layers allocate more attention to visual tokens, while the deeper decoder layers concentrate on textual semantics. This observation motivates a hierarchical weight merging approach that combines a visual-pretrained MLLM with a reasoning-specialized LLM. To this end, we propose a layer-wise, Taylor-derived closed-form fusion mechanism that integrates reasoning capacity into deep decoder layers while preserving visual grounding in shallow decoder layers. Extensive experiments on challenging multimodal reasoning benchmarks demonstrate the effectiveness of our approach. On the MMMU benchmark, our model FRANK-38B achieves an accuracy of 69.2, outperforming the strongest baseline InternVL2.5-38B by +5.3, and even surpasses the proprietary GPT-4o model. Our project homepage is at: http://iip.whu.edu.cn/frank/index.html
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
