组合与融合：重新审视多模态推理中的基础瓶颈
Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning
AI总结：
本文通过逻辑基础评估框架，发现多模态推理中模态交互的六个模式影响推理效果，指出任务组合和融合是主要瓶颈，提出通过分步提示和早融控制提升推理性能。
摘要：
多模态大语言模型（MLLMs）通过整合文本、视觉和音频等多样化输入提升推理能力。然而，跨模态推理仍缺乏系统研究，现有结果存在矛盾。本文提出一个逻辑基础评估框架，将多模态推理分为六个交互模式，分析模态间事实分布和逻辑组合的影响。实验表明，额外模态只有在提供独立且充分的推理路径时才提升性能，而冗余或链式蕴含支持常损害性能。推理退化表现为弱模态拖低整体表现、冲突偏袒某些模态、以及不同模态信号整合无效。本文识别出两个核心问题：任务组合瓶颈（识别与推理无法联合执行）和融合瓶颈（早期整合引入偏差）。进一步研究发现，注意力模式无法编码事实重要性，但分步提示（识别后推理）可恢复性能，验证了任务组合瓶颈。此外，模态身份在早期层可恢复，软化早期融合注意力可提升推理，揭示了偏见融合为另一失败模式。总体而言，整合而非感知是多模态推理的主要障碍，提示了组合感知训练和早融控制作为有前景的方向。
英文摘要：
Multimodal large language models (MLLMs) promise enhanced reasoning by integrating diverse inputs such as text, vision, and audio. Yet cross-modal reasoning remains underexplored, with conflicting reports on whether added modalities help or harm performance. These inconsistencies stem from a lack of controlled evaluation frameworks and analysis of models' internals to isolate when and why modality interactions support or undermine reasoning. We address this gap through a logic-grounded evaluation framework that categorizes multimodal reasoning into six interaction patterns, varying how facts are distributed across modalities and logically combined. Empirically, additional modalities enhance reasoning only when they provide independent and sufficient reasoning paths, while redundant or chained entailment support often hurts performance. Moreover, reasoning degrades in three systematic ways: weaker modalities drag down overall performance, conflicts bias preference toward certain modalities, and joint signals from different modalities fail to be integrated effectively. Therefore, we identify two core failures: task-composition bottleneck, where recognition and reasoning cannot be jointly executed in one pass, and fusion bottleneck, where early integration introduces bias. For further investigation, we find that attention patterns fail to encode fact usefulness, but a simple two-step prompting (recognize then reason) restores performance, confirming the task-composition bottleneck. Moreover, modality identity remains recoverable in early layers, and softening attention in early fusion improves reasoning, highlighting biased fusion as another failure mode. Overall, our findings show that integration, not perception, is the main barrier to multimodal reasoning, suggesting composition-aware training and early fusion control as promising directions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
