当一种模态破坏其他模态：多模态推理的诊断视角
When One Modality Sabotages the Others: A Diagnostic Lens on Multimodal Reasoning
AI总结：
本文提出模态破坏诊断失败模式及轻量级模型无关评估层，将各模态视为智能体进行审计，揭示多模态融合中的错误主导机制，为推理诊断与干预提供框架。
摘要：
尽管多模态大语言模型（MLLMs）迅速发展，其推理轨迹仍不透明：通常不清楚哪种模态驱动了预测、冲突如何解决，或何时某一流占据主导。本文引入模态破坏这一诊断失败模式，即高置信度的单模态错误覆盖其他证据并误导融合结果。为分析此类动态，我们提出一个轻量级、模型无关的评估层，将每种模态视为一个智能体，生成候选标签和用于审计的简短自我评估。简单的融合机制聚合这些输出，暴露贡献者（支持正确结果的模态）和破坏者（产生误导的模态）。在多模态情感识别基准上对基础模型进行案例研究应用我们的诊断层，揭示了系统性的可靠性特征，为失败是源于数据集伪影还是模型局限性提供了见解。更广泛地说，我们的框架为多模态推理提供了诊断支架，支持对融合动态的原则性审计，并为可能的干预提供信息。
英文摘要：
Despite rapid growth in multimodal large language models (MLLMs), their reasoning traces remain opaque: it is often unclear which modality drives a prediction, how conflicts are resolved, or when one stream dominates. In this paper, we introduce modality sabotage, a diagnostic failure mode in which a high-confidence unimodal error overrides other evidence and misleads the fused result. To analyze such dynamics, we propose a lightweight, model-agnostic evaluation layer that treats each modality as an agent, producing candidate labels and a brief self-assessment used for auditing. A simple fusion mechanism aggregates these outputs, exposing contributors (modalities supporting correct outcomes) and saboteurs (modalities that mislead). Applying our diagnostic layer in a case study on multimodal emotion recognition benchmarks with foundation models revealed systematic reliability profiles, providing insight into whether failures may arise from dataset artifacts or model limitations. More broadly, our framework offers a diagnostic scaffold for multimodal reasoning, supporting principled auditing of fusion dynamics and informing possible interventions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
