多模态大语言模型中模态干扰的诊断与缓解
Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models
AI总结：
本文提出了一种统一的微调框架，通过结合启发式和对抗性扰动的数据增强与输出一致性正则化，有效缓解多模态大语言模型中的模态干扰问题，提升模型的鲁棒性和泛化能力。
摘要：
多模态大语言模型在多模态基准测试中表现出色，但在面对虚假模态干扰时往往表现不佳，例如在视觉理解中出现无关文本，或在问答中出现无关视觉内容。其核心在于，模态干扰指的是非必要模态的虚假信号会扭曲模型决策，我们通过因果性和扰动基的诊断实验系统地分析了这一问题。为了解决这一问题，我们提出了一种统一的微调框架，结合启发式和对抗性扰动的数据增强方法，以及原始输入和扰动输入之间的输出级一致性正则化。在图像密集、文本密集和多模态基准测试中进行的广泛实验，涵盖了多种MLLM架构和模型规模，证明了在单模态鲁棒性和泛化能力上的一致改进，同时提高了标准多模态性能。
英文摘要：
Multimodal Large Language Models demonstrate strong performance on multimodal benchmarks, yet often exhibit poor robustness when exposed to spurious modality interference, such as irrelevant text in vision understanding, or irrelevant visual content in question answering. At its core, modality interference refers to cases where spurious signals from non-essential modalities distort model decisions, which we systematically analyze through causal, perturbation-based diagnostic experiments. To address this problem, we propose a unified finetuning framework that combines heuristic and adversarial perturbation-based data augmentation with output-level consistency regularization between original and perturbed inputs. Extensive experiments across image-heavy, text-heavy, and multimodal benchmarks, spanning multiple MLLM architectures and model scales, demonstrate consistent improvements in unimodal robustness and generalization, while improving standard multimodal performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
