多模态因果推理基准：挑战视觉大语言模型跨模态辨别因果联系
Multimodal Causal Reasoning Benchmark: Challenging Vision Large Language Models to Discern Causal Links Across Modalities
AI总结：
本文提出多模态因果推理基准MuCR，利用合成孪生图像和文本对挑战MLLMs，发现其跨模态推理弱于纯文本，并开发VcCoT策略通过突出视觉线索有效增强该能力。
摘要：
多模态大语言模型（MLLMs）在包括因果推理在内的复杂文本推理任务中展现出了卓越的思维链（CoT）推理能力。然而，当关键线索隐藏在视觉细节中时，这些因果关系是否仍然显而易见？如果不是，哪些因素可能影响跨模态泛化？我们能否有效增强它们在文本和视觉两种模态上进行稳健因果推理的能力？受这些问题驱动，我们引入了MuCR——一个新颖的多模态因果推理基准，利用合成的孪生图像和文本对来挑战MLLMs。此外，我们从多个角度开发了定制化指标，包括图像级匹配、短语级理解和句子级解释，以全面评估MLLMs的理解能力。我们的实验表明，当前MLLMs在多模态因果推理方面的表现不如它们在纯文本环境中的表现。此外，我们发现跨图像识别视觉线索是有效跨模态泛化的关键。最后，我们提出了一种VcCoT策略，该策略能更好地突出视觉线索，我们的结果证实了其在增强多模态因果推理方面的有效性。项目可在以下网址获取：https://github.com/Zhiyuan-Li-John/MuCR
英文摘要：
Multimodal Large Language Models (MLLMs) have showcased exceptional Chain-of-Thought (CoT) reasoning ability in complex textual inference tasks including causal reasoning. However, will these causalities remain straightforward when crucial hints hide in visual details? If not, what factors might influence cross-modal generalization? Whether we can effectively enhance their capacity for robust causal inference across both text and vision? Motivated by these, we introduce MuCR - a novel Multimodal Causal Reasoning benchmark that leverages synthetic siamese images and text pairs to challenge MLLMs. Additionally, we develop tailored metrics from multiple perspectives, including image-level match, phrase-level understanding, and sentence-level explanation, to comprehensively assess MLLMs' comprehension abilities. Our experiments reveal that current MLLMs fall short in multimodal causal reasoning compared to their performance in purely textual settings. Additionally, we find that identifying visual cues across images is key to effective cross-modal generalization. Finally, we propose a VcCoT strategy that better highlights visual cues, and our results confirm its efficacy in enhancing multimodal causal reasoning. The project is available at: https://github.com/Zhiyuan-Li-John/MuCR
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
