arXiv 2403.183462024-11-13cs.CLcs.CV

量化与缓解多模态大语言模型中的单模态偏差：因果视角

Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective

Shanghai AI Laboratory（上海人工智能实验室）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）

Meiqi Chen, Yixin Cao, Yan Zhang, Chaochao Lu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态大模型过度依赖单模态偏差导致错误的问题，提出因果框架分析其对预测的影响，并构建MORE数据集与因果增强智能体框架CAVE，有效提升了模型的推理能力与偏差缓解效果。

摘要：

大型语言模型（LLMs）的最新进展促进了多模态LLMs（MLLMs）的发展。尽管能力出色，MLLMs常因过度依赖单模态偏差（如语言偏差和视觉偏差）导致复杂多模态任务中答案错误或幻觉。为研究此问题，我们提出因果框架解释视觉问答（VQA）中的偏差。在此框架内，我们进行深入因果分析以评估这些偏差对MLLM预测的因果效应。基于分析，我们引入：1)一个新型MORE数据集，包含12,000个需要多跳推理且克服单模态偏差的挑战性VQA实例。2)一个因果增强的智能体框架CAVE，引导模型全面整合不同模态信息并缓解偏差。实验表明，MLLMs在MORE上表现不佳，表明存在强烈单模态偏差和有限语义理解。然而，当集成我们的CAVE时，推理和偏差缓解方面出现可喜改进。这些发现为开发更稳健MLLMs提供重要见解，并助力推进具备更深理解与推理能力的多模态AI系统这一更广泛目标。项目页面位于https://github.com/OpenCausaLab/MORE。

英文摘要：

Recent advancements in Large Language Models (LLMs) have facilitated the development of Multimodal LLMs (MLLMs). Despite their impressive capabilities, MLLMs often suffer from over-reliance on unimodal biases (e.g., language bias and vision bias), leading to incorrect answers or hallucinations in complex multimodal tasks. To investigate this issue, we propose a causal framework to interpret the biases in Visual Question Answering (VQA) problems. Within this framework, we conduct an in-depth causal analysis to assess the causal effect of these biases on MLLM predictions. Based on the analysis, we introduce 1) a novel MORE dataset with 12,000 challenging VQA instances requiring multi-hop reasoning and overcoming unimodal biases. 2) a causality-enhanced agent framework CAVE that guides models to comprehensively integrate information from different modalities and mitigate biases. Our experiments show that MLLMs perform poorly on MORE, indicating strong unimodal biases and limited semantic understanding. However, when integrated with our CAVE, promising improvements in reasoning and bias mitigation can be seen. These findings provide important insights for the development of more robust MLLMs and contribute to the broader goal of advancing multimodal AI systems capable of deeper understanding and reasoning. Our project page is at https://github.com/OpenCausaLab/MORE.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。