如果电视关着会怎样？考察多模态语言模型的反事实推理能力
What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models
AI总结：
本文构建涵盖数值与布尔查询、混合真实与合成数据的 C-VQA 数据集，评估多模态大模型的视觉反事实推理能力，发现部分模型性能最高下降 40%。
摘要：
反事实推理是人类认知的一个基本方面，它涉及思考既定事实或过去事件的替代情形，能够显著增强我们在规划和决策方面的能力。鉴于当前多模态大型语言模型的进展，我们探索了它们在反事实推理方面的有效性。为推动这一研究，我们引入了一个名为 C-VQA 的新颖数据集，专门用于测试现代多模态大型语言模型的反事实推理能力。该数据集通过向原始问题中注入反事实预设而构建，涵盖数值型和布尔型查询等多种类型。它混合包含真实数据与合成数据，代表了广泛的难度层级。我们利用该数据集对当代视觉-语言模型进行了全面评估，结果显示其性能出现大幅下降，部分模型的降幅最高达 40%，这凸显出当前模型与类人视觉推理能力之间仍存在显著差距。我们希望该数据集能够成为评估模型反事实推理能力的重要基准。代码和数据集已公开发布于 https://bzhao.me/C-VQA/。
英文摘要：
Counterfactual reasoning, a fundamental aspect of human cognition, involves contemplating alternatives to established facts or past events, significantly enhancing our abilities in planning and decision-making. In light of the advancements in current multi-modal large language models, we explore their effectiveness in counterfactual reasoning. To facilitate this investigation, we introduce a novel dataset, C-VQA, specifically designed to test the counterfactual reasoning capabilities of modern multi-modal large language models. This dataset is constructed by infusing original questions with counterfactual presuppositions, spanning various types such as numerical and boolean queries. It encompasses a mix of real and synthetic data, representing a wide range of difficulty levels. Our thorough evaluations of contemporary vision-language models using this dataset have revealed substantial performance drops, with some models showing up to a 40% decrease, highlighting a significant gap between current models and human-like vision reasoning capabilities. We hope our dataset will serve as a vital benchmark for evaluating the counterfactual reasoning capabilities of models. Code and dataset are publicly available at https://bzhao.me/C-VQA/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
