通过结构化相关性图诊断视觉-语言模型中的因果推理
Diagnosing Causal Reasoning in Vision-Language Models via Structured Relevance Graphs
AI总结：
本文提出ViLCaR基准，通过结构化相关性图评估视觉-语言模型的因果推理能力，发现其局限性源于结构指导不足而非推理能力不足。
摘要：
大型视觉-语言模型（LVLMs）在视觉问答基准测试中表现出色，但往往依赖于虚假相关性而非真正的因果推理。现有的评估主要评估答案的正确性，使不清楚失败是源于推理能力有限还是误识别因果相关信息。我们引入了视觉-语言因果图（VLCGs），这是一种结构化的、查询条件的表示方法，明确编码因果相关的对象、属性、关系和场景基础假设。在此基础上，我们提出了ViLCaR，一个包含因果归因、因果推理和问答任务的诊断基准，以及与图对齐的评估指标，这些指标评估了相关性识别，而不仅仅是最终答案的准确性。在最先进的LVLMs上的实验表明，注入结构化相关性信息相比零样本和标准上下文学习显著提高了归因和推理的一致性。这些发现表明，当前LVLM因果推理的局限性主要源于结构指导不足，而不是推理能力的缺乏。
英文摘要：
Large Vision-Language Models (LVLMs) achieve strong performance on visual question answering benchmarks, yet often rely on spurious correlations rather than genuine causal reasoning. Existing evaluations primarily assess the correctness of the answers, making it unclear whether failures arise from limited reasoning capability or from misidentifying causally relevant information. We introduce Vision-Language Causal Graphs (VLCGs), a structured, query-conditioned representation that explicitly encodes causally relevant objects, attributes, relations, and scene-grounded assumptions. Building on this representation, we present ViLCaR, a diagnostic benchmark comprising tasks for Causal Attribution, Causal Inference, and Question Answering, along with graph-aligned evaluation metrics that assess relevance identification beyond final answer accuracy. Experiments in state-of-the-art LVLMs show that injecting structured relevance information significantly improves attribution and inference consistency compared to zero-shot and standard in-context learning. These findings suggest that current limitations in LVLM causal reasoning stem primarily from insufficient structural guidance rather than a lack of reasoning capacity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
