Chartographer: 用于评估视觉语言模型的反事实图表生成
Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models
AI总结：
提出 Chartographer 框架，通过将图表逆向工程为可执行代码并生成反事实变体，揭示视觉语言模型在图表问答中的视觉推理缺陷。
摘要：
图表问答（QA）基准旨在提出需要视觉推理才能正确回答的问题，但模型通常可以通过捷径或基于自身背景知识对图表的先验熟悉度来达到解决方案。为了严格评估视觉推理，我们提出了反事实图表，其中图表问答任务保持不变，但底层图表和相应答案发生变化。我们引入了 Chartographer，一个将图表逆向工程为可执行代码、验证重构保真度、生成种子控制的反事实变体以及从可执行问答逻辑中推导新答案的框架。我们将该框架应用于现有的图表 QA 数据集，并评估了专有和开源视觉语言模型（VLM），测量了变化敏感性和泛化能力。反事实图表揭示了单一图表性能所隐藏的失败：VLM 在正确回答原始图表后通常无法泛化。我们发现，当更新后的图表需要新的视觉推理路径时，失败最为普遍。
英文摘要：
Chart question-answering (QA) benchmarks aim to pose questions that require visual reasoning to correctly answer, but vision-language models (VLMs) can often reach solutions through shortcuts or prior familiarity with a chart or question. To strictly evaluate visual reasoning, we propose counterfactual charts where the chart-question task remains fixed, but the underlying data and the corresponding answer are varied. We introduce Chartographer, a framework to reverse engineer charts into executable code, validate reconstruction fidelity, generate counterfactual variants, and derive new answers from executable QA logic. We apply this framework to existing chart QA datasets and evaluate proprietary and open-source VLMs, measuring variant sensitivity and generalizability. Counterfactual charts reveal failures hidden by single-chart performance: VLMs often fail to generalize after answering the original chart correctly. We find that failures are most prevalent when updated charts require novel visual reasoning pathways.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
