Chart-HQA：图表中假设性问题回答的基准
Chart-HQA: A Benchmark for Hypothetical Question Answering in Charts
AI总结：
本文提出图表假设性问题回答基准Chart-HQA及人机交互数据合成方法HAI，通过施加假设条件迫使多模态大语言模型进行反事实推理，并揭示现有模型在该任务上存在泛化与推理不平衡问题。
摘要：
多模态大语言模型（MLLMs）因其强大的视觉-语义理解能力而受到广泛关注。现有的大多数图表基准评估的是MLLMs从图表中解析信息以回答问题的能力。然而，它们忽视了MLLMs固有的输出偏差，即模型依赖其参数化记忆来回答问题，而不是真正理解图表内容。为了解决这一局限性，我们引入了一个新颖的图表假设性问题回答（HQA）任务，该任务对同一问题施加假设条件，迫使模型基于图表内容进行反事实推理。此外，我们引入了HAI，一种人机交互式数据合成方法，利用大语言模型（LLMs）的高效文本编辑能力与人类专家知识相结合，以低成本生成多样且高质量的HQA数据。利用HAI，我们构建了Chart-HQA，一个由公开可用数据源合成的具有挑战性的基准。在18个不同模型规模的MLLMs上的评估结果表明，当前模型在HQA任务上面临显著的泛化挑战，并表现出不平衡的推理性能。
英文摘要：
Multimodal Large Language Models (MLLMs) have garnered significant attention for their strong visual-semantic understanding. Most existing chart benchmarks evaluate MLLMs' ability to parse information from charts to answer questions. However, they overlook the inherent output biases of MLLMs, where models rely on their parametric memory to answer questions rather than genuinely understanding the chart content. To address this limitation, we introduce a novel Chart Hypothetical Question Answering (HQA) task, which imposes assumptions on the same question to compel models to engage in counterfactual reasoning based on the chart content. Furthermore, we introduce HAI, a human-AI interactive data synthesis approach that leverages the efficient text-editing capabilities of LLMs alongside human expert knowledge to generate diverse and high-quality HQA data at a low cost. Using HAI, we construct Chart-HQA, a challenging benchmark synthesized from publicly available data sources. Evaluation results on 18 MLLMs of varying model sizes reveal that current models face significant generalization challenges and exhibit imbalanced reasoning performance on the HQA task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
