发表机构
York University; Vector Institute; Connected Minds（约克大学; 向量研究所; Connected Minds）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
提出ChartBias基准，首个系统审计视觉语言模型图表解读偏见，覆盖六类社会属性，发现叙述偏移、群体幻觉和偏好极性三种失败模式，并提出多智能体缓解框架以提升公平性与一致性。
AI 中文摘要
视觉语言模型（VLMs）越来越多地被用于解读图表，并为具有社会影响的数据生成自然语言解释。然而，当仅改变所提及的社会群体时，它们可能对同一图表产生不同的叙述，从而强化刻板印象并误导决策。尽管存在这些风险，目前尚无基准测试能够系统地评估跨社会维度的图表解读偏见。我们引入了ChartBias，这是首个用于审计基于VLM的图表解读偏见的基准测试。ChartBias包含820个手工筛选的真实世界图表，涵盖六个属性：种族、收入、年龄、宗教、移民身份和性别，产生4,319个有效的图表-属性实例和8,638对配对生成，其中图表固定，仅交换群体术语。在12个专有和开源VLM上，总计155,484个模型响应中，我们发现了三种普遍的失败模式：叙述偏移（同一图表，不同叙述）、群体幻觉（将图表分配给无证据的群体）和偏好极性（有利趋势常与某一群体关联）。我们进一步提出了一个多智能体缓解框架，通过将基于图表的证据提取与群体条件生成分离，并使用反事实判断器验证群体驱动的差异是否得到图表支持，从而作为强基线。该框架大幅减少了叙述偏移，同时保留了基于图表的推理。我们的研究结果表明，评估图表理解不仅需要衡量准确性，还需要衡量跨社会群体的公平性和一致性。我们在以下网址发布ChartBias：此https URL。
英文摘要
Vision-language models (VLMs) are increasingly used to interpret charts and generate natural-language explanations for socially consequential data. However, they may produce different narratives for the same chart when only the referenced social group changes, reinforcing stereotypes and misleading decisions. Despite these risks, no benchmark exists for systematically evaluating bias in chart interpretation across social dimensions. We introduce ChartBias, the first benchmark for auditing bias in VLM-based chart interpretation. ChartBias contains 820 manually curated real-world charts spanning six attributes: race, income, age, religion, immigration status, and gender, yielding 4,319 valid chart, attribute instances and 8,638 paired generations where the chart is fixed and only the group term is swapped. Across 12 proprietary and open-source VLMs, totaling 155,484 model responses, we find three widespread failure modes: narrative shift (same chart, different narratives), group hallucination (assigning a chart to a group without evidence), and preference polarity (favourable trends often linked to one group). We further propose a multi-agent mitigation framework that serves as a strong baseline by separating chart-grounded evidence extraction from group-conditioned generation and using a counterfactual judge to verify that group-driven differences are supported by the chart. The framework substantially reduces narrative shift while preserving chart-grounded reasoning. Our findings show that evaluating chart understanding requires measuring not only accuracy, but also fairness and consistency across social groups. We release ChartBias at https://github.com/vis-nlp/ChartBiasBench.
