在标准化可视化素养测试上对视觉语言模型进行基准测试
Benchmarking Visual Language Models on Standardized Visualization Literacy Tests
AI总结：
本研究首次用VLAT和CALVI标准化测试系统评估四款主流VLM的可视化素养，发现模型基础图表解读能力尚可但识别误导性元素、高密度可视化及异常检测能力薄弱，且不同模型不确定性管理行为有差异，为VLM可视化任务优化提供基准与方向。
摘要：
视觉语言模型（Visual Language Models, VLMs）在可视化系统中的集成度日益提升，这要求我们全面了解其视觉解读能力与局限性。现有研究虽已对单个模型展开考察，但尚未有针对VLMs可视化素养的系统性比较研究。我们通过使用标准化评估工具——可视化素养评估测试（Visualization Literacy Assessment Test, VLAT）与可视化素养批判性思维评估（Critical Thinking Assessment for Literacy in Visualizations, CALVI），对四款主流VLM（GPT-4、Claude、Gemini和Llama）开展了首次严格评估，填补了这一空白。我们的方法创新性地将随机试验与结构化提示技术相结合，以控制顺序效应和响应变异性——这是许多VLM评估中忽略的关键考量因素。分析结果显示，尽管部分模型在基础图表解读方面表现出色（Claude在VLAT上的准确率达67.9%），但所有模型在识别误导性可视化元素方面都存在显著困难（在CALVI上的最高准确率仅为30.0%）。我们发现了不同的性能模式：模型在解读折线图等常规图表（准确率76%-96%）和识别层级结构（准确率80%-100%）方面能力较强，但在涉及多种编码的高密度数据可视化（气泡图：18.6%-61.4%）和异常检测（准确率25%-30%）方面始终存在困难。值得注意的是，我们观察到不同模型的不确定性管理行为存在明显差异，Gemini表现出更高的谨慎性（22.5%的问题弃权（不执行）），而其他模型的弃权率为7%-8%。这些发现为可视化领域提供了关键见解，建立了可靠的VLM评估基准，明确了当前模型的不足，并凸显了针对可视化任务优化VLM架构的必要性。
英文摘要：
The increasing integration of Visual Language Models (VLMs) into visualization systems demands a comprehensive understanding of their visual interpretation capabilities and constraints. While existing research has examined individual models, systematic comparisons of VLMs' visualization literacy remain unexplored. We bridge this gap through a rigorous, first-of-its-kind evaluation of four leading VLMs (GPT-4, Claude, Gemini, and Llama) using standardized assessments: the Visualization Literacy Assessment Test (VLAT) and Critical Thinking Assessment for Literacy in Visualizations (CALVI). Our methodology uniquely combines randomized trials with structured prompting techniques to control for order effects and response variability - a critical consideration overlooked in many VLM evaluations. Our analysis reveals that while specific models demonstrate competence in basic chart interpretation (Claude achieving 67.9% accuracy on VLAT), all models exhibit substantial difficulties in identifying misleading visualization elements (maximum 30.0\% accuracy on CALVI). We uncover distinct performance patterns: strong capabilities in interpreting conventional charts like line charts (76-96% accuracy) and detecting hierarchical structures (80-100% accuracy), but consistent difficulties with data-dense visualizations involving multiple encodings (bubble charts: 18.6-61.4%) and anomaly detection (25-30% accuracy). Significantly, we observe distinct uncertainty management behavior across models, with Gemini displaying heightened caution (22.5% question omission) compared to others (7-8%). These findings provide crucial insights for the visualization community by establishing reliable VLM evaluation benchmarks, identifying areas where current models fall short, and highlighting the need for targeted improvements in VLM architectures for visualization tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
