ChartArena: 跨语言、场景和格式的图表解析基准测试
ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats
AI总结:
提出ChartArena,一个覆盖8种图表族、3种视觉场景的双语基准,通过人机协作标注和格式无关评估协议,系统评估26个多模态大模型的图表解析能力,揭示前沿模型差距与挑战。
摘要:
图表是传达定量和关系信息的主要媒介,但系统地评估图表解析模型仍然困难。现有基准专注于狭窄的图表类型,而流程图和思维导图等图表结构在很大程度上未被涉及,同时模型输出格式不兼容,数据集也极少包含实践中遇到的打印或手绘图像。为解决这些问题,我们引入了ChartArena,一个全面的双语基准,涵盖8种图表族,包括数值图表和图表结构,每种图表在三种视觉场景中评估:数字渲染、打印照片和手绘照片。该数据集通过人机协作标注流程构建,并经过多阶段人工验证以确保标注可靠性。为实现公平的跨模型比较,我们进一步设计了一种格式无关的评估协议,将异构输出映射到两个规范语义空间:归一化三元组视图和有向图视图,并使用结构感知指标进行评分。通过对26个领先的多模态大语言模型的广泛评估,我们观察到三个一致的发现:(i) 前沿专有模型(如Gemini 3.1 Pro)总体领先,但最强的开源系统正在迅速缩小差距;(ii) 文档解析模型能较好地处理数值图表,但在图表结构上表现大幅落后;(iii) 专家图表解析器仍局限于狭窄的图表族。在所有模型中,雷达图和手绘场景尤其具有挑战性。这些发现表明,ChartArena揭示了清晰的能力差距,并为未来的进展提供了统一基础。ChartArena公开在 https://github.com/pspdada/ChartArena。
英文摘要:
Charts are a primary medium for conveying quantitative and relational information, yet systematically evaluating chart parsing models remains difficult. Existing benchmarks focus on narrow chart types and leave diagrammatic structures such as flowcharts and mind maps largely unaddressed, while models produce outputs in incompatible formats, and datasets rarely include the printed or hand-drawn images encountered in practice. To address these issues, we introduce ChartArena, a comprehensive bilingual benchmark covering eight chart families spanning both numeric charts and diagrammatic structures, each evaluated across three visual scenarios: digital renderings, printed photos, and hand-drawn photos. The dataset is built via a human-agent collaborative annotation pipeline with multi-stage human verification to ensure annotation reliability. To enable fair cross-model comparison, we further design a format-agnostic evaluation protocol that maps heterogeneous outputs into two canonical semantic spaces, a normalized triple view and a directed graph view, and scores them with structure-aware metrics. Through extensive evaluation of 26 leading MLLMs, we observe three consistent findings: (i) frontier proprietary models such as Gemini 3.1 Pro lead overall, yet the strongest open-source systems are rapidly closing the gap; (ii) document parsing models handle numeric charts reasonably but fall sharply behind on diagrammatic structures; and (iii) expert chart parsers remain limited to narrow chart families. Across all models, radar charts and hand-drawn scenarios stay especially challenging. These findings show that ChartArena exposes clear capability gaps and provides a unified foundation for future progress. ChartArena is publicly available at https://github.com/pspdada/ChartArena.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。