arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.18521cs.CLcs.CV
收藏

CharXiv:揭示多模态大语言模型在真实图表理解中的差距

CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

Princeton University(普林斯顿大学);University of Wisconsin, Madison(威斯康星大学麦迪逊分校);The University of Hong Kong(香港大学)

Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, Danqi Chen

AI总结:

提出CharXiv评估套件,含2,323个真实图表,揭示MLLMs图表理解差距,最强专有模型GPT-4o准确率47.1%,开源模型InternVL仅29.2%,远低于人类80.5%。

摘要:

图表理解在将多模态大语言模型(MLLMs)应用于分析科学论文或财务报告等现实任务时发挥着关键作用。然而,现有数据集通常侧重于基于模板问题的过度简化和同质化图表,导致对进展的评估过于乐观。我们证明,尽管开源模型在这些基准测试上可能表现优于强大的专有模型,但使用略有不同的图表或问题进行简单压力测试,性能可能下降高达34.5%。在这项工作中,我们提出了CharXiv,一个包含来自arXiv论文的2,323个自然、具有挑战性和多样化的图表的综合评估套件。CharXiv包括两类问题:1)关于检查基本图表元素的描述性问题,以及2)需要综合图表中复杂视觉元素信息的推理问题。为确保质量,所有图表和问题均由人类专家精心挑选、策划和验证。我们的结果揭示了最强专有模型(即GPT-4o)与最强开源模型(即InternVL Chat V1.5)之间推理能力的显著且此前被低估的差距,前者准确率达到47.1%,后者为29.2%。所有模型都远落后于人类80.5%的性能,凸显了现有MLLMs在图表理解能力上的弱点。我们希望CharXiv通过提供更真实和忠实的进展衡量标准,促进未来对MLLM图表理解的研究。项目页面和排行榜:https://charxiv.github.io/

英文摘要:

Chart understanding plays a pivotal role when applying Multimodal Large Language Models (MLLMs) to real-world tasks such as analyzing scientific papers or financial reports. However, existing datasets often focus on oversimplified and homogeneous charts with template-based questions, leading to an over-optimistic measure of progress. We demonstrate that although open-source models can appear to outperform strong proprietary models on these benchmarks, a simple stress test with slightly different charts or questions can deteriorate performance by up to 34.5%. In this work, we propose CharXiv, a comprehensive evaluation suite involving 2,323 natural, challenging, and diverse charts from arXiv papers. CharXiv includes two types of questions: 1) descriptive questions about examining basic chart elements and 2) reasoning questions that require synthesizing information across complex visual elements in the chart. To ensure quality, all charts and questions are handpicked, curated, and verified by human experts. Our results reveal a substantial, previously underestimated gap between the reasoning skills of the strongest proprietary model (i.e., GPT-4o), which achieves 47.1% accuracy, and the strongest open-source model (i.e., InternVL Chat V1.5), which achieves 29.2%. All models lag far behind human performance of 80.5%, underscoring weaknesses in the chart understanding capabilities of existing MLLMs. We hope CharXiv facilitates future research on MLLM chart understanding by providing a more realistic and faithful measure of progress. Project page and leaderboard: https://charxiv.github.io/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑