arXiv 2405.070012024-11-06cs.CLcs.AIcs.CV

ChartInsights：评估多模态大语言模型在低级图表问答中的表现

ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering

The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；South China University of Technology（华南理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Renmin University of China（中国人民大学）

Yifan Wu, Lutao Yan, Leixian Shen, Yunhai Wang, Nan Tang, Yuyu Luo

备注：: EMNLP 2024 Conference Paper

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出ChartInsights数据集，系统评估19个多模态大模型在低级图表问答中的表现，并设计Chain-of-Charts提示策略，将准确率提升至84.32%。

摘要：

图表问答（ChartQA）任务在解释和提取可视化图表中的见解方面发挥着关键作用。尽管近期多模态大语言模型（MLLMs）如GPT-4o在高级ChartQA任务（如图表描述）中展现出潜力，但它们在低级ChartQA任务（例如识别相关性）中的有效性仍未得到充分探索。在本文中，我们通过使用新构建的数据集ChartInsights来评估MLLMs在低级ChartQA上的表现，填补了这一空白。该数据集包含22,347个（图表，任务，查询，答案）四元组，覆盖7种图表类型中的10个数据分析任务。我们系统评估了19个先进的MLLMs，包括12个开源模型和7个闭源模型。这些模型的平均准确率为39.8%，其中GPT-4o以69.17%的准确率最高。为了进一步探索MLLMs在低级ChartQA中的局限性，我们进行了改变图表视觉元素（例如更改配色方案、添加图像噪声）的实验，以评估它们对任务有效性的影响。此外，我们提出了一种新的文本提示策略Chain-of-Charts，专门针对低级ChartQA任务，该策略将性能提升了14.41%，达到83.58%的准确率。最后，结合一种引导注意力到相关视觉元素的视觉提示策略，进一步将准确率提升至84.32%。

英文摘要：

Chart question answering (ChartQA) tasks play a critical role in interpreting and extracting insights from visualization charts. While recent advancements in multimodal large language models (MLLMs) like GPT-4o have shown promise in high-level ChartQA tasks, such as chart captioning, their effectiveness in low-level ChartQA tasks (e.g., identifying correlations) remains underexplored. In this paper, we address this gap by evaluating MLLMs on low-level ChartQA using a newly curated dataset, ChartInsights, which consists of 22,347 (chart, task, query, answer) covering 10 data analysis tasks across 7 chart types. We systematically evaluate 19 advanced MLLMs, including 12 open-source and 7 closed-source models. The average accuracy rate across these models is 39.8%, with GPT-4o achieving the highest accuracy at 69.17%. To further explore the limitations of MLLMs in low-level ChartQA, we conduct experiments that alter visual elements of charts (e.g., changing color schemes, adding image noise) to assess their impact on the task effectiveness. Furthermore, we propose a new textual prompt strategy, Chain-of-Charts, tailored for low-level ChartQA tasks, which boosts performance by 14.41%, achieving an accuracy of 83.58%. Finally, incorporating a visual prompt strategy that directs attention to relevant visual elements further improves accuracy to 84.32%.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。