arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.02617cs.CL
收藏

PUB:用于评估大型语言模型对合成视觉数据解读能力的图表理解基准与数据集

PUB: Plot Understanding Benchmark and Dataset for Evaluating Large Language Models on Synthetic Visual Data Interpretation

Ideas NCBR;Lodz University of Technology(罗兹理工大学);University of Warsaw(华沙大学)

Aneta Pawelec, Victoria Sara Wesołowska, Zuzanna Bączek, Piotr Sankowski

AI总结:

本文提出合成视觉数据解读基准PUB,通过自动生成的多种图表数据集和多模态提示评估ChatGPT、Gemini等LLMs的图表理解能力,揭示其优势与不足,为提升语言模型视觉解读能力提供基础性评估工具。

摘要:

大型语言模型(LLMs)解读数据视觉表示的能力对于推进其在数据分析和决策过程中的应用至关重要。本文提出了一个新的合成数据集,旨在评估LLMs解读各种形式数据可视化的熟练程度,包括时间序列图、直方图、小提琴图、箱线图和聚类图等图表。我们的数据集使用受控参数生成,以确保全面覆盖潜在的真实世界场景。我们采用与图像中视觉数据相关的问题构成多模态文本提示,对ChatGPT或Gemini等多个最先进模型进行基准测试,评估它们的理解能力和解读准确性。为确保数据完整性,我们的基准数据集是自动生成的,使其完全新颖且未被测试模型预先接触过。这一策略使我们能够评估模型真正解读和理解数据的能力,消除了预先学习响应的可能性,并允许对模型能力进行无偏评估。我们还引入了定量指标来评估模型的性能,提供了一个稳健且全面的评估工具。使用该数据集对多个最先进的LLMs进行基准测试揭示了不同程度的成功,突出了在解读不同类型视觉数据方面的具体优势和不足。结果为了解LLMs当前能力提供了有价值的见解,并确定了需要改进的关键领域。这项工作为未来旨在增强语言模型视觉解读能力的研究和开发建立了基础性基准。未来,具有稳健视觉解读能力的改进型LLMs可以显著助力自动化数据分析、科学研究、教育工具和商业智能应用。

英文摘要:

The ability of large language models (LLMs) to interpret visual representations of data is crucial for advancing their application in data analysis and decision-making processes. This paper presents a novel synthetic dataset designed to evaluate the proficiency of LLMs in interpreting various forms of data visualizations, including plots like time series, histograms, violins, boxplots, and clusters. Our dataset is generated using controlled parameters to ensure comprehensive coverage of potential real-world scenarios. We employ multimodal text prompts with questions related to visual data in images to benchmark several state-of-the-art models like ChatGPT or Gemini, assessing their understanding and interpretative accuracy. To ensure data integrity, our benchmark dataset is generated automatically, making it entirely new and free from prior exposure to the models being tested. This strategy allows us to evaluate the models' ability to truly interpret and understand the data, eliminating possibility of pre-learned responses, and allowing for an unbiased evaluation of the models' capabilities. We also introduce quantitative metrics to assess the performance of the models, providing a robust and comprehensive evaluation tool. Benchmarking several state-of-the-art LLMs with this dataset reveals varying degrees of success, highlighting specific strengths and weaknesses in interpreting diverse types of visual data. The results provide valuable insights into the current capabilities of LLMs and identify key areas for improvement. This work establishes a foundational benchmark for future research and development aimed at enhancing the visual interpretative abilities of language models. In the future, improved LLMs with robust visual interpretation skills can significantly aid in automated data analysis, scientific research, educational tools, and business intelligence applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑