The ability of large language models (LLMs) to interpret visual representations of data is crucial for advancing their application in data analysis and decision-making processes. This paper presents a novel synthetic dataset designed to evaluate the proficiency of LLMs in interpreting various forms of data visualizations, including plots like time series, histograms, violins, boxplots, and clusters. Our dataset is generated using controlled parameters to ensure comprehensive coverage of potential real-world scenarios. We employ multimodal text prompts with questions related to visual data in images to benchmark several state-of-the-art models like ChatGPT or Gemini, assessing their understanding and interpretative accuracy.
To ensure data integrity, our benchmark dataset is generated automatically, making it entirely new and free from prior exposure to the models being tested. This strategy allows us to evaluate the models' ability to truly interpret and understand the data, eliminating possibility of pre-learned responses, and allowing for an unbiased evaluation of the models' capabilities. We also introduce quantitative metrics to assess the performance of the models, providing a robust and comprehensive evaluation tool.
Benchmarking several state-of-the-art LLMs with this dataset reveals varying degrees of success, highlighting specific strengths and weaknesses in interpreting diverse types of visual data. The results provide valuable insights into the current capabilities of LLMs and identify key areas for improvement. This work establishes a foundational benchmark for future research and development aimed at enhancing the visual interpretative abilities of language models. In the future, improved LLMs with robust visual interpretation skills can significantly aid in automated data analysis, scientific research, educational tools, and business intelligence applications.
RephQA: Evaluating Readability of Large Language Models in Public Health Question Answering
Yale University, Department of Computer Science(耶鲁大学计算机科学系);Yale University, School of Nursing(耶鲁大学护理学院);Yale University, School of Public Health(耶鲁大学公共卫生学院);Northeastern University(东北大学)
OPOR-Bench: Evaluating Large Language Models on Online Public Opinion Report Generation
Communication University of China(中国传媒大学);Harbin Institute of Technology(哈尔滨工业大学);China Academy of Railway Sciences Corporation Limited(中国铁道科学研究院集团有限公司);School of Engineering, Santa Clara University(圣克拉拉大学工程学院)
AutoForest: Automatically Generating Forest Plots from Biomedical Studies with End-to-End Evidence Extraction and Synthesis
IBM Research(IBM研究院);Dublin City University(都柏林城市大学);UCL(伦敦大学学院);University of Oxford(牛津大学);IT:U Interdisciplinary Transformation University Austria(奥地利 interdisciplinary Transformation 大学)
Evaluating Large Language Model Creativity from a Literary Perspective
Imperial College London(帝国理工学院);Google DeepMind(谷歌DeepMind);Institute of Historical Research, School of Advanced Study, University of London(伦敦大学高级研究院历史研究所);University of Southampton(南安普顿大学)
Plot-guided Adversarial Example Construction for Evaluating Open-domain Story Generation
University of Southern California(南加州大学);Information Sciences Institute(信息科学研究所);University of California, Los Angeles(加州大学洛杉矶分校);Indian Institute of Technology Roorkee(印度理工学院鲁尔基分校)
MapIQ: Evaluating Multimodal Large Language Models for Map Question Answering
School of Computing and Augmented Intelligence(计算与增强智能学院);Arizona State University(亚利桑那州立大学);Department of Computer Science(计算机科学系);International Institute of Information Technology(国际信息科技研究所)
Data Science Approaches to Evaluating Honours Candidates
Ministry of Justice(司法部);Joint Counter-Terrorism Prisons and Probation Hub(联合反恐监狱与缓刑中心);Arndit Ltd.(阿恩迪特有限公司);Indian Institute of Technology Delhi(印度德里理工学院);Kainos(凯诺斯公司)