OlympiadBench:一个通过奥赛级双语多模态科学问题促进AGI的挑战性基准
OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
AI总结:
本文提出奥赛级双语多模态科学基准OlympiadBench,含8,476道数学物理竞赛题及专家级推理标注,评估显示GPT-4V平均得分仅17.97%,揭示现有模型在复杂科学推理上的不足。
摘要:
近期进展显示,大语言模型(LLMs)和大规模多模态模型(LMMs)已在多种任务上超越普通人类能力,接近多个领域人类专家的熟练水平。随着传统基准对这些模型变得不再具有挑战性,新的严格挑战对于衡量其高级能力至关重要。在本工作中,我们提出了OlympiadBench,一个奥赛级双语多模态科学基准,包含来自奥赛级数学和物理竞赛(包括中国高考)的8,476道题目。每道题都配有专家级标注,用于逐步推理。我们在OlympiadBench上评估顶级模型,实施了一套综合评估方法以准确评价模型回答。值得注意的是,表现最佳的模型GPT-4V在OlympiadBench上仅获得17.97%的平均得分,其中物理仅为10.74%,凸显了该基准的严格性和物理推理的复杂性。我们对GPT-4V的分析指出了普遍存在的幻觉、知识遗漏和逻辑谬误问题。我们希望这一具有挑战性的基准能够成为助力未来AGI研究工作的宝贵资源。数据和评估代码可在\url{https://github.com/OpenBMB/OlympiadBench}获取。
英文摘要:
Recent advancements have seen Large Language Models (LLMs) and Large Multimodal Models (LMMs) surpassing general human capabilities in various tasks, approaching the proficiency level of human experts across multiple domains. With traditional benchmarks becoming less challenging for these models, new rigorous challenges are essential to gauge their advanced abilities. In this work, we present OlympiadBench, an Olympiad-level bilingual multimodal scientific benchmark, featuring 8,476 problems from Olympiad-level mathematics and physics competitions, including the Chinese college entrance exam. Each problem is detailed with expert-level annotations for step-by-step reasoning. Evaluating top-tier models on OlympiadBench, we implement a comprehensive assessment methodology to accurately evaluate model responses. Notably, the best-performing model, GPT-4V, attains an average score of 17.97% on OlympiadBench, with a mere 10.74% in physics, highlighting the benchmark rigor and the intricacy of physical reasoning. Our analysis orienting GPT-4V points out prevalent issues with hallucinations, knowledge omissions, and logical fallacies. We hope that our challenging benchmark can serve as a valuable resource for helping future AGI research endeavors. The data and evaluation code are available at \url{https://github.com/OpenBMB/OlympiadBench}
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。