OlympicArena：面向超级智能AI的多学科认知推理基准测试
OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI
AI总结：
本文提出OlympicArena基准，包含11,163道双语多模态问题，覆盖7个领域62项奥赛，全面评估AI认知推理能力，发现GPT-4o总体准确率仅39.97%，揭示了当前模型在复杂推理与多模态整合上的局限。
摘要：
人工智能（AI）的发展因大型语言模型（LLMs）和大型多模态模型的进展而显著加速，逐渐在解决问题和科学发现（即AI4Science）中展现出曾经为人类智力所独有的认知推理能力。为了全面评估当前模型在认知推理能力方面的表现，我们引入了OlympicArena，其中包含11,163道双语问题，涵盖纯文本和图文交错两种模态。这些挑战涉及跨越7个领域和62项国际奥林匹克竞赛的广泛学科，并经过了严格的数据泄露检查。我们认为，奥林匹克竞赛问题中的挑战非常适合评估AI的认知推理能力，因为它们具有复杂性和跨学科性质，这对于应对复杂的科学挑战和促进发现至关重要。除了使用仅基于答案的标准评估各学科的表现外，我们还从多个视角进行了详细的实验和分析。我们深入探讨了模型的认知推理能力、在不同模态下的表现，以及它们在过程级评估中的结果，这对于需要冗长解答的复杂推理任务至关重要。我们的广泛评估表明，即使是像GPT-4o这样的先进模型也仅达到39.97%的总体准确率，这说明了当前AI在复杂推理和多模态整合方面的局限性。通过OlympicArena，我们旨在推动AI迈向超级智能，使其能够应对科学及其他领域中更复杂的挑战。我们还提供了一套全面的资源以支持AI研究，包括基准数据集、开源标注平台、详细的评估工具以及具备自动提交功能的排行榜。
英文摘要：
The evolution of Artificial Intelligence (AI) has been significantly accelerated by advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), gradually showcasing potential cognitive reasoning abilities in problem-solving and scientific discovery (i.e., AI4Science) once exclusive to human intellect. To comprehensively evaluate current models' performance in cognitive reasoning abilities, we introduce OlympicArena, which includes 11,163 bilingual problems across both text-only and interleaved text-image modalities. These challenges encompass a wide range of disciplines spanning seven fields and 62 international Olympic competitions, rigorously examined for data leakage. We argue that the challenges in Olympic competition problems are ideal for evaluating AI's cognitive reasoning due to their complexity and interdisciplinary nature, which are essential for tackling complex scientific challenges and facilitating discoveries. Beyond evaluating performance across various disciplines using answer-only criteria, we conduct detailed experiments and analyses from multiple perspectives. We delve into the models' cognitive reasoning abilities, their performance across different modalities, and their outcomes in process-level evaluations, which are vital for tasks requiring complex reasoning with lengthy solutions. Our extensive evaluations reveal that even advanced models like GPT-4o only achieve a 39.97% overall accuracy, illustrating current AI limitations in complex reasoning and multimodal integration. Through the OlympicArena, we aim to advance AI towards superintelligence, equipping it to address more complex challenges in science and beyond. We also provide a comprehensive set of resources to support AI research, including a benchmark dataset, an open-source annotation platform, a detailed evaluation tool, and a leaderboard with automatic submission features.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
