U-MATH：一个大学级评估LLM数学能力的基准测试
U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs
AI总结：
U-MATH是一个大学级数学能力评估基准，包含1100个开放性问题，用于评估LLM在多模态推理和解决方案判断方面的性能。
摘要：
当前对LLM数学能力的评估存在局限，现有基准测试要么规模较小，主要聚焦于初等和高中问题，要么在主题多样性上不足。此外，任务中视觉元素的引入仍鲜有探索。为解决这些问题，我们引入U-MATH，一个包含1100个未发表的开放性大学级问题的新型基准测试，问题来源于教学材料。它在六个核心科目上保持平衡，其中20%为多模态问题。鉴于U-MATH问题的开放性，我们采用LLM来判断生成解决方案的正确性。为此，我们发布了μ-MATH数据集，用于评估LLM在判断解决方案能力方面的表现。对领先LLM进行基准测试显示，多模态推理存在明显局限，最大准确率在文本任务上达到93.1%，但在视觉任务上仅为58.5%。此外，解决方案判断极具挑战性，即使最先进模型也需达到显著高水平性能，即使最先进模型仍只能达到不完美的F1分数90.1%。
英文摘要：
The current evaluation of mathematical skills in LLMs is limited, as existing benchmarks are either relatively small, primarily focus on elementary and high-school problems, or lack diversity in topics. Additionally, the inclusion of visual elements in tasks remains largely under-explored. To address these gaps, we introduce U-MATH, a novel benchmark of 1,100 unpublished open-ended university-level problems sourced from teaching materials. It is balanced across six core subjects, with 20% of multimodal problems. Given the open-ended nature of U-MATH problems, we employ an LLM to judge the correctness of generated solutions. To this end, we release $μ$-MATH, a dataset to evaluate the LLMs' capabilities in judging solutions. Benchmarking leading LLMs reveals marked limitations in multi-modal reasoning, with maximum accuracy reaching 93.1\% on textual tasks but only 58.5\% on visual ones. Furthermore, solution judgment proves challenging, requiring the most advanced models to achieve meaningfully high performance, even still peaking at an imperfect F1-score of 90.1\%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
