ConceptMath：一个用于衡量大语言模型数学推理能力的双语概念粒度基准
ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models
AI总结：
本文提出双语概念粒度基准 ConceptMath，揭示大语言模型在不同数学概念上的性能差异，并提出高效微调策略以增强薄弱环节。
摘要：
本文介绍了 ConceptMath，一个双语（英文和中文）、细粒度的基准，用于评估大语言模型（LLMs）的概念级数学推理能力。与以平均准确率评估一般数学推理的传统基准不同，ConceptMath 将数学问题系统地组织在一个数学概念层级之下，从而能够以概念级准确率在不同粒度上评估数学推理。基于我们的 ConceptMath，我们评估了广泛的 LLMs，并观察到现有 LLMs 虽然在传统基准上取得了较高的平均准确率，但在不同数学概念上表现出显著的性能差异，甚至在最基本的概念上可能出现灾难性失败。此外，我们还提出了一种高效的微调策略，以增强现有 LLMs 的薄弱环节。最后，我们希望 ConceptMath 能够指导开发者理解其模型的细粒度数学能力，并促进基础模型的发展。
英文摘要：
This paper introduces ConceptMath, a bilingual (English and Chinese), fine-grained benchmark that evaluates concept-wise mathematical reasoning of Large Language Models (LLMs). Unlike traditional benchmarks that evaluate general mathematical reasoning with an average accuracy, ConceptMath systematically organizes math problems under a hierarchy of math concepts, so that mathematical reasoning can be evaluated at different granularity with concept-wise accuracies. Based on our ConcepthMath, we evaluate a broad range of LLMs, and we observe existing LLMs, though achieving high average accuracies on traditional benchmarks, exhibit significant performance variations across different math concepts and may even fail catastrophically on the most basic ones. Besides, we also introduce an efficient fine-tuning strategy to enhance the weaknesses of existing LLMs. Finally, we hope ConceptMath could guide the developers to understand the fine-grained mathematical abilities of their models and facilitate the growth of foundation models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
