DeepMath-Creative：一个评估大型语言模型数学创造力的基准
DeepMath-Creative: A Benchmark for Evaluating Mathematical Creativity of Large Language Models
AI总结：
本文提出数学创造力评估标准与DeepMath-Creative基准，系统评测主流LLM，发现最强模型仅在基础构造题上达70%，复杂开放题表现薄弱。
摘要：
为提升大型语言模型（LLM）的数学能力，DeepMath团队发起了一项开源计划，旨在开发一个开放的数学LLM并系统评估其数学创造力。本文是该计划的首个贡献。尽管数学LLM的近期发展主要强调推理技能，从初等至本科水平数学任务的基准中可见一斑，但这些模型的创造能力受到的关注相对较少，评估数据集仍然稀缺。为弥补这一差距，我们提出了一套数学创造力评估标准，并推出DeepMath-Creative，这是一个新颖、高质量的基准，包含横跨代数、几何、分析及其他领域的构造性问题。我们利用该数据集对主流LLM的创造性解题能力进行了系统评估。实验结果表明，即使在宽松的评分标准下——强调核心解答组成部分并忽略细微不准确之处，例如小的逻辑缺口、不完整的论证或冗余解释——表现最佳的模型O3 Mini也仅达到70%的准确率，且主要集中在基础本科水平的构造性任务上。在更复杂的问题上，性能急剧下降，各模型无法为开放性问题提供实质性策略。这些发现表明，尽管当前LLM在熟悉且难度较低的问题上表现出一定程度的构造能力，但这种表现很可能源于对记忆模式的重组，而非真正的创造性洞见或新颖综合。
英文摘要：
To advance the mathematical proficiency of large language models (LLMs), the DeepMath team has launched an open-source initiative aimed at developing an open mathematical LLM and systematically evaluating its mathematical creativity. This paper represents the initial contribution of this initiative. While recent developments in mathematical LLMs have predominantly emphasized reasoning skills, as evidenced by benchmarks on elementary to undergraduate-level mathematical tasks, the creative capabilities of these models have received comparatively little attention, and evaluation datasets remain scarce. To address this gap, we propose an evaluation criteria for mathematical creativity and introduce DeepMath-Creative, a novel, high-quality benchmark comprising constructive problems across algebra, geometry, analysis, and other domains. We conduct a systematic evaluation of mainstream LLMs' creative problem-solving abilities using this dataset. Experimental results show that even under lenient scoring criteria -- emphasizing core solution components and disregarding minor inaccuracies, such as small logical gaps, incomplete justifications, or redundant explanations -- the best-performing model, O3 Mini, achieves merely 70% accuracy, primarily on basic undergraduate-level constructive tasks. Performance declines sharply on more complex problems, with models failing to provide substantive strategies for open problems. These findings suggest that, although current LLMs display a degree of constructive proficiency on familiar and lower-difficulty problems, such performance is likely attributable to the recombination of memorized patterns rather than authentic creative insight or novel synthesis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
