使用MATH数据集测量数学问题解决能力
Measuring Mathematical Problem Solving With the MATH Dataset
摘要:
许多智力活动需要数学问题的解决,但这种技能仍然超出了计算机的能力。为了在机器学习模型中衡量这种能力,我们引入了MATH,这是一个包含12,500个具有挑战性的竞赛数学问题的新数据集。数学中的每个问题都有一个完整的分步解决方案,可以用来教模型生成答案推导和解释。为了促进未来的研究和提高数学的准确性,我们还提供了一个大型的辅助预训练数据集,帮助教授模型数学基础。即使我们能够提高数学的准确性,我们的结果表明,准确性仍然相对较低,即使有巨大的Transformer模型。此外,我们发现,简单地增加预算和模型参数计数将是不切实际的实现强大的数学推理,如果缩放趋势继续。虽然缩放Transformers可以自动解决大多数其他基于文本的任务,但缩放目前并不能解决数学问题。为了在数学问题解决方面有更多的牵引力,我们可能需要更广泛的研究社区的新算法进步。
英文摘要:
Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are able to increase accuracy on MATH, our results show that accuracy remains relatively low, even with enormous Transformer models. Moreover, we find that simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue. While scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH. To have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。