NumGLUE：一套基本但复杂的数学推理任务
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks
摘要：
鉴于数字在文本中无处不在的性质，用数字进行推理来执行简单的计算是AI系统的一项重要技能。虽然为此开发了许多数据集和模型，但最先进的人工智能系统是脆弱的;当它们出现在稍微不同的场景中时，无法执行底层的数学推理。从在自然语言理解背景下提出的GLUE中汲取灵感，我们提出了NumGLUE，这是一个多任务基准，用于评估AI系统在八个不同任务上的性能，其核心要求简单的算术理解。我们发现，这个基准远远不能用神经模型来解决，包括最先进的大规模语言模型，它们的表现明显比人类差（低46.4%）。此外，NumGLUE促进了跨任务的知识共享，特别是那些训练数据有限的任务，当一个模型在所有任务上联合训练时，与特定于任务的建模相比，其优越的性能（每个任务的平均增益为3.4%）证明了这一点。最后，我们希望NumGLUE能够鼓励系统在语言中执行强大和通用的算术推理，这是朝着能够执行更复杂的数学推理迈出的第一步。
英文摘要：
Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems. While many datasets and models have been developed to this end, state-of-the-art AI systems are brittle; failing to perform the underlying mathematical reasoning when they appear in a slightly different scenario. Drawing inspiration from GLUE that was proposed in the context of natural language understanding, we propose NumGLUE, a multi-task benchmark that evaluates the performance of AI systems on eight different tasks, that at their core require simple arithmetic understanding. We show that this benchmark is far from being solved with neural models including state-of-the-art large-scale language models performing significantly worse than humans (lower by 46.4%). Further, NumGLUE promotes sharing knowledge across tasks, especially those with limited training data as evidenced by the superior performance (average gain of 3.4% on each task) when a model is jointly trained on all the tasks as opposed to task-specific modeling. Finally, we hope that NumGLUE will encourage systems that perform robust and general arithmetic reasoning within language, a first step towards being able to perform more complex mathematical reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
