arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.09241cs.LGcs.CL
收藏

TinyGSM:用小型语言模型在GSM8K上实现80%以上的准确率

TinyGSM: achieving >80% on GSM8k with small language models

Carnegie Mellon University(卡内基梅隆大学);Microsoft Research(微软研究院)

Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, Yi Zhang

AI总结:

本研究针对小型语言模型数学推理能力不足的问题,构建了由GPT-3.5生成的含1230万道题的TinyGSM合成数据集,结合1.3B生成模型与验证器,在GSM8K基准上达到81.5%准确率,超越大模型并追平教师模型。

摘要:

小规模模型具备诸多计算优势,但规模对问题求解能力的关键影响程度仍属待解问题。具体到小学数学题求解领域,目前在GSM8K基准测试中突破80%准确率门槛所需的最小模型规模仍为34B。本研究探讨了高质量数据集如何成为小型语言模型习得数学推理能力的关键。我们推出了\texttt{TinyGSM},这是一个包含1230万道小学数学题及对应Python解答的合成数据集,完全由GPT-3.5生成。在\texttt{TinyGSM}上微调后,我们发现由1.3B生成模型和1.3B验证器模型组成的组合可达到81.5%的准确率,性能优于规模大出数个数量级的现有模型。这一表现也可与生成我们模型训练数据的GPT-3.5“教师”模型(77.4%)相媲美。我们的方法十分简洁,包含两个核心组成部分:1)高质量数据集\texttt{TinyGSM};2)验证器的使用,即从多个候选生成结果中选出最终输出。

英文摘要:

Small-scale models offer various computational advantages, and yet to which extent size is critical for problem-solving abilities remains an open question. Specifically for solving grade school math, the smallest model size so far required to break the 80\% barrier on the GSM8K benchmark remains to be 34B. Our work studies how high-quality datasets may be the key for small language models to acquire mathematical reasoning. We introduce \texttt{TinyGSM}, a synthetic dataset of 12.3M grade school math problems paired with Python solutions, generated fully by GPT-3.5. After finetuning on \texttt{TinyGSM}, we find that a duo of a 1.3B generation model and a 1.3B verifier model can achieve 81.5\% accuracy, outperforming existing models that are orders of magnitude larger. This also rivals the performance of the GPT-3.5 ``teacher'' model (77.4\%), from which our model's training data is generated. Our approach is simple and has two key components: 1) the high-quality dataset \texttt{TinyGSM}, 2) the use of a verifier, which selects the final outputs from multiple candidate generations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑