arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.20311cs.AIcs.CLcs.LG
收藏

语言模型的物理学:第2.1部分,小学数学与隐藏的推理过程

Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process

CMU(卡内基梅隆大学);Meta FAIR;MBZUAI(穆罕默德·本·扎耶德人工智能大学)

Tian Ye, Zicheng Xu, Yuanzhi Li, Zeyuan Allen-Zhu

AI总结:

本文通过受控实验研究语言模型解决小学数学问题的隐藏推理机制,探讨其是否真正具备推理技能及与人类推理的异同,揭示了超越当前LLM理解的内部机制。

摘要:

语言模型的最新进展展示了其解决数学推理问题的能力,在GSM8K等小学水平数学基准上实现了近乎完美的准确率。本文正式研究语言模型如何解决这些问题。我们设计了一系列受控实验来解决几个基本问题:(1) 语言模型能否真正发展出推理技能,还是仅仅记忆模板?(2) 模型的隐藏(心理)推理过程是什么?(3) 模型解决数学问题所用的技能与人类相似还是不同?(4) 在类GSM8K数据集上训练的模型是否发展出了解决GSM8K问题所需之外的推理技能?(5) 什么心理过程导致模型出现推理错误?(6) 模型需要多大或多深才能有效解决GSM8K级别的数学问题?我们的研究揭示了语言模型解决数学问题的许多隐藏机制,提供了超越当前对LLMs理解的见解。

英文摘要:

Recent advances in language models have demonstrated their capability to solve mathematical reasoning problems, achieving near-perfect accuracy on grade-school level math benchmarks like GSM8K. In this paper, we formally study how language models solve these problems. We design a series of controlled experiments to address several fundamental questions: (1) Can language models truly develop reasoning skills, or do they simply memorize templates? (2) What is the model's hidden (mental) reasoning process? (3) Do models solve math questions using skills similar to or different from humans? (4) Do models trained on GSM8K-like datasets develop reasoning skills beyond those necessary for solving GSM8K problems? (5) What mental process causes models to make reasoning mistakes? (6) How large or deep must a model be to effectively solve GSM8K-level math questions? Our study uncovers many hidden mechanisms by which language models solve mathematical questions, providing insights that extend beyond current understandings of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑