arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.15580cs.LGcs.CL
收藏

语言模型在算术中是符号学习者

Language Models are Symbolic Learners in Arithmetic

Department of Computer Science(计算机科学系);Rice University(里士满大学);College of Computing(计算学院);Georgia Institute of Technology(佐治亚理工学院);Duke University(杜克大学)

Chunyuan Deng, Zhiqi Li, Roy Xie, Ruidi Chang, Hanjie Chen

AI总结:

本文研究语言模型在算术运算中通过学习符号捷径而非算法来掌握算术能力。

摘要:

语言模型在进行算术运算时,普遍存在的问题是这些模型是否真正计算还是仅仅掌握表面的模式匹配。在本文中,我们主张后者,提出了证据表明语言模型充当贪婪的符号学习者,优先选择最简单的可能捷径以适应数据集的统计特性来解决算术任务。为了研究这一点,我们引入了子组归纳,这是从Solomonoff归纳(SI)中衍生出的实用框架,SI是目前最强大的通用预测器之一。我们的框架通过将算术问题分解为子组来分析,即在少数输入数字和单个输出数字之间进行最小映射。我们的主要指标,子组质量,衡量这些捷径的可行性。实验揭示了多数字乘法中出现明显的U型准确性模式:语言模型迅速掌握第一个和最后一个输出数字,而在中间数字上则遇到困难。我们证明这种U型并非偶然;它完美地映射了最简单的可能子组的质量,即需要最少输入标记的子组。这种对齐表明了一种核心的学习机制:语言模型首先学习容易的、低标记的捷径,只有在训练过程中才引入更复杂的、多标记的模式。它们并不学习乘法的算法,而是学习一层次的越来越复杂的符号到符号的映射。最终,我们的发现表明,语言模型在算术掌握的道路上并非由算法铺就,而是由一系列简单、层次化学习的符号捷径构成。

英文摘要:

The prevailing question in LM performing arithmetic is whether these models learn to truly compute or if they simply master superficial pattern matching. In this paper, we argues for the latter, presenting evidence that LMs act as greedy symbolic learners, prioritizing the simplest possible shortcuts to fit the stats of dataset to solve arithmetic tasks. To investigate this, we introduce subgroup induction, a practical framework adapted from Solomonoff Induction (SI), one of the most powerful universal predictors. Our framework analyzes arithmetic problems by breaking them down into subgroups-minimal mappings between a few input digits and a single output digit. Our primary metric, subgroup quality, measures the viability of these shortcuts. Experiments reveal a distinct U-shaped accuracy pattern in multi-digit multiplication: LMs quickly master the first and last output digits while struggling with those in the middle. We demonstrate this U-shape is not coincidental; it perfectly mirrors the quality of the simplest possible subgroups, those requiring the fewest input tokens. This alignment suggests a core learning mechanism: LMs first learn easy, low-token shortcuts and only incorporate more complex, multi-token patterns as training progresses. They do not learn the algorithm of multiplication but rather a hierarchy of increasingly complex symbol-to-symbol mappings. Ultimately, our findings suggest that the path to arithmetic mastery for LMs is not paved with algorithms, but with a cascade of simple, hierarchically-learned symbolic shortcuts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑