arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.00873cs.AIcs.CLcs.LG
收藏

语言模型使用三角学进行加法运算

Language Models Use Trigonometry to Do Addition

Massachusetts Institute of Technology(麻省理工学院)

Subhash Kantamneni, Max Tegmark

AI总结:

本文逆向工程了三个中等规模LLM的加法计算机制,发现数字被表示为广义螺旋线,并提出LLM通过“Clock”算法操纵该螺旋线来完成加法,首次在表示层面解释了LLM的数学能力。

摘要:

数学推理是大语言模型(LLM)能力的一个日益重要的指标,然而我们缺乏对LLM如何处理甚至简单数学任务的理解。为了解决这个问题,我们逆向工程了三个中等规模的LLM如何计算加法。我们首先发现,数字在这些LLM中被表示为一条广义螺旋线,这对于加法和减法任务具有强烈的因果牵连,并且对整数除法、乘法和模运算也具有因果相关性。然后,我们提出LLM通过使用“Clock”算法操纵这条广义螺旋线来计算加法:为了求解 $a+b$,$a$ 和 $b$ 的螺旋线被操纵以产生 $a+b$ 的答案螺旋线,然后将其读出为模型logits。我们用这些螺旋线对有影响力的MLP输出、注意力头输出甚至单个神经元预激活进行建模,并通过因果干预验证了我们的理解。通过证明LLM在螺旋线上表示数字并操纵该螺旋线来执行加法,我们提出了对LLM数学能力的首个表示层面的解释。

英文摘要:

Mathematical reasoning is an increasingly important indicator of large language model (LLM) capabilities, yet we lack understanding of how LLMs process even simple mathematical tasks. To address this, we reverse engineer how three mid-sized LLMs compute addition. We first discover that numbers are represented in these LLMs as a generalized helix, which is strongly causally implicated for the tasks of addition and subtraction, and is also causally relevant for integer division, multiplication, and modular arithmetic. We then propose that LLMs compute addition by manipulating this generalized helix using the "Clock" algorithm: to solve $a+b$, the helices for $a$ and $b$ are manipulated to produce the $a+b$ answer helix which is then read out to model logits. We model influential MLP outputs, attention head outputs, and even individual neuron preactivations with these helices and verify our understanding with causal interventions. By demonstrating that LLMs represent numbers on a helix and manipulate this helix to perform addition, we present the first representation-level explanation of an LLM's mathematical capability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑