arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.03445cs.LGcs.CL
收藏

预训练大型语言模型使用傅里叶特征来计算加法

Pre-trained Large Language Models Use Fourier Features to Compute Addition

University of Southern California(南加州大学)

Tianyi Zhou, Deqing Fu, Vatsal Sharan, Robin Jia

AI总结:

本文揭示预训练LLM利用傅里叶特征计算加法,MLP用低频特征近似数值大小,注意力层用高频特征执行模加法,证明预训练表示能解锁Transformer学习精确算法机制的能力。

摘要:

预训练大型语言模型(LLMs)展现出令人印象深刻的数学推理能力,然而它们如何计算诸如加法等基本算术运算仍不清楚。本文表明,预训练LLMs使用傅里叶特征——即隐藏状态中通过一组在频域内稀疏的特征来表示数字的维度——来进行数字加法。在模型内部,MLP和注意力层以互补的方式使用傅里叶特征:MLP层主要利用低频特征来近似答案的数值大小,而注意力层主要利用高频特征来执行模加法(例如,计算答案是奇数还是偶数)。预训练对于这一机制至关重要:从零开始训练以进行数字加法的模型仅利用低频特征,导致准确率较低。将预训练的词嵌入引入到随机初始化的模型中可以恢复其性能。总体而言,我们的分析表明,合适的预训练表示(如傅里叶特征)能够解锁Transformer学习算法任务精确机制的能力。

英文摘要:

Pre-trained large language models (LLMs) exhibit impressive mathematical reasoning capabilities, yet how they compute basic arithmetic, such as addition, remains unclear. This paper shows that pre-trained LLMs add numbers using Fourier features -- dimensions in the hidden state that represent numbers via a set of features sparse in the frequency domain. Within the model, MLP and attention layers use Fourier features in complementary ways: MLP layers primarily approximate the magnitude of the answer using low-frequency features, while attention layers primarily perform modular addition (e.g., computing whether the answer is even or odd) using high-frequency features. Pre-training is crucial for this mechanism: models trained from scratch to add numbers only exploit low-frequency features, leading to lower accuracy. Introducing pre-trained token embeddings to a randomly initialized model rescues its performance. Overall, our analysis demonstrates that appropriate pre-trained representations (e.g., Fourier features) can unlock the ability of Transformers to learn precise mechanisms for algorithmic tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑