arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.09741cs.CLcs.LG
收藏

FoNE: 通过傅里叶特征实现精确的单token数字嵌入

FoNE: Precise Single-Token Number Embeddings via Fourier Features

Department of Computer Science, University of Southern California(南加州大学计算机科学系)

Tianyi Zhou, Deqing Fu, Mahdi Soltanolkotabi, Robin Jia, Vatsal Sharan

AI总结:

FoNE通过傅里叶特征直接将数字映射到嵌入空间,以单token和两维嵌入维度表示数字,提升训练和推理效率,同时在加减乘等数值任务中实现更高准确率。

摘要:

大型语言模型(LLMs)通常使用多个token表示数字,需要模型聚合这些token以解释数值,这种碎片化导致训练和推理效率低下,并影响数字相关任务的性能。受预训练LLMs内部学习数字token的傅里叶特征的启发,我们提出傅里叶数字嵌入(FoNE),一种新颖的方法,直接将数字映射到嵌入空间中,利用傅里叶特征。FoNE将每个数字编码为单个token,每个数字仅需两个嵌入维度,有效捕捉数值而不产生碎片。这种紧凑的表示加速了训练和推理。与传统的子词和逐位嵌入相比,FoNE不仅减少了计算开销,还在各种数值任务中实现了更高的准确性。在6位十进制加法中,FoNE在达到99%准确率时,所需数据仅为子词和逐位嵌入的64倍,分别使用3倍和6倍更少的token per number。此外,FoNE是唯一在超过100,000个测试示例上实现加法、减法和乘法100%准确率的方法。代码和可视化可在https://fouriernumber.github.io/上获得。

英文摘要:

Large Language Models (LLMs) typically represent numbers using multiple tokens, which requires the model to aggregate these tokens to interpret numerical values. This fragmentation makes both training and inference less efficient and adversely affects the model's performance on number-related tasks. Inspired by the observation that pre-trained LLMs internally learn Fourier-like features for number tokens, we propose Fourier Number Embedding (FoNE), a novel method that directly maps numbers into the embedding space with their Fourier features. FoNE encodes each number as a single token with only two embedding dimensions per digit, effectively capturing numerical values without fragmentation. This compact representation accelerates both training and inference. Compared to traditional subword and digit-wise embeddings, FoNE not only reduces computational overhead but also achieves higher accuracy across various numerical tasks including addition, subtraction and multiplication. On 6-digit decimal addition, FoNE requires 64$\times$ less data to achieve 99% accuracy than subword and digit-wise embeddings while using 3$\times$ and 6$\times$ fewer tokens per number, respectively. Furthermore, FoNE is the only method that yields 100% accuracy on over 100,000 test examples for addition, subtraction, and multiplication. The codes and visualization are available at https://fouriernumber.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑