PoPE：面向大语言模型的基于勒让德正交多项式的位置编码
PoPE: Legendre Orthogonal Polynomials Based Position Encoding for Large Language Models
AI总结：
针对正弦位置编码在高维表示上的局限，本文提出基于勒让德正交多项式的 PoPE，并在 Multi30k 翻译任务中验证其性能与收敛优势。
摘要：
原始 transformer 中使用的基线绝对位置编码（APE）方法已有若干改进。本研究旨在考察在更高维度上位置编码表示不足所带来的影响，这些影响源于正弦基函数的选择，涉及注意力机制的关键方面、模型学习相对位置信息的能力以及模型收敛性。通过理论洞见与实证分析相结合，我们阐明这些挑战如何超越 APE，并可能对相对位置编码（RPE）方法的性能产生不利影响，例如旋转位置编码（RoPE）。随后，我们提出一种名为基于正交多项式的位置编码（PoPE）的创新解决方案，以应对现有方法的部分局限。PoPE 方法通过利用正交勒让德多项式对位置信息进行编码。勒让德多项式作为基函数为位置编码提供了若干理想性质，包括改善的相关结构、非周期性、正交性以及不同阶多项式之间截然不同的函数形式。实验结果表明，采用 PoPE 的 transformer 模型在 Multi30k 英译德翻译任务上优于基线 transformer 模型，从而确立了新的性能基准。此外，基于 PoPE 的 transformer 表现出显著加快的收敛速度。另外，我们还将基于 PoPE 的优越性能提出关于位置编码的新颖理论观点。
英文摘要：
There are several improvements proposed over the baseline Absolute Positional Encoding (APE) method used in original transformer. In this study, we aim to investigate the implications of inadequately representing positional encoding in higher dimensions on crucial aspects of the attention mechanism, the model's capacity to learn relative positional information, and the convergence of models, all stemming from the choice of sinusoidal basis functions. Through a combination of theoretical insights and empirical analyses, we elucidate how these challenges extend beyond APEs and may adversely affect the performance of Relative Positional Encoding (RPE) methods, such as Rotatory Positional Encoding (RoPE). Subsequently, we introduce an innovative solution termed Orthogonal Polynomial Based Positional Encoding (PoPE) to address some of the limitations associated with existing methods. The PoPE method encodes positional information by leveraging Orthogonal Legendre polynomials. Legendre polynomials as basis functions offers several desirable properties for positional encoding, including improved correlation structure, non-periodicity, orthogonality, and distinct functional forms among polynomials of varying orders. Our experimental findings demonstrate that transformer models incorporating PoPE outperform baseline transformer models on the $Multi30k$ English-to-German translation task, thus establishing a new performance benchmark. Furthermore, PoPE-based transformers exhibit significantly accelerated convergence rates. Additionally, we will present novel theoretical perspectives on position encoding based on the superior performance of PoPE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
