arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02521cs.LGcs.AIeess.SP
收藏

缩放点积注意力实现了对输入向量在共同表面上的投影

Scaled Dot-Product Attention implements projection of inputs onto a common surface

Department of Electrical Engineering and Computer Science(电气工程与计算机科学系);University of California, Irvine(加州大学伊文斯顿分校)

Terence D Sanger

AI总结:

本文提出SDPA可重写为输入向量在共同表面的投影,揭示其在时间依赖性上下文中的作用,为非线性时间序列处理提供新视角。

摘要:

缩放点积注意力(SDPA)是大型语言模型和其他非线性信号处理应用成功的关键组件。SDPA的原理基于数据库理论中借鉴的“查询、键、值”概念,但这些概念难以与数学信号处理中的标准方法相协调。我们展示了SDPA可以以一种不同的但数学上等价的形式重写,作为输入向量在由输入本身决定的共同表面上的投影。因此,SDPA在输入中发现时间依赖性和上下文依赖性的非线性依赖关系。SDPA的重写形式允许前馈和学习算法的速度提升,但更重要的是提出了潜在的扩展。在语言的背景下,我们将SDPA的作用重新解释为发现由输入向量集所处的表面决定的时间依赖性上下文意义。输入标记嵌入随后由局部上下文表面进行修改。这种解释与“自注意力”的概念有显著差异,并为SDPA在具有时间变化的局部非线性依赖的时间序列数据中的应用提供了有力的正当性。

英文摘要:

Scaled dot-product attention (SDPA) is a fundamental component responsible for the success of large-language models and other nonlinear signal processing applications. The rationale for SDPA has been based upon "query, key, value" concepts borrowed from database theory, but these concepts are difficult to reconcile with standard methods in mathematical signal processing. We show that SDPA can be rewritten in a different but mathematically equivalent form as a projection of the input vectors onto a common surface determined by the inputs themselves. Therefore SDPA discovers nonlinear dependencies in the input that are time-dependent and context-dependent. The rewritten form of SDPA permits increased speed of both feedforward and learning algorithms, but more importantly suggests potential extensions. In the context of language, we re-interpret the role of SDPA as finding a time-dependent contextual meaning determined by the surface on which the set of input vectors lies. Input token embeddings are then modified by the local context surface. This interpretation differs substantially from the concept of "self-attention", and provides a strong justification for the use of SDPA for time-series data with time-varying local nonlinear dependencies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑