无限上下文Transformer在路径积分形式体系中的折叠上下文压缩
Folded Context Condensation in Path Integral Formalism for Infinite Context Transformers
AI总结：
本文在路径积分形式体系下重新解释Transformer核心机制，将注意力机制转化为转移路径积分并压缩上下文为记忆片段循环处理，实现线性内存扩展和长期信息保留。
摘要：
在这项工作中，我们通过在 Path Integral formalism（路径积分形式体系）框架内重新解释 Transformer 的核心机制，提出了 Transformer 算法的广义形式化。在此视角下，注意力机制被重新转化为一个整合所有导向未来 token 状态的可能转移路径的过程，其时间演化由 Feed-Forward Network（前馈网络）控制。通过将 Transformer 的每个组件系统性地映射到 Path Integral 形式中的对应部分，我们获得了一种更紧凑且高效的表示，其中序列的上下文信息被压缩成类似记忆的片段。这些片段在 Transformer 各层间被循环处理，从而实现更有效的长期信息保留。我们通过 Passkey retrieval 任务和一项摘要任务验证了该方法的有效性，证明所提方法在保留历史信息的同时，其内存使用量随序列长度呈线性扩展。这与标准注意力机制中通常观察到的非线性内存增长形成对比。我们期望 Transformer 架构的这种量子启发广义化将为提升未来 Transformer 模型的效率和表达力开辟新途径。
英文摘要：
In this work, we present a generalized formulation of the Transformer algorithm by reinterpreting its core mechanisms within the framework of Path Integral formalism. In this perspective, the attention mechanism is recast as a process that integrates all possible transition paths leading to future token states, with temporal evolution governed by the Feed-Forward Network. By systematically mapping each component of the Transformer to its counterpart in the Path Integral formulation, we obtain a more compact and efficient representation, in which the contextual information of a sequence is condensed into memory-like segments. These segments are recurrently processed across Transformer layers, enabling more effective long-term information retention. We validate the effectiveness of this approach through the Passkey retrieval task and a summarization task, demonstrating that the proposed method preserves historical information while exhibiting memory usage that scales linearly with sequence length. This contrasts with the non-linear memory growth typically observed in standard attention mechanisms. We expect that this quantum-inspired generalization of the Transformer architecture will open new avenues for enhancing both the efficiency and expressiveness of future Transformer models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
