通过结构化广义线性令牌混合在表达性与复杂性之间进行权衡
Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing
AI总结：
本文提出一个统一框架，将令牌混合层分解为直接输入-输出影响和递归传播，通过设计结构化递归模式在运行时复杂度和表达性之间进行可证明的权衡，并在合成任务和语言建模上验证。
摘要：
令牌混合层在语言模型学习和生成长期依赖关系中起着关键作用。其效率依赖于解码速度与内存需求以及缓存大小之间的必要权衡。考虑因果生成，本文通过一个统一框架探索新的权衡，该框架分离了两个关键特征：(i) 在一个生成步骤中输入对输出的直接影响；(ii) 通过过去输出进行信息的递归传播。该框架涵盖了注意力机制和状态空间模型等主要架构，但也通过允许每个状态依赖于多个过去状态（而不仅仅是直接前驱）来推广递归方程。通过引入结构，我们设计了新的递归模式，这些模式可证明达到所需的复杂度，同时提供关于其表达性的理论见解——以原则性的方式用运行时换取表达性。在合成任务以及语言建模上进行了实证验证。这些结果共同提供了一个统一的工具包，用于理解和设计跨模型家族的高效且富有表达性的令牌混合器。
英文摘要：
Token mixing layers play a key role in how language models can learn and generate long-range dependencies. Their efficiency relies on the necessary trade-off between decoding speed and the memory requirements, along with the cache size. Considering causal generation, this paper explores new trade-offs thanks to a unified framework which separates two crucial features: (i) the direct influence of inputs on outputs in one generation step; (ii) the recurrent propagation of information through past outputs. This framework encompasses major architectures such as attention and state-space models, but also generalizes the recurrence equations by allowing each state to depend on multiple past states rather than only the immediate predecessor. By introducing structure, we design new recurrence patterns that provably achieve the desired complexity, while providing theoretical insights on their expressivity -- trading runtime for expressivity in a principled way. Empirical validation is performed on synthetic tasks, along with language modeling. Together, these results provide a unified toolkit for the understanding and design of efficient and expressive token mixers across model families.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
