回声状态变换器：对有限记忆的注意力
Echo State Transformer: Attention Over Finite Memories
AI总结：
回声状态变换器通过结合Transformer注意力机制与共振计算节点，实现线性复杂度的高效时间序列处理，优于现有方法。
摘要：
尽管大型语言模型及其底层Transformer架构非常高效，但它们并未反映我们的大脑如何处理和学习语言等多样化认知任务，也未体现如何利用工作记忆。此外，Transformer面临计算限制：序列长度的二次复杂度增长。受这些限制的启发，我们旨在设计能够利用高效工作记忆动态以克服标准计算障碍的架构。我们引入了回声状态变换器（EST），一种混合架构，它解决了这一挑战，并在分类和检测任务中展示了最先进的性能。EST将Transformer的注意力机制与共振计算的节点相结合，创建了一个固定大小的记忆系统。受回声状态网络的启发，我们的方法利用几个共振器（随机递归网络）并行作为轻量且高效的内存。这些独立单元具有不同的和学习的内部动态，具有自适应的泄漏率，使它们能够动态调整自身的时间性。通过在这些固定数量的单元上应用注意力，而不是在输入标记上，EST实现了整个序列的线性复杂度，有效地解决了标准Transformer的二次扩展问题。我们评估了EST在最近的时间序列基准上的表现：时间序列库，其中包括五个类别中的69个任务。结果表明，EST在五个类别中的两个类别中总体排名第一，优于强大的最先进基线，在分类和异常检测任务中表现优异，同时在短期预测中保持竞争力。这些结果表明，通过将注意力机制从整个输入序列转移到固定的一组动态记忆单元，可以保持对时间事件的高敏感性，同时实现每步的常数计算复杂度。
英文摘要：
While Large Language Models and their underlying Transformer architecture are remarkably efficient, they do not reflect how our brain processes and learns a diversity of cognitive tasks such as language, nor how it leverages working memory. Furthermore, Transformers encounters a computational limitation: quadratic complexity growth with sequence length. Motivated by these limitations, we aim to design architectures that leverage efficient working memory dynamics to overcome standard computational barriers. We introduce Echo State Transformers (EST), a hybrid architecture that resolves this challenge while demonstrating state of the art performance in classification and detection tasks. EST integrates the Transformer attention mechanisms with nodes from Reservoir Computing to create a fixed-size memory system. Drawing inspiration from Echo State Networks, our approach leverages several reservoirs (random recurrent networks) in parallel as a lightweight and efficient working memory. These independent units possess distinct and learned internal dynamics with an adaptive leak rate, enabling them to dynamically adjust their own temporality. By applying attention on those fixed number of units instead of input tokens, EST achieves linear complexity for the whole sequence, effectively breaking the quadratic scaling problem of standard Transformers. We evaluate ESTs on a recent timeseries benchmark: the Time Series Library, which comprises 69 tasks across five categories. Results show that ESTs ranks first overall in two of five categories, outperforming strong state-of-the-art baselines on classification and anomaly detection tasks, while remaining competitive on short-term forecasting. These results demonstrate that by shifting the attention mechanism from the entire input sequence to a fixed set of evolving memory units, it is possible to maintains high sensitivity to temporal events while achieving constant computational complexity per step.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
