发表机构
International Centre for Neuromorphic Systems; The University of Manchester（国际神经形态系统中心; 曼彻斯特大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
ADPTNet通过结合线性注意力与黎曼优化的局部拓扑共轭，实现数据自适应、长程依赖、GPU并行及非线性递归，提升序列建模性能并引入高效并行扩展。
AI 中文摘要
神经形态计算的一个核心目标是提供一种可行的替代方案，以取代高度耗能的基于Transformer的人工智能。然而，高效的替代方案难以捕捉到使Transformer成为序列建模事实标准的一系列特性。任何现实的竞争者必须具有数据自适应性、能够捕捉长程依赖、并且可在GPU上并行化，同时还需具备非线性递归性以实现复杂推理。基于听觉皮层以固定时间尺度运作的证据，本工作提出自适应与规定时间尺度网络（ADPTNet）作为同时实现上述四种特性的潜在解决方案。ADPTNet围绕局部拓扑共轭构建，这些共轭通过线性注意力与黎曼优化的新颖组合获得，并应用于静态全局动力学。这实现了非线性但可预测的长期行为。动力系统理论证明为ADPTNet时间尺度（其李雅普诺夫谱）的参数化控制提供了理论保证。ADPTNet在选择性复制任务上相较于现有平衡长程记忆与适应性的方法Hawk提升了性能，同时在状态跟踪上优于如Mamba等线性SSM。在顺序CIFAR-10上，ADPTNet匹配了线性SSM的准确率，并以更少的参数超越了现有选择性模型（包括Transformer）。我们还引入了一种神经形态的SpikingADPTNet，在Spiking Speech Commands数据集上取得了新的最先进准确率（83.56%±0.15）。最后，ADPTNet的恒定时间尺度使得DEER并行模拟算法的两种高效、无雅可比扩展（Conv和Forward DEER）成为可能，且保持相同的平均收敛性。Conv DEER除了网络的前向传播外不增加任何计算开销，并首次通过迭代卷积实现了非线性RNN的并行化。
英文摘要
A central aim of neuromorphic computing is to provide a viable alternative to highly energy-intensive Transformer-based AI. However, efficient alternatives struggle to capture the set of qualities that have secured the Transformer's status as the de facto standard in sequence modelling. Any realistic contender must be data-adaptive, able to capture long-range dependencies, and GPU-parallelisable, but also non-linearly recurrent to enable complex reasoning. Based on evidence suggesting the auditory cortex operates on fixed timescales, this work proposes the ADaptive with Prescriptive Timescales Network (ADPTNet) as a potential solution to achieving all four properties simultaneously. ADPTNet is built around local topological conjugates, obtained by a novel combination of linear attention and Riemannian optimisation, applied to static global dynamics. This enables non-linear yet predictable long-term behaviour. Dynamical systems theory proofs provide theoretical guarantees for the parametric control of ADPTNet's timescales (its Lyapunov spectrum). ADPTNet improves performance on Selective Copying over Hawk, the existing method balancing long-range memory and adaptability, while also improving state tracking over linear SSMs like Mamba. On sequential CIFAR-10, ADPTNet matches linear SSM accuracy and outperforms existing selective models (incl. the Transformer), using fewer parameters. We also introduce a neuromorphic SpikingADPTNet, which achieves a new state-of-the-art accuracy on the Spiking Speech Commands dataset ($83.56\%\pm0.15$). Finally, ADPTNet's constant timescales enable two efficient, Jacobian-free extensions to the DEER parallel simulation algorithm (Conv and Forward DEER) that retain the same average convergence. Conv DEER adds no computational overhead beyond the network's forward pass and enables non-linear RNN parallelisation via iterated convolutions for the first time.
Comments80 pages
