前瞻性编码改善深度连续时间循环网络的学习效果
Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networks
查看机构详情
- Telepath
- New York University（纽约大学）
机构由 AI 辅助整理，请以论文原文为准。
浏览论文内容
中文总结 AI 辅助
该研究开发递归正交滤波器（RQFs）并提出前瞻性输入编码修正，缓解深度连续时间循环网络的梯度衰减，在语音命令等任务上取得优异性能，验证了其参数高效性。
中文摘要 AI 辅助
时间积分赋予连续时间循环网络记忆能力，但在深度堆叠的网络中，它同时会延迟自底向上信号并衰减自顶向下误差。我们开发了递归正交滤波器（Recursive Quadrature Filters, RQFs），这是一类受生物学启发的复值时间滤波器，属于对角状态空间模型（State-Space Models, SSMs）的特例，我们探究是否可通过使各层的自底向上输入具有前瞻性来解决该失效模式。从能量模型出发，我们推导了RQF的动力学特性，表明每个RQF是带通滤波器，其可学习参数控制调谐频率和带宽。随后，我们使用无参数的两抽头更新使各层的自底向上输入具有前瞻性，该更新不改变循环转移和平行扫描。我们将此修正扩展至一般对角SSMs，表明当截断时间梯度（即仅空间反向传播）时，它可缓解依赖深度的梯度衰减。我们在RQFs、S5和ORGaNICs（一种非线性门控循环神经网络）上评估该干预，这些模型使用全时间反向传播（Backpropagation Through Time, BPTT）和仅空间反向传播进行训练。在全BPTT下，前瞻性变体在所有模型和配置中均匹配或优于其非前瞻性对照。一个无残差、宽度为32的6层RQF在原始音频语音命令任务上达到96.09%的准确率，参数规模为3.19万；一个宽度为64的6层RQF在16384步的Path-X任务上达到83.56%。这些结果表明RQFs是一种参数高效的循环基底，而前瞻性输入编码是针对深度连续时间循环网络的输入侧修正方法。
英文摘要
Temporal integration gives continuous-time recurrent networks memory, but in deep stacks it also delays bottom-up signals and attenuates top-down errors. We develop Recursive Quadrature Filters (RQFs), biologically motivated complex-valued temporal filters that are a special case of diagonal state-space models (SSMs), and ask whether this failure mode can be addressed by making each layer's bottom-up input prospective. Starting from an energy model, we derive the RQF dynamics and show that each RQF is a band-pass filter whose learnable parameters control its tuning frequency and bandwidth. We then make each layer's bottom-up input prospective using a parameter-free two-tap update that leaves the recurrent transition and parallel scan unchanged. We extend this correction to general diagonal SSMs and show that it mitigates depth-dependent gradient attenuation when temporal gradients are truncated, i.e., spatial-only backpropagation. We evaluate the intervention in RQFs, S5, and ORGaNICs (a nonlinear gated RNN) trained using full backpropagation through time (BPTT) and spatial-only backpropagation. Under full BPTT, prospective variants match or outperform their non-prospective controls in every model and configuration. A non-residual width-32 six-layer RQF reaches 96.09% accuracy on raw-audio Speech Commands with 31.9k parameters; a width-64 six-layer RQF reaches 83.56% on the 16,384-step Path-X task. These results identify RQFs as a parameter-efficient recurrent substrate and prospective-input coding as an input-side correction for deep continuous-time recurrent networks.
