解构循环、注意力与门控：探究Transformer与门控循环神经网络在动力系统预测中的可迁移性
Deconstructing Recurrence, Attention, and Gating: Investigating the transferability of Transformers and Gated Recurrent Neural Networks in forecasting of dynamical systems
AI总结：
本文通过消融研究解构RNN的门控与循环及Transformer的注意力机制，构建混合架构，发现门控和注意力普遍提升RNN性能，而循环对Transformer有害，并提出结合循环高速网络与门控注意力的新架构在高维时空预测中表现最佳。
摘要：
机器学习架构，包括Transformer和循环神经网络（RNN），已经彻底变革了从文本处理到极端天气等应用领域的预测。值得注意的是，为自然语言处理等应用调优的先进网络架构，可以迁移到其他任务，例如时空预测任务。然而，目前缺乏消融研究来说明实现这种预测精度的关键组成部分。这类研究的缺失虽然可以归因于相关的计算成本，但加剧了这些模型应被视为黑箱的观念。在这项工作中，我们分解了最强大神经架构的关键架构组件，即RNN中的门控和循环，以及Transformer中的注意力机制。然后，我们从标准模块中合成并构建了新颖的混合架构，执行消融研究以确定哪些机制对每个任务有效。将这些组件视为可以增强标准架构的超参数的重要性，在各种预测数据集上得到了展示，这些数据集包括多尺度Lorenz 96系统的时空混沌动力学、Kuramoto-Sivashinsky方程，以及标准的真实世界时间序列基准。一个关键发现是，神经门控和注意力在大多数任务中提高了所有标准RNN的性能，而在Transformer中添加循环概念则是有害的。此外，我们的研究揭示了一种新颖的、稀疏使用的架构，该架构将循环高速网络（Recurrent Highway Networks）与神经门控和注意力机制相结合，在动力系统的高维时空预测中成为表现最佳的架构。
英文摘要：
Machine learning architectures, including transformers and recurrent neural networks (RNNs) have revolutionized forecasting in applications ranging from text processing to extreme weather. Notably, advanced network architectures, tuned for applications such as natural language processing, are transferable to other tasks such as spatiotemporal forecasting tasks. However, there is a scarcity of ablation studies to illustrate the key components that enable this forecasting accuracy. The absence of such studies, although explainable due to the associated computational cost, intensifies the belief that these models ought to be considered as black boxes. In this work, we decompose the key architectural components of the most powerful neural architectures, namely gating and recurrence in RNNs, and attention mechanisms in transformers. Then, we synthesize and build novel hybrid architectures from the standard blocks, performing ablation studies to identify which mechanisms are effective for each task. The importance of considering these components as hyper-parameters that can augment the standard architectures is exhibited on various forecasting datasets, from the spatiotemporal chaotic dynamics of the multiscale Lorenz 96 system, the Kuramoto-Sivashinsky equation, as well as standard real world time-series benchmarks. A key finding is that neural gating and attention improves the performance of all standard RNNs in most tasks, while the addition of a notion of recurrence in transformers is detrimental. Furthermore, our study reveals that a novel, sparsely used, architecture which integrates Recurrent Highway Networks with neural gating and attention mechanisms, emerges as the best performing architecture in high-dimensional spatiotemporal forecasting of dynamical systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
