RNN体系结构的形式化层次
A Formal Hierarchy of RNN Architectures
摘要：
我们开发了RNN架构表达能力的正式层次结构。该层次结构基于两个形式属性：空间复杂度，它测量RNN的内存，以及有理递归，定义为递归更新是否可以由加权有限状态机描述。我们在这个层次结构中放置了几个RNN变体。例如，我们证明了LSTM不是理性的，这将其与相关的QRNN正式分离（Bradbury et al.，2016）。我们还展示了这些模型的表达能力是如何通过堆叠多个层或使用不同的池化函数组合来扩展的。我们的研究结果建立在“饱和”RNN理论的基础上（Merrill，2019）。虽然将这些发现正式扩展到不饱和RNN还有待于将来的工作，但我们假设不饱和RNN的实际可学习能力遵循类似的层次结构。在形式语言上训练非饱和网络的实验结果支持了这一猜想。
英文摘要：
We develop a formal hierarchy of the expressive capacity of RNN architectures. The hierarchy is based on two formal properties: space complexity, which measures the RNN's memory, and rational recurrence, defined as whether the recurrent update can be described by a weighted finite-state machine. We place several RNN variants within this hierarchy. For example, we prove the LSTM is not rational, which formally separates it from the related QRNN (Bradbury et al., 2016). We also show how these models' expressive capacity is expanded by stacking multiple layers or composing them with different pooling functions. Our results build on the theory of "saturated" RNNs (Merrill, 2019). While formally extending these findings to unsaturated RNNs is left to future work, we hypothesize that the practical learnable capacity of unsaturated RNNs obeys a similar hierarchy. Experimental findings from training unsaturated networks on formal languages support this conjecture.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
