Second-order Recurrent Neural Networks (2RNNs) extend RNNs by leveraging second-order interactions for sequence modelling. These models are provably more expressive than their first-order counterparts and have connections to well-studied models from formal language theory. However, their large parameter tensor makes computations intractable. To circumvent this issue, one approach known as MIRNN consists in limiting the type of interactions used by the model. Another is to leverage tensor decomposition to diminish the parameter count. In this work, we study the model resulting from parameterizing 2RNNs using the CP decomposition, which we call CPRNN. Intuitively, the rank of the decomposition should reduce expressivity. We analyze how rank and hidden size affect model capacity and show the relationships between RNNs, 2RNNs, MIRNNs, and CPRNNs based on these parameters. We support these results empirically with experiments on the Penn Treebank dataset which demonstrate that, with a fixed parameter budget, CPRNNs outperforms RNNs, 2RNNs, and MIRNNs with the right choice of rank and hidden size.
Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
SMILE Lab, University of Electronic Science and Technology of China（电子科技大学SMILE实验室）；Dept. of Computer Science, Brown University（布朗大学计算机科学系）；School of Computing, University of Utah（犹他大学计算机学院）；Xjera Labs, Pte.Ltd（Xjera实验室私人有限公司）
Echoes of the past: A unified perspective on fading memory and echo states
Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore（数学科学系，物理与数学科学学院，南洋理工大学，新加坡）；Nanyang Technological University, Singapore（南洋理工大学，新加坡）
A Decomposition Perspective to Long-context Reasoning for LLMs
Large Language Model Department, Tencent（腾讯大语言模型部门）；Xi'an Jiaotong University（西安交通大学）；Tencent（腾讯）；Fudan University（复旦大学）；City University of Hong Kong（香港城市大学）
