基于快速乘法的Kronecker CP分解压缩RNN
Kronecker CP Decomposition with Fast Multiplication for Compressing RNNs
摘要：
递归神经网络（RNN）在面向序列数据的任务中具有强大的功能，例如自然语言处理和视频识别。然而，由于现代RNN，包括长短期记忆（LSTM）和门控递归单元（GRU）网络，具有复杂的拓扑结构和昂贵的空间/计算复杂度，压缩它们成为近年来的热门和有前途的话题。在大量的压缩方法中，张量分解，例如，张量训练（TT）、块项（BT）、张量环（TR）和分层塔克（HT），似乎是最令人惊奇的方法，因为可以获得非常高的压缩比。然而，这些张量分解格式中没有一种能够同时提供空间和计算效率。在本文中，我们考虑压缩RNN基于一种新的Kronecker CANDECOMP/PARAFAC（KCP）分解，这是来自Kronecker张量（KT）分解，通过提出两个快速算法之间的乘法输入和张量分解的权重。根据我们基于UCF 11、Youtube Celebrities Face和UCF 50数据集的实验，可以验证所提出的KCP-RNN具有与其他张量分解格式相当的准确性性能，甚至可以获得278、219倍的压缩比通过低排名KCP。更重要的是，KCP-RNN是有效的空间和计算复杂度相比，其他张量分解的类似的秩。此外，我们发现KCP具有最大的潜力，并行计算，以加速神经网络的计算。
英文摘要：
Recurrent neural networks (RNNs) are powerful in the tasks oriented to sequential data, such as natural language processing and video recognition. However, since the modern RNNs, including long-short term memory (LSTM) and gated recurrent unit (GRU) networks, have complex topologies and expensive space/computation complexity, compressing them becomes a hot and promising topic in recent years. Among plenty of compression methods, tensor decomposition, e.g., tensor train (TT), block term (BT), tensor ring (TR) and hierarchical Tucker (HT), appears to be the most amazing approach since a very high compression ratio might be obtained. Nevertheless, none of these tensor decomposition formats can provide both the space and computation efficiency. In this paper, we consider to compress RNNs based on a novel Kronecker CANDECOMP/PARAFAC (KCP) decomposition, which is derived from Kronecker tensor (KT) decomposition, by proposing two fast algorithms of multiplication between the input and the tensor-decomposed weight. According to our experiments based on UCF11, Youtube Celebrities Face and UCF50 datasets, it can be verified that the proposed KCP-RNNs have comparable performance of accuracy with those in other tensor-decomposed formats, and even 278,219x compression ratio could be obtained by the low rank KCP. More importantly, KCP-RNNs are efficient in both space and computation complexity compared with other tensor-decomposed ones under similar ranks. Besides, we find KCP has the best potential for parallel computing to accelerate the calculations in neural networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
