低位移秩压缩变换的学习
Learning Compressed Transforms with Low Displacement Rank
摘要:
结构矩阵的低位移秩(LDR)框架通过两个位移算子和一个低秩残差来表示矩阵。在深度学习中LDR矩阵的现有使用已经应用了固定位移算子,其编码类似于卷积的移位不变性的形式。我们引入了一类具有更一般位移算子的LDR矩阵,并明确地学习了算子和低秩分量。这个类在保持压缩和有效计算的同时推广了以前的几个构造。我们证明了多层神经网络的VC维的结构化的权重矩阵的界限,并表明经验,我们的紧凑的参数化可以减少样本的学习复杂性。当在图像分类和语言建模任务中替换全连接、卷积和递归神经网络中的权重层时,我们的新类超过了现有压缩方法的准确性,在某些任务中,性能也优于一般的非结构化层,同时使用的参数减少了20倍以上。
英文摘要:
The low displacement rank (LDR) framework for structured matrices represents a matrix through two displacement operators and a low-rank residual. Existing use of LDR matrices in deep learning has applied fixed displacement operators encoding forms of shift invariance akin to convolutions. We introduce a class of LDR matrices with more general displacement operators, and explicitly learn over both the operators and the low-rank component. This class generalizes several previous constructions while preserving compression and efficient computation. We prove bounds on the VC dimension of multi-layer neural networks with structured weight matrices and show empirically that our compact parameterization can reduce the sample complexity of learning. When replacing weight layers in fully-connected, convolutional, and recurrent neural networks for image classification and language modeling tasks, our new classes exceed the accuracy of existing compression approaches, and on some tasks also outperform general unstructured layers while using more than 20x fewer parameters.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。