ResidualTransformer：面向Transformer层的残差低秩学习与权重共享
ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers
AI总结：
针对常驻设备内存受限问题，提出ResidualTransformer方法，通过在Transformer层间共享满秩权重并引入独有低秩残差分量与对角矩阵，在1万小时语音任务上将编码器大小缩减约3倍且性能微降。
摘要：
在常驻设备上部署语音处理模型时，其内存限制是主要关注点之一。虽然用充足数据训练的较大模型通常表现更好，但使其适应设备内存是一项艰巨挑战。本文旨在通过重新参数化Transformer编码器层间的模型权重，并采用特殊的权重组合与结构来减小模型大小。具体而言，受ResNet及近期LoRA工作的启发，我们提出一种名为ResidualTransformer的方法，其中Transformer层中的每个权重矩阵包含：1）与相邻层共享的满秩分量；2）自身独有的低秩分量。这些低秩矩阵仅导致模型大小的小幅增加。此外，我们加入对角权重矩阵以提升低秩矩阵的建模能力。在1万小时语音识别和语音翻译任务上的实验表明，Transformer编码器大小可缩减约3倍，且性能下降极其轻微。
英文摘要：
Memory constraint of always-on devices is one of the major concerns when deploying speech processing models on these devices. While larger models trained with sufficiently large amount of data generally perform better, making them fit in the device memory is a demanding challenge. In this paper, we aim to reduce model size by reparameterizing model weights across Transformer encoder layers and assuming a special weight composition and structure. More specifically, inspired by ResNet and the more recent LoRA work, we propose an approach named ResidualTransformer, where each weight matrix in a Transformer layer comprises 1) a shared full-rank component with its adjacent layers, and 2) a unique low-rank component to itself. The low-rank matrices only account for a small amount of model size increase. In addition, we add diagonal weight matrices to improve modeling capacity of the low-rank matrices. Experiments of our 10k-hour speech recognition and speech translation tasks show that the Transformer encoder size can be reduced by ~3X with very slight performance degradation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
