arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.03089cs.LG
收藏

通过线性层组合的低秩诱导训练实现结构保持的网络压缩

Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Composition

Michigan State University(密歇根州立大学);University of Michigan - Ann Arbor(密歇根大学安娜堡分校)

Xitong Zhang, Ismail R. Alkhouri, Rongrong Wang

AI总结:

本文提出低秩诱导训练方法 LoRITa,通过线性层组合和奇异值截断实现结构保持的网络压缩,无需预训练初始化、预先指定秩或迭代计算 SVD,在多个模型和数据集上取得领先的 FLOPs 与参数压缩效果。

摘要:

深度神经网络(DNNs)在解决许多以往无法处理的任务方面取得了显著成功。然而,DNNs 带来的存储和计算需求对在资源受限设备上部署这些训练好的模型构成了挑战。因此,近年来提出了大量的压缩和剪枝技术。低秩分解技术是解决这一问题最常用的方法之一。与训练后压缩相比,压缩促进训练仍然探索不足。在本文中,我们提出了一种具有理论依据的技术,称为低秩诱导训练(LoRITa),该技术通过线性层的组合促进低秩性,并利用奇异值截断进行压缩。这一过程无需在推理时改变结构,也无需除标准权重衰减正则化之外的约束和/或额外优化。此外,LoRITa 消除了以下需求:(i)使用预训练模型进行初始化,(ii)在训练前指定秩选择,以及(iii)在每次迭代中计算 SVD。我们的实验结果(i)证明了该方法的有效性,使用 MNIST 在全连接网络上、CIFAR10 在 Vision Transformers 上以及 CIFAR10/100 和 ImageNet 在卷积神经网络上进行了验证,并且(ii)表明,与领先的结构化剪枝和低秩训练方法相比,我们在 FLOPs 和参数量下降方面取得了具有竞争力或最先进的结果。我们的代码可在 \url{https://github.com/XitongSystem/LoRITa/tree/main} 获取。

英文摘要:

Deep Neural Networks (DNNs) have achieved remarkable success in addressing many previously unsolvable tasks. However, the storage and computational requirements associated with DNNs pose a challenge for deploying these trained models on resource-limited devices. Therefore, a plethora of compression and pruning techniques have been proposed in recent years. Low-rank decomposition techniques are among the approaches most utilized to address this problem. Compared to post-training compression, compression-promoted training is still under-explored. In this paper, we present a theoretically-justified technique termed Low-Rank Induced Training (LoRITa), that promotes low-rankness through the composition of linear layers and compresses by using singular value truncation. This is achieved without the need to change the structure at inference time or require constrained and/or additional optimization, other than the standard weight decay regularization. Moreover, LoRITa eliminates the need to (i) initialize with pre-trained models, (ii) specify rank selection prior to training, and (iii) compute SVD in each iteration. Our experimental results (i) demonstrate the effectiveness of our approach using MNIST on Fully Connected Networks, CIFAR10 on Vision Transformers, and CIFAR10/100 and ImageNet on Convolutional Neural Networks, and (ii) illustrate that we achieve either competitive or state-of-the-art results when compared to leading structured pruning and low-rank training methods in terms of FLOPs and parameters drop. Our code is available at \url{https://github.com/XitongSystem/LoRITa/tree/main}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑