三元奇异值分解作为线性映射中更优的参数化形式
Ternary Singular Value Decomposition as a Better Parameterized Form in Linear Mapping
AI总结:
本文提出三元奇异值分解 TSVD,通过将 SVD 的 U、V 约束为 {±1,0} 矩阵,以加法替代乘法,并给出直接与训练转换算法,在多类网络和大模型上实现先进压缩性能。
摘要:
我们提出了一种简单而新颖的线性映射参数化形式,可实现显著的网络压缩性能:一种称为三元奇异值分解(Ternary SVD,TSVD)的伪 SVD。与普通 SVD 不同,TSVD 将 SVD 中的 U 和 V 矩阵限制为取值于 {±1, 0} 的三元矩阵形式。这意味着在计算 U(·) 和 V(·) 时,TSVD 不需要使用昂贵的乘法指令,而只需要加法指令。我们分别为 TSVD 提供了直接转换和训练转换算法,类似于训练后量化(Post Training Quantization)和量化感知训练(Quantization Aware Training)。此外,我们还从理论上分析了直接转换算法的收敛性。在实验中,我们证明 TSVD 能够在多种类型的网络和任务中达到最先进的网络压缩性能,包括 ConvNext、Swim、BERT 等当前基线模型,以及 OPT 等大型语言模型。
英文摘要:
We present a simple yet novel parameterized form of linear mapping to achieves remarkable network compression performance: a pseudo SVD called Ternary SVD (TSVD). Unlike vanilla SVD, TSVD limits the $U$ and $V$ matrices in SVD to ternary matrices form in $\{\pm 1, 0\}$. This means that instead of using the expensive multiplication instructions, TSVD only requires addition instructions when computing $U(\cdot)$ and $V(\cdot)$. We provide direct and training transition algorithms for TSVD like Post Training Quantization and Quantization Aware Training respectively. Additionally, we analyze the convergence of the direct transition algorithms in theory. In experiments, we demonstrate that TSVD can achieve state-of-the-art network compression performance in various types of networks and tasks, including current baseline models such as ConvNext, Swim, BERT, and large language model like OPT.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。