AdaSVD：用于大型语言模型的自适应奇异值分解
AdaSVD: Adaptive Singular Value Decomposition for Large Language Models
AI总结：
AdaSVD提出自适应SVD压缩方法，通过adaComp补偿截断误差和adaCR分配层特定压缩比，在多个LLM/VLM上超越SOTA，显著降低内存需求。
摘要：
大型语言模型（LLMs）在自然语言处理（NLP）任务中取得了显著成功，但其巨大的内存需求给在资源受限设备上的部署带来了重大挑战。奇异值分解（SVD）已成为一种有前景的LLM压缩技术，可大幅减少内存开销。然而，现有的基于SVD的方法往往难以有效缓解SVD截断引入的误差，导致与原始模型相比存在明显的性能差距。此外，对所有Transformer层应用统一的压缩比未能考虑不同层的重要性差异。为解决这些挑战，我们提出了AdaSVD，一种基于SVD的自适应LLM压缩方法。具体而言，AdaSVD引入了adaComp，通过交替更新奇异矩阵$\mathcal{U}$和$\mathcal{V}^\top$来自适应地补偿SVD截断误差。此外，AdaSVD引入了adaCR，根据每层的相对重要性自适应地分配层特定的压缩比。在多个LLM/VLM系列和评估指标上的大量实验表明，AdaSVD始终优于最先进的（SOTA）基于SVD的方法，在显著降低内存需求的同时实现了更优的性能。AdaSVD的代码和模型将在https://github.com/ZHITENGLI/AdaSVD上提供。
英文摘要：
Large language models (LLMs) have achieved remarkable success in natural language processing (NLP) tasks, yet their substantial memory requirements present significant challenges for deployment on resource-constrained devices. Singular Value Decomposition (SVD) has emerged as a promising compression technique for LLMs, offering considerable reductions in memory overhead. However, existing SVD-based methods often struggle to effectively mitigate the errors introduced by SVD truncation, leading to a noticeable performance gap when compared to the original models. Furthermore, applying a uniform compression ratio across all transformer layers fails to account for the varying importance of different layers. To address these challenges, we propose AdaSVD, an adaptive SVD-based LLM compression approach. Specifically, AdaSVD introduces adaComp, which adaptively compensates for SVD truncation errors by alternately updating the singular matrices $\mathcal{U}$ and $\mathcal{V}^\top$. Additionally, AdaSVD introduces adaCR, which adaptively assigns layer-specific compression ratios based on the relative importance of each layer. Extensive experiments across multiple LLM/VLM families and evaluation metrics demonstrate that AdaSVD consistently outperforms state-of-the-art (SOTA) SVD-based methods, achieving superior performance with significantly reduced memory requirements. Code and models of AdaSVD will be available at https://github.com/ZHITENGLI/AdaSVD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
