arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.01609cs.CL
收藏

Swift-SVD:在低秩LLM压缩中理论最优与实用效率的结合

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

Nanyang Technological University(南洋理工大学)

Ruoling Qi, Yirui Liu, Xuaner Wu, Xiangyu Wang, Ming Li, Chen Chen, Jian Chen, Yin Chen, Qizhen Weng

AI总结:

本文提出Swift-SVD框架,通过激活感知和闭式压缩方法,在保证理论最优的同时提升实用效率和数值稳定性,实验显示其在压缩精度和端到端压缩时间上有显著优势。

摘要:

大型语言模型的部署受到静态权重和动态键值缓存的内存和带宽需求的限制。基于SVD的压缩提供了一种硬件友好的解决方案来降低这些成本。然而,现有方法存在两个关键限制:一些方法在重建误差上不最优,而另一些方法在理论上最优但实际效率低。本文提出Swift-SVD,一种激活感知的闭式压缩框架,同时保证理论最优、实用效率和数值稳定性。Swift-SVD在给定一批输入的情况下逐步聚合输出激活的协方差,并在聚合后执行一次特征值分解,实现免训练、快速且最优的层级低秩近似。我们采用有效秩分析局部层级压缩性,并设计一种动态秩分配策略,同时考虑局部重建损失和端到端层重要性。在六个LLM和八个数据集上的广泛实验表明,Swift-SVD优于现有最先进基线,实现最优压缩精度的同时,端到端压缩时间提升了3-70倍。我们的代码可在https://github.com/hiahei/Swift-SVD获取。

英文摘要:

The deployment of Large Language Models is constrained by the memory and bandwidth demands of static weights and dynamic Key-Value cache. SVD-based compression provides a hardware-friendly solution to reduce these costs. However, existing methods suffer from two key limitations: some are suboptimal in reconstruction error, while others are theoretically optimal but practically inefficient. In this paper, we propose Swift-SVD, an activation-aware, closed-form compression framework that simultaneously guarantees theoretical optimum, practical efficiency and numerical stability. Swift-SVD incrementally aggregates covariance of output activations given a batch of inputs and performs a single eigenvalue decomposition after aggregation, enabling training-free, fast, and optimal layer-wise low-rank approximation. We employ effective rank to analyze local layer-wise compressibility and design a dynamic rank allocation strategy that jointly accounts for local reconstruction loss and end-to-end layer importance. Extensive experiments across six LLMs and eight datasets demonstrate that Swift-SVD outperforms state-of-the-art baselines, achieving optimal compression accuracy while delivering 3-70X speedups in end-to-end compression time. Our code is available at https://github.com/hiahei/Swift-SVD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑