arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.03051cs.CL
收藏

SAES-SVD: 自适应抑制累积和局部误差用于基于SVD的LLM压缩

SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compression

Houmo AI;Nanjing University(南京大学)

Xing Hu, Dawei Yang, Yuan Cheng, Zhixuan Chen, Zukang Xu

AI总结:

SAES-SVD通过联合优化层内重建和层间误差补偿,有效抑制LLM压缩中的累积和局部误差,提升压缩后性能。

摘要:

大语言模型(LLMs)参数规模的快速增长对高效的压缩技术提出了高度需求。作为一种硬件无关且高度兼容的技术,低秩压缩已被广泛采用。然而,现有方法通常通过最小化每层的重建误差独立地压缩每一层,忽略了关键限制:重建误差通过网络传播并累积,导致与全精度基线的全局偏差被放大。为了解决这一问题,我们提出了自适应误差抑制SVD(SAES-SVD),一种用于LLM压缩的框架,该框架联合优化层内重建和层间误差补偿。SAES-SVD由两个新颖的组件组成:(1)累积误差感知层压缩(CEALC),将压缩目标制定为局部重建和加权累积误差补偿的组合。基于此,我们推导出一个闭合形式的低秩解,该解依赖于二次激活统计,这明确地将每一层的输出与全精度对应物对齐,以补偿累积误差。(2)自适应协作误差抑制(ACES),它自动调整权重系数以增强CEALC中压缩目标的低秩结构。具体来说,系数被优化以在固定秩下最大化压缩层输出的Frobenius范数与压缩目标的Frobenius范数的比率,从而确保有效利用秩预算。在多种LLM架构和任务上的广泛实验表明,SAES-SVD在不进行微调或混合秩策略的情况下,始终提高了压缩后的性能。

英文摘要:

The rapid growth in the parameter scale of large language models (LLMs) has created a high demand for efficient compression techniques. As a hardware-agnostic and highly compatible technique, low-rank compression has been widely adopted. However, existing methods typically compress each layer independently by minimizing per-layer reconstruction error, overlooking a critical limitation: the reconstruction error propagates and accumulates through the network, which leads to amplified global deviations from the full-precision baseline. To address this, we propose Self-Adaptive Error Suppression SVD (SAES-SVD), a LLMs compression framework that jointly optimizes intra-layer reconstruction and inter-layer error compensation. SAES-SVD is composed of two novel components: (1) Cumulative Error-Aware Layer Compression (CEALC), which formulates the compression objective as a combination of local reconstruction and weighted cumulative error compensation. Based on it, we derive a closed-form low-rank solution relied on second-order activation statistics, which explicitly aligns each layer's output with its full-precision counterpart to compensate for accumulated errors. (2) Adaptive Collaborative Error Suppression (ACES), which automatically adjusts the weighting coefficient to enhance the low-rank structure of the compression objective in CEALC. Specifically, the coefficient is optimized to maximize the ratio between the Frobenius norm of the compressed layer's output and that of the compression objective under a fixed rank, thus ensuring that the rank budget is utilized effectively. Extensive experiments across multiple LLM architectures and tasks show that, without fine-tuning or mixed-rank strategies, SAES-SVD consistently improves post-compression performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑