突破压缩瓶颈：从理论到实践
Break Through the Compression Bottleneck: From Theory to Practice
浏览论文内容
中文总结 AI 辅助
研究语言模型压缩瓶颈问题，通过数学证明和实验验证低秩分解与量化非正交，组合会致性能降，进而提出对角粘合方法（DAM），可有效结合二者并减轻性能损失，为相关研究奠定基础。
中文摘要 AI 辅助
随着语言模型参数规模不断增大，需有效模型压缩来降低计算和内存开销。现有压缩方法存在瓶颈问题，压缩率增加时性能显著下降。低秩分解和量化是两种可降低大语言模型计算和内存需求且保持精度的方法，组合二者有望突破瓶颈。本文首次证明二者非正交，通过实验验证了这一发现，其组合会导致性能显著下降。重要的是，提出了对角粘合方法（DAM），能有效结合二者并减轻性能损失，为模型压缩研究奠定了理论和实验基础。
英文摘要
As the parameter size of language models continues to grow, effective model compression is required to reduce their computational and memory overhead. Existing compression methods suffer from bottleneck issues: when the compression ratio is increased, performance degrades significantly. Low-rank decomposition and quantization are two prominent compression methods that have been proven to significantly reduce the computational and memory requirements of Large Language Models (LLMs) while maintaining model accuracy. Evidently, combining these two methods will break through the existing compression bottleneck. However, how these two methods interact when combined remains a critical question for developers, as many assume they are orthogonal, meaning their combination would not introduce additional errors beyond those independently introduced by each method. This paper provides the first mathematical proof that low-rank decomposition and quantization are non-orthogonal. We validate these findings through a series of experiments on large language models. Our results demonstrate that these methods are non-orthogonal, and their combination leads to significant performance degradation. Importantly, we propose a novel approach Diagonal Adhesive Method (DAM), which can effectively combine the two methods and mitigate the performance loss. Our research provides deep insights into model compression and lays a solid theoretical and experimental foundation for future related studies.
发表机构
- Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
- University of Chinese Academy of Sciences（中国科学院大学）
- Beijing Academy of Artificial Intelligence（北京人工智能研究院）
机构由 AI 辅助整理，请以论文原文为准。
