arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.20682cs.LG
收藏

方差并非重要性:Transformer压缩的结构分析及其模型规模

Variance Is Not Importance: Structural Analysis of Transformer Compressibility Across Model Scales

fraQtl AI Research(fraQtl人工智能研究院)

Samuel Salfati

AI总结:

本文通过40多项实验研究Transformer压缩,发现方差与重要性无关,块线性依赖分布,重建墙现象,线性随深度增加,30%token计算简单。

摘要:

我们通过在GPT-2和Mistral 7B上的40多项实验系统地研究了Transformer压缩。我们的分析涵盖了谱压缩、块级功能替换、基于旋转的量化、激活几何和自适应早退。我们识别出五个与压缩相关的结构属性。首先,方差并非重要性:高方差激活方向与预测方向大约96%不相关(通过CCA测量),投影到这些子空间保留超过90%的方差,同时降低困惑度。其次,块线性是条件性的:Transformer块在GPT-2上近似线性(R²~0.95),在Mistral块31上为0.93,仅在正确的上游分布下成立;修改早期块会导致分布偏移,降低下游近似效果。第三,重建墙现象:将权重分解为量化组件的方法通过交叉项放大误差,使直接量化更优。第四,线性随深度增加:Mistral 7B从R²=0.17(块0)到R²=0.93(块31)逐渐增加,表明非线性特征构造与线性细化的分界。第五,大约30%的token计算简单,通过退出头和KL散度敏感性验证。我们证明单块线性替换在Mistral 7B最终块实现34倍压缩,困惑度增加1.71,而多块替换因残余误差积累和分布偏移失败。这些发现表明静态后训练压缩有根本限制,推动适应性、按token计算作为更有效方向。

英文摘要:

We present a systematic empirical study of transformer compression through over 40 experiments on GPT-2 (124M parameters) and Mistral 7B (7.24B parameters). Our analysis covers spectral compression, block-level function replacement, rotation-based quantization, activation geometry, and adaptive early exit. We identify five structural properties relevant to compression. (1) Variance is not importance: high-variance activation directions are approximately 96 percent uncorrelated with predictive directions (measured via CCA), and projecting onto these subspaces preserves over 90 percent of variance while degrading perplexity. (2) Block linearity is conditional: transformer blocks are approximately linear (R^2 ~ 0.95 on GPT-2, 0.93 on Mistral block 31) only under the correct upstream distribution; modifying earlier blocks induces distribution shift that degrades downstream approximations. (3) The reconstruction wall: approaches that factor weights into quantized components amplify errors through cross-terms, making direct quantization strictly superior. (4) Linearity increases with depth: Mistral 7B exhibits a progression from R^2 = 0.17 (block 0) to R^2 = 0.93 (block 31), indicating a division between nonlinear feature construction and linear refinement. (5) Approximately 30 percent of tokens are computationally easy, confirmed via exit heads and KL divergence sensitivity. We demonstrate that single-block linear replacement achieves 34x compression with a 1.71 perplexity increase on the final block of Mistral 7B, while multi-block replacement fails due to residual error accumulation and distribution shift. These findings suggest fundamental limits to static post-training compression and motivate adaptive, per-token computation as a more effective direction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑