We present a systematic empirical study of transformer compression through over 40 experiments on GPT-2 (124M parameters) and Mistral 7B (7.24B parameters). Our analysis covers spectral compression, block-level function replacement, rotation-based quantization, activation geometry, and adaptive early exit.
We identify five structural properties relevant to compression. (1) Variance is not importance: high-variance activation directions are approximately 96 percent uncorrelated with predictive directions (measured via CCA), and projecting onto these subspaces preserves over 90 percent of variance while degrading perplexity. (2) Block linearity is conditional: transformer blocks are approximately linear (R^2 ~ 0.95 on GPT-2, 0.93 on Mistral block 31) only under the correct upstream distribution; modifying earlier blocks induces distribution shift that degrades downstream approximations. (3) The reconstruction wall: approaches that factor weights into quantized components amplify errors through cross-terms, making direct quantization strictly superior. (4) Linearity increases with depth: Mistral 7B exhibits a progression from R^2 = 0.17 (block 0) to R^2 = 0.93 (block 31), indicating a division between nonlinear feature construction and linear refinement. (5) Approximately 30 percent of tokens are computationally easy, confirmed via exit heads and KL divergence sensitivity.
We demonstrate that single-block linear replacement achieves 34x compression with a 1.71 perplexity increase on the final block of Mistral 7B, while multi-block replacement fails due to residual error accumulation and distribution shift. These findings suggest fundamental limits to static post-training compression and motivate adaptive, per-token computation as a more effective direction.
A Theory for Compressibility of Graph Transformers for Transductive Learning
University of British Columbia(不列颠哥伦比亚大学);Carnegie Mellon University(卡内基梅隆大学);Meta;CMU(卡内基梅隆大学);Google Research(谷歌研究院);UBC(不列颠哥伦比亚大学);Amii(阿尔伯塔创新人工智能研究所)
Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility
Center for Applied Mathematics(应用数学中心);Cornell University(康奈尔大学);Amazon Web Services(亚马逊网络服务);Amazon Selling Partner Services(亚马逊销售合作伙伴服务);Amazon Supply Chain Optimization Technologies(亚马逊供应链优化技术)
Joint rotational invariance and adversarial training of a dual-stream Transformer yields state of the art Brain-Score for Area V4
The Center for Brains, Minds and Machines(大脑、心智与机器中心);Massachusetts Institute of Technology(麻省理工学院)
现代在大脑评分竞赛中获得高分的视觉模型并不源于Vision Transformers。然而,在本文中,我们通过展示双流Transformer,CrossViT $~\textit{a la}$ Chen et al.(2021),在联合旋转不变和对抗优化程序下,在2022年总Brain-Score竞赛中获得第二名,提供了反对Vision Transformers(ViT)在感知上与人类视觉表示不一致的意外趋势的证据(Schrimpf et al.,2020 b)在所有视觉类别中平均,并且在比赛时在V4区域的最高可解释方差中排名第一。此外,我们目前的基于变换器的模型还实现了区域V4、IT和行为的更大的可解释方差,而不是生物启发的CNN(ResNet 50),其集成了类似于额叶V1的计算模块(Dapello等人,2020年)。为了评估优化方案对CrossViT架构的贡献,我们对不同优化的CrossViT的对抗鲁棒性,常见的腐败基准,中腹部刺激解释和特征反转进行了几次额外的实验。与我们最初的预期,我们的家庭的结果提供了临时支持$\textit{"All roads lead to Rome"}$参数通过联合优化规则,即使是非生物学动机的模型,如Vision Transformers的视觉。代码可在https://github.com/williamberrios/BrainScore-T…
Beyond Variance: Knowledge-Aware LLM Compression via Fisher-Aligned Subspace Diagnostics
Department of Computer Science, Iowa State University(计算机科学系,爱荷华州立大学);Department of Civil, Construction & Environmental Engineering, Iowa State University(土木、建设与环境工程系,爱荷华州立大学)