发表机构
College of Computing and Data Science, Nanyang Technological University; Interdisciplinary Graduate Programme, Nanyang Technological University; School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学计算与数据科学学院; 南洋理工大学跨学科研究生项目; 南洋理工大学电气与电子工程学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究针对大语言模型结构化剪枝中独立评分单元不合理的问题，提出CoCurve方法，通过二阶泰勒展开得到Fisher矩阵计算协同剪枝曲率边，联合修剪注意力和FFN单元，仅用M次前向传播求解预算二次规划，无需标签、微调等完成剪枝。
AI 中文摘要
结构化剪枝通过移除整个计算单元（如注意力头和前馈（FFN）通道组）来压缩大语言模型（LLMs）。然而，大多数无训练方法独立对这些单元进行排序，将修剪一组单元的损失视为其个体损失之和。对于通过共享残差流耦合子层的Transformer模型，这种观点不成立。两个单独较弱的单元可能共同不可或缺，但独立评分却忽略这种依赖并将它们一起移除。我们引入CoCurve（跨模块协同剪枝曲率），一种仅校准、无需微调的方法，可联合修剪注意力和FFN单元。通过对冻结模型与其掩码副本之间的token级KL进行二阶泰勒展开，得到一个单一的Fisher矩阵，其对角线是经典节点显著性，非对角线元素是协同剪枝曲率边：一起移除两个单元的额外损害。在单消融加性近似下，该矩阵简化为单单元消融特征的Gram积，因此从M次前向传播中恢复完整的M×M交互，无需成对扫描或梯度。剪枝随后简化为一个预算二次规划，在共享注意力 - FFN预算下一次性求解，无需标签、微调或恢复。
英文摘要
Resource-constrained deployment requires sustaining large language model (LLM) capabilities as models scale under fixed memory and computation budgets. Structured pruning advances this deployment frontier with smaller dense checkpoints, yet deciding what to prune remains bottlenecked by interactions among joint removals. We introduce Cross-Module Co-Pruning Curvature (CoCurve), which formulates structured pruning as set-dependent predictive risk over a unified inventory of attention heads and feed-forward groups. A co-pruning graph built from single-unit forward ablations assigns individual risk to nodes and reinforcement or cancellation to interaction edges, conditioning each decision on the units already removed. We evaluate 6 LLMs (3B--70B) and 3 vision--language models (VLMs) across 3 perplexity corpora, 12 language tasks, and 7 multimodal benchmarks. Across the five-model 20--40% grid and the 70B 10--50% sweep, CoCurve ranks first in 53/60 corpus comparisons (15/15 at 70B); matched-quality interpolation permits 2.2--6.6 more pruning points in 9/10 cases, while CoCurve leads all 6 VLM Avg$_7$ blocks. After the same lightweight recovery, its retained structures remain strongest through 50% pruning, where the 8B checkpoint regains 10.8 Avg$_{12}$ points; physical slicing delivers $1.58\times$ dense prefill throughput with 41% lower peak memory. Mechanism analysis across 10 LLMs and 7 VLMs finds organized within- and cross-module edge structure; matched low-saliency, high-coupling removals degrade 19/20 capability groups by up to 23.7 points.
