发表机构
Tsinghua University; Huawei Tech. Co. Ltd.（清华大学; 华为技术有限公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对现有结构化剪枝忽略单元间关联的问题，提出关联感知剪枝方法，通过建模依赖的二次规划与贪婪算法优化选择，实现自适应层间稀疏分配，在主流LLMs上取得有竞争力的精度-效率权衡。
AI 中文摘要
结构化剪枝是一种在保持硬件效率的同时降低大型语言模型（LLMs）大量推理成本的有前景的方法。许多现有方法孤立地评估可剪枝单元（如通道或头）的重要性，隐含地假设剪枝误差是可加的。这种独立性假设常因模型权重的非正交性以及单元激活之间的强相关性而失效，可能导致性能下降。为解决此问题，我们提出了一种关联感知结构化剪枝方法。我们将剪枝目标表述为一个基数约束的二元二次规划，该规划显式地建模了重建误差中跨单元的依赖关系。由于该二元二次规划是NP难的且难以精确求解，我们开发了一种基于依赖感知边际成本的贪婪交互算法来优化单元选择。此外，我们引入了一种基于梯度的策略，以实现整个模型的自适应逐层稀疏度分配。在主流LLMs上的大量实验表明，与代表性的结构化剪枝基线相比，纳入关联信息可产生有竞争力的精度-效率权衡。
英文摘要
Structured pruning is a promising approach for reducing the substantial inference costs of Large Language Models (LLMs) while maintaining hardware efficiency. Many existing methods assess the importance of prunable units (e.g., channels or heads) in isolation, implicitly assuming that pruning errors are additive. This independence assumption is often invalidated by the non-orthogonality of model weights and strong correlations between unit activations, potentially leading to performance degradation. To address this, we propose a Correlation-Aware Structured Pruning method. We formulate the pruning objective as a cardinality-constrained binary quadratic program that explicitly models cross-unit dependencies in the reconstruction error. Since this binary quadratic program is NP-hard and difficult to solve exactly, we develop a greedy interaction algorithm based on dependency-aware marginal costs to optimize unit selection. Furthermore, we incorporate a gradient-based strategy to achieve adaptive layer-wise sparsity allocation across the entire model. Extensive experiments on mainstream LLMs demonstrate that incorporating correlation information yields competitive accuracy-efficiency trade-offs compared to representative structured pruning baselines.
