剪枝作为合作博弈：用于大型语言模型的代理辅助层贡献估计
Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for Large Language Models
AI总结：
本文提出基于博弈论的层剪枝方法，利用代理网络估计层贡献，实现高效且有效的大型语言模型剪枝。
摘要：
尽管大型语言模型（LLMs）在各种任务上表现出色，但其在现实场景中的部署仍然受到高计算需求的限制。逐层剪枝是一种常用的策略，用于减轻推理成本，可以部分解决这一挑战。然而，现有方法通常依赖于静态启发式规则，未能考虑层之间的相互依赖性，从而限制了剪枝过程的有效性。为此，本文提出了一种博弈论框架，将层剪枝作为合作博弈来建模，其中每一层都作为参与者，模型性能作为效用。由于精确计算Shapley值对大型语言模型（LLMs）来说计算上不可行，我们提出使用轻量级的代理网络来估计逐层边际贡献。该网络可以以低计算成本预测LLM在任意层组合下的性能。此外，我们采用分层蒙特卡洛掩码采样进一步减少Shapley值估计的成本。该方法捕捉了层间的依赖性，并动态识别关键层进行剪枝。广泛的实验表明，我们的方法在困惑度和零样本准确性方面表现一致优越，实现了更高效和有效的大型语言模型逐层剪枝。
英文摘要：
While large language models (LLMs) demonstrate impressive performance across various tasks, their deployment in real-world scenarios is still constrained by high computational demands. Layer-wise pruning, a commonly employed strategy to mitigate inference costs, can partially address this challenge. However, existing approaches generally depend on static heuristic rules and fail to account for the interdependencies among layers, thereby limiting the effectiveness of the pruning process. To this end, this paper proposes a game-theoretic framework that formulates layer pruning as a cooperative game in which each layer acts as a player and model performance serves as the utility. As computing exact Shapley values is computationally infeasible for large language models (LLMs), we propose using a lightweight surrogate network to estimate layer-wise marginal contributions. This network can predict LLM performance for arbitrary layer combinations at a low computational cost. Additionally, we employ stratified Monte Carlo mask sampling to further reduce the cost of Sharpley value estimation. This approach captures inter-layer dependencies and dynamically identifies critical layers for pruning. Extensive experiments demonstrate the consistent superiority of our method in terms of perplexity and zero-shot accuracy, achieving more efficient and effective layer-wise pruning for large language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
