arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01997cs.LGcs.AI
收藏

关于生成推理中大语言模型层剪枝的极限

On the Limits of Layer Pruning for Generative Reasoning in Large Language Models

New York University Abu Dhabi(纽约大学阿布扎比分校)

Safal Shrestha, Anubhav Shrestha, Aadim Nepal, Minwu Kim, Keith Ross

AI总结:

研究发现层剪枝在分类任务中有效压缩模型但生成推理任务表现较差,揭示了剪枝对算法能力如算术和括号生成的影响,指出在受限条件下生成推理能力恢复有限。

摘要:

近期研究表明,层剪枝可有效压缩大语言模型(LLMs)并在分类基准上保持强性能,通常无需微调。相比之下,生成推理任务如GSM8K和HumanEval+表现出显著的恢复减弱。我们发现,除了表面文本退化外,剪枝导致关键算法能力的丧失,包括算术计算和平衡括号生成。在现实后训练约束下,无访问预训练规模数据或计算资源的情况下,我们评估了基于监督微调的最小恢复策略。该方法在分类任务中可恢复至基线性能的90%,但生成推理的恢复仍受根本限制。值得注意的是,即使在剪枝后微调约4000亿token的模型也未能恢复原始推理性能,表明此类能力不易恢复。此限制在简单的算术任务上也持续存在,此类任务无需多步生成。总体而言,我们界定了层剪枝在生成推理中的实际极限,并提供了在受限后训练条件下深度减少有效的指导。

英文摘要:

Recent work has shown that layer pruning can effectively compress large language models (LLMs) while retaining strong performance on classification benchmarks, often with little or no finetuning. In contrast, generative reasoning tasks, such as GSM8K and HumanEval\textsuperscript{+}, exhibit substantially weaker recovery. We show that beyond surface-level text degradation, pruning leads to a loss of key algorithmic capabilities, including arithmetic computation and balanced parenthesis generation. Under realistic post-training constraints, using a single 80GB GPU and without access to pretraining-scale data or compute, we evaluate a simple recovery strategy based on supervised finetuning with self-generated responses. This approach recovers up to 90\% of baseline performance on classification tasks, but recovery for generative reasoning remains limited. We further find that this gap persists even under a favorable task-aligned recovery setting, where pruned models are fully finetuned on self-generated GSM8K responses, suggesting that the degradation is not merely due to generic instruction data or parameter-efficient tuning. As complementary evidence, we analyze a depth-pruned model trained with nearly 100B post-pruning tokens and find that deficits persist even on simple arithmetic tasks that do not require multi-step generation. Overall, we characterize practical recovery limits of layer pruning for generative reasoning and provide guidance on when depth reduction is effective under constrained post-training regimes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑