只需剪枝一次：利用策略学习设计免校准的模型压缩
You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning
AI总结：
针对LLM压缩依赖校准数据且高压缩率下性能下降的问题，提出PruneNet，将剪枝重构为策略学习，免校准，15分钟压缩LLaMA-2-7B，30%压缩率下保留超80%性能。
摘要：
大型语言模型（LLMs）规模的不断增大，因其高昂的计算和内存需求，给部署带来了重大挑战。当前的模型剪枝技术试图通过严重依赖外部校准数据集来确定剪枝或压缩哪些参数来缓解这些问题，从而限制了它们在不同压缩比率下的灵活性和可扩展性。此外，这些方法在较高的压缩率下，尤其是在下游任务中，常常导致严重的性能下降。在本文中，我们提出了PruneNet，一种新颖的模型压缩方法，通过将模型剪枝重新表述为策略学习过程来解决这些局限性。PruneNet将剪枝过程与模型架构解耦，消除了对校准数据集的需求。它学习一种随机剪枝策略，仅基于模型的内在属性评估参数重要性，同时保留谱结构以最小化信息损失。PruneNet可以在仅15分钟内压缩LLaMA-2-7B模型，在30%的压缩比率下实现其零样本性能超过80%的保留，优于现有仅保留75%性能的方法。此外，在复杂的多任务语言理解任务中，PruneNet通过保留原始模型高达80%的性能展示了其鲁棒性，证明了自己是传统结构化压缩技术的优越替代方案。
英文摘要：
The ever-increasing size of large language models (LLMs) presents significant challenges for deployment due to their heavy computational and memory requirements. Current model pruning techniques attempt to alleviate these issues by relying heavily on external calibration datasets to determine which parameters to prune or compress, thus limiting their flexibility and scalability across different compression ratios. Moreover, these methods often cause severe performance degradation, particularly in downstream tasks, when subjected to higher compression rates. In this paper, we propose PruneNet, a novel model compression method that addresses these limitations by reformulating model pruning as a policy learning process. PruneNet decouples the pruning process from the model architecture, eliminating the need for calibration datasets. It learns a stochastic pruning policy to assess parameter importance solely based on intrinsic model properties while preserving the spectral structure to minimize information loss. PruneNet can compress the LLaMA-2-7B model in just 15 minutes, achieving over 80% retention of its zero-shot performance with a 30% compression ratio, outperforming existing methods that retain only 75% performance. Furthermore, on complex multitask language understanding tasks, PruneNet demonstrates its robustness by preserving up to 80% performance of the original model, proving itself a superior alternative to conventional structured compression techniques.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
