评估压缩技术对大语言模型任务特定性能的影响
Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Models
AI总结：
该研究评估三种主流压缩方法对LLaMA-2-7B模型的影响，发现仅用困惑度评估压缩效果不足，提出JS散度作为更全面指标，并证实任务特定校准数据可提升压缩模型下游性能。
摘要：
大语言模型（LLMs）具备强大能力，但会产生高昂的计算成本，这推动了对高效压缩技术的需求。本研究评估了Magnitude Pruning（幅值剪枝）、SparseGPT和Wanda这几种主流压缩方法对LLaMA-2-7B模型的影响，重点关注模型尺寸缩减、下游任务性能与校准数据作用之间的权衡。研究结果表明，尽管SparseGPT和Wanda即使在50%稀疏度下仍能保持困惑度，但它们在下游任务上会出现显著性能下降，这凸显了仅将困惑度作为评估指标的不足。为解决这一问题，我们引入Jensen-Shannon（JS）散度作为更全面的指标，它能够捕捉压缩后模型行为的细微变化。我们进一步证明，与通用校准数据相比，任务特定校准数据能显著提升压缩模型的下游性能。本研究强调，为充分理解LLM压缩的复杂性及其对实际应用的影响，需要采用多样化的评估指标并谨慎选择校准数据。
英文摘要：
Large language models (LLMs) offer powerful capabilities but incur substantial computational costs, driving the need for efficient compression techniques. This study evaluates the impact of popular compression methods - Magnitude Pruning, SparseGPT, and Wanda - on the LLaMA-2-7B model, focusing on the trade-offs between model size reduction, downstream task performance, and the role of calibration data. Our findings reveal that while SparseGPT and Wanda preserve perplexity even at 50% sparsity, they suffer significant degradation on downstream tasks, highlighting the inadequacy of perplexity as the sole evaluation metric. To address this, we introduce Jensen-Shannon (JS) Divergence as a more comprehensive metric that captures nuanced changes in model behavior post-compression. We further demonstrate that task-specific calibration data significantly enhances the downstream performance of compressed models compared to general calibration data. This research underscores the necessity for diverse evaluation metrics and careful calibration data selection to fully understand the complexities of LLM compression and its implications for practical applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
