论语言定向剪枝的局限性：多语言大语言模型剪枝中校准语言的影响研究
On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning
AI总结：
本文针对多语言大语言模型剪枝场景，首次实证探究校准语言的影响，发现目标语言校准无法稳定提升下游性能，指出现有剪枝方法易丢失关键的语言无关特征。
摘要：
近期大语言模型（LLM）剪枝领域的进展已在训练后、无需重训练的场景下取得了最优（SotA）压缩效果，同时维持了较高的预测性能。然而，尽管现代大语言模型具备多语言属性且常被用于非英语场景，此前的研究主要考虑基于英语文本进行校准。本篇分析论文针对面向单语言应用的多语言模型剪枝所带来的性能变化与内部表征变化展开了深度调研。我们开展了首个全面的实证研究，在不同语言、任务、模型及SotA剪枝技术的设置下，对比了多语言模型剪枝时使用不同校准语言的效果。我们进一步分析了剪枝后模型的潜在子空间、剪枝掩码与单个神经元。结果显示，尽管基于目标语言校准可有效保留困惑度并得到较高的信噪比，但并不能持续提升下游任务性能。我们从三个不同层面对内部表征展开的进一步分析凸显了当前剪枝方法的更广泛局限性：这类方法虽能有效保留语言专属特征等主导信息，但这不足以抵消知识保留与推理所需的、细微的语言无关特征的损失。
英文摘要：
Recent advances in large language model (LLM) pruning have shown state-of-the-art (SotA) compression results in post-training and retraining-free settings while maintaining high predictive performance. However, previous research mainly considered calibrating based on English text, despite the multilingual nature of modern LLMs and their frequent use in non-English languages. This analysis paper conducts an in-depth investigation of the performance and internal representation changes associated with pruning multilingual language models for monolingual applications. We present the first comprehensive empirical study, comparing different calibration languages for pruning multilingual models across diverse languages, tasks, models, and SotA pruning techniques. We further analyze the latent subspaces, pruning masks, and individual neurons within pruned models. Our results reveal that while calibration on the target language effectively retains perplexity and yields high signal-to-noise ratios, it does not consistently improve downstream task performance. Further analysis of internal representations at three different levels highlights broader limitations of current pruning approaches: While they effectively preserve dominant information like language-specific features, this is insufficient to counteract the loss of nuanced, language-agnostic features that are crucial for knowledge retention and reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
