arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.03142cs.LGcs.CL
收藏

少即是KEN:一种面向大语言模型的通用且简单的非参数剪枝算法

Less is KEN: a Universal and Simple Non-Parametric Pruning Algorithm for Large Language Models

University of Rome Tor Vergata(罗马第二大学);Campus Bio-Medico University of Rome(罗马生物医学大学校园大学)

Michele Mastromattei, Fabio Massimo Zanzotto

AI总结:

本文提出基于核密度估计的非参数剪枝算法KEN,通过保留关键参数并恢复其余参数至预训练状态,在至少减少25%参数的情况下保持或提升大语言模型性能,并配套可视化工具KEN$_{viz}$。

摘要:

由于神经网络模型的复杂性及其在各个领域的广泛应用,神经网络剪枝已变得日益关键。现有剪枝算法往往存在架构特异性、过度复杂以及依赖高成本计算等局限,使其难以在实际应用中落地。本文提出KEN:一种基于核密度估计(KDE)的简洁、通用且非结构化的剪枝算法。KEN旨在通过选择性保留最重要的参数,同时将其他参数恢复至预训练状态,从而构建优化的Transformer模型。该策略在保持模型性能的同时,仅需存储优化后的子网络,带来显著的内存节省。在七个不同的大语言模型上的广泛评估表明,KEN能够达到与原未剪枝版本相当或更优的性能,且参数至少减少25%。此外,与既有剪枝算法和PEFT算法的深入对比进一步证实了KEN的有效性。我们还引入了KEN$_{viz}$,一种可解释工具,能够从不同视角可视化KEN所获得的优化模型构成。

英文摘要:

Neural network pruning has become increasingly crucial due to the complexity of these models and their widespread use in various fields. Existing pruning algorithms often suffer from limitations such as architecture specificity, excessive complexity and reliance on demanding calculations, rendering them impractical for real-world applications. This paper introduces KEN: a straightforward, universal and unstructured pruning algorithm based on Kernel Density Estimation (KDE). KEN aims to construct optimized transformers by selectively preserving the most significant parameters while restoring others to their pre-training state. This strategy preserves model performance while enabling storage of only the optimized subnetwork, leading to substantial memory savings. Extensive evaluations across seven different LLMs demonstrate that KEN achieves equal or better performance than their original unpruned versions, with a minimum parameter reduction of 25%. Furthermore, in-depth comparisons with established pruning and PEFT algorithms confirm KEN effectiveness. We further introduce KEN$_{viz}$, an explainable tool that visualizes the optimized model composition achieved by KEN from different points of view.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑