通过识别和保留功能网络来剪枝大型语言模型
Pruning Large Language Models by Identifying and Preserving Functional Networks
AI总结：
本研究受大脑功能网络启发，提出通过识别并保留LLM中的功能网络及关键神经元进行结构化剪枝，以降低内存消耗并加速推理，实验验证了其有效性。
摘要：
结构化剪枝是压缩大型语言模型（LLM）以减少GPU内存消耗和加速推理速度的代表性技术之一。它在提高LLM在实际应用中的效率方面具有重要的实用价值。当前的结构化剪枝方法通常依赖于对结构单元重要性的评估，并剪除重要性较低的单元。其中大多数方法忽视了人工神经元之间的交互和协作，而这些对于LLM的功能至关重要，导致LLM的宏观功能架构被破坏，进而造成剪枝性能下降。受人工神经网络与人脑功能神经网络之间固有相似性的启发，我们缓解了这一挑战，并在这项研究中提出通过识别和保留LLM内的功能网络来剪枝LLM。为实现这一目标，我们将LLM视为数字大脑，并将LLM分解为功能网络，类似于在神经影像数据中识别功能脑网络。随后，通过保留这些功能网络中的关键神经元来剪枝LLM。实验结果表明，所提出的方法能够成功识别和定位LLM中的功能网络和关键神经元，从而实现高效的模型剪枝。我们的代码可在https://github.com/WhatAboutMyStar/LLM_ACTIVATION获取。
英文摘要：
Structured pruning is one of the representative techniques for compressing large language models (LLMs) to reduce GPU memory consumption and accelerate inference speed. It offers significant practical value in improving the efficiency of LLMs in real-world applications. Current structured pruning methods typically rely on assessment of the importance of the structure units and pruning the units with less importance. Most of them overlooks the interaction and collaboration among artificial neurons that are crucial for the functionalities of LLMs, leading to a disruption in the macro functional architecture of LLMs and consequently a pruning performance degradation. Inspired by the inherent similarities between artificial neural networks and functional neural networks in the human brain, we alleviate this challenge and propose to prune LLMs by identifying and preserving functional networks within LLMs in this study. To achieve this, we treat an LLM as a digital brain and decompose the LLM into functional networks, analogous to identifying functional brain networks in neuroimaging data. Afterwards, an LLM is pruned by preserving the key neurons within these functional networks. Experimental results demonstrate that the proposed method can successfully identify and locate functional networks and key neurons in LLMs, enabling efficient model pruning. Our code is available at https://github.com/WhatAboutMyStar/LLM_ACTIVATION.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
