SLaB：用于高效大语言模型的稀疏低秩二进制分解
SLaB: Sparse-Lowrank-Binary Decomposition for Efficient Large Language Models
AI总结：
SLaB通过将线性层权重分解为稀疏矩阵、低秩矩阵和二进制矩阵，实现大语言模型的高效压缩，在50%压缩下使困惑度降低36%，在零样本任务中提升准确率8.98%。
摘要：
大语言模型（LLMs）的快速增长带来了显著的部署挑战，因其巨大的计算和内存需求。尽管模型压缩如网络剪枝提供了潜在解决方案，但现有方法往往难以在高压缩比下保持良好性能。为此，我们提出SLaB，一种新颖的框架，将每个线性层权重分解为三个互补的组成部分：稀疏矩阵、低秩矩阵和二进制矩阵。SLaB无需重新训练，并利用激活感知的剪枝分数来指导分解过程。在LLama家族模型上的实验表明，SLaB在50%压缩下比现有方法将困惑度降低高达36%，并在零样本任务中比基线模型提升准确率高达8.98%。
英文摘要：
The rapid growth of large language models (LLMs) presents significant deployment challenges due to their massive computational and memory demands. While model compression, such as network pruning, offers potential solutions, most existing methods often fail to maintain good performance at high compression ratios. To address this, we propose SLaB, a novel framework that decomposes each linear layer weight into three complementary components: a sparse matrix, a low-rank matrix, and a binary matrix. SLaB eliminates the need for retraining and leverages activation-aware pruning scores to guide the decomposition process. Experiments on Llama-family models demonstrate that SLaB achieves state-of-the-art performance, reducing perplexity by up to 36% compared to existing methods at 50% compression and improving accuracy by up to 8.98% over the baseline on zero-shot tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
