arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.07892cs.LGcs.AI
收藏

Sherry:通过细粒度稀疏化实现高效的1.25位三元量化

Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification

City University of Hong Kong(香港城市大学);Tencent(腾讯);McGill University(麦吉尔大学)

Hong Huang, Decheng Wu, Qiangqiang Hu, Guanghua Yu, Jinhai Yang, Jianchen Zhu, Xue Liu, Dapeng Wu

AI总结:

Sherry通过细粒度稀疏化实现高效的1.25位三元量化,显著减少模型大小并提升推理速度。

摘要:

大型语言模型(LLMs)在资源受限的边缘设备上的部署越来越受到内存和计算需求的限制。虽然三元量化通过将权重减少到{-1, 0, +1}提供了一个有吸引力的解决方案,但现有实现与商用硬件存在根本性的不匹配。大多数现有方法必须在2位对齐打包(导致显著的位浪费)或1.67位不规则打包(降级推理速度)之间做出选择。为了解决这种矛盾,我们提出了Sherry,一个高效的三元量化框架。Sherry引入了3:4的细粒度稀疏性,通过将四个权重打包成五个位,实现了正则化的1.25位宽度,恢复了2的幂次对齐。此外,我们识别出稀疏三元训练中的权重陷阱问题,这会导致表示崩溃。为了解决这个问题,Sherry引入了Arenas,一种退火残差突触机制,在训练过程中保持表示多样性。在LLaMA-3.2上五个基准的实证评估显示,Sherry在匹配最先进的三元性能的同时显著减少了模型大小。值得注意的是,在Intel i7-14700HX CPU上,我们的1B模型在与SOTA基线相比时实现了零精度损失,同时提供了25%的位节省和10%的速度提升。代码可在https://github.com/Tencent/AngelSlim上获得。

英文摘要:

The deployment of Large Language Models (LLMs) on resource-constrained edge devices is increasingly hindered by prohibitive memory and computational requirements. While ternary quantization offers a compelling solution by reducing weights to {-1, 0, +1}, current implementations suffer from a fundamental misalignment with commodity hardware. Most existing methods must choose between 2-bit aligned packing, which incurs significant bit wastage, or 1.67-bit irregular packing, which degrades inference speed. To resolve this tension, we propose Sherry, a hardware-efficient ternary quantization framework. Sherry introduces a 3:4 fine-grained sparsity that achieves a regularized 1.25-bit width by packing blocks of four weights into five bits, restoring power-of-two alignment. Furthermore, we identify weight trapping issue in sparse ternary training, which leads to representational collapse. To address this, Sherry introduces Arenas, an annealing residual synapse mechanism that maintains representational diversity during training. Empirical evaluations on LLaMA-3.2 across five benchmarks demonstrate that Sherry matches state-of-the-art ternary performance while significantly reducing model size. Notably, on an Intel i7-14700HX CPU, our 1B model achieves zero accuracy loss compared to SOTA baselines while providing 25% bit savings and 10% speed up. The code is available at https://github.com/Tencent/AngelSlim .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑