arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.07374cs.CLcs.AI
收藏

TernaryLM: 通过原生1.5位量化与自适应层间缩放实现内存高效的语言模型

TernaryLM: Memory-Efficient Language Modeling via Native 1.5-Bit Quantization with Adaptive Layer-wise Scaling

KIIT Bhubaneshwar, India(印度KIIT大学);IIIT Hyderabad, India(印度国际信息技术学院海德拉巴分校)

Nisharg Nargund, Priyesh Shukla

AI总结:

TernaryLM通过原生1.5位量化与自适应层间缩放,在保持语言建模能力的同时显著减少内存使用,实验显示其在多个任务上的表现优异,且在小数据集上防止过拟合。

摘要:

大型语言模型(LLMs)虽然性能优异,但需要大量计算资源,限制了在边缘设备和资源受限环境中的部署。我们提出了TernaryLM,一个具有13200万参数的transformer模型,原生使用三元量化({-1, 0, +1},有效精度约为log2(3) ~ 1.58位),实现了显著的内存减少而不牺牲语言建模能力。与后训练量化方法不同,TernaryLM通过直通估计器和自适应的每层缩放因子从头学习量化感知表示。我们的实验显示:(1)在TinyStories上验证 perplexity 为58.42,交叉种子标准差为±0.17 PPL,证实了稳定的优化;(2)在MRPC任务上表现出色,使用比DistilBERT少55倍的预训练数据,F1得分达到82.47%;(3)内存减少2.4倍(498 MB vs 1,197 MB for an FP32模型),延迟相等;(4)三元约束产生隐含的正则化效果,使训练/验证比例为1.05倍,而FP32基线为3.51倍,证明离散权重在小语料库上防止过拟合。我们提供了层间稀疏性分析,显示中间transformer层(L5-L9)达到60-62%的量化稀疏度,而边界层为45-55%,确立了非均匀精度分配的设计原则。我们的实现和训练模型在https://github.com/1nisharg/TernaryLM-Memory-Efficient-Language-Modeling上公开可用。

英文摘要:

Large language models (LLMs) achieve remarkable performance but demand substantial computational resources, limiting deployment on edge devices and resource-constrained environments. We present TernaryLM, a 132M-parameter transformer trained natively with ternary quantization {-1, 0, +1} (log2(3) ~ 1.58-bit effective precision), achieving significant memory reduction without sacrificing language modeling capability. Unlike post-training quantization approaches that quantize pre-trained full-precision models, TernaryLM learns quantization-aware representations from scratch using straight-through estimators and adaptive per-layer scaling factors. Our experiments demonstrate: (1) validation perplexity of 58.42 on TinyStories with a cross-seed standard deviation of +/- 0.17 PPL, confirming stable optimization; (2) strong downstream transfer with 82.47% F1 on MRPC, surpassing DistilBERT despite using 55x less pretraining data; (3) 2.4x memory reduction (498 MB vs 1,197 MB for an FP32 model of identical architecture) with latency parity; and (4) an implicit regularization effect whereby the ternary constraint yields a train/val ratio of 1.05x versus 3.51x for the FP32 baseline, demonstrating that discrete weights prevent overfitting on small corpora. We provide layer-wise sparsity analysis revealing that middle transformer layers (L5-L9) achieve 60-62% quantization sparsity versus 45-55% for boundary layers, establishing an actionable design principle for non-uniform precision allocation. Our implementation and trained models are publicly available at https://github.com/1nisharg/TernaryLM-Memory-Efficient-Language-Modeling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑