大型语言模型(LLMs)虽然性能优异,但需要大量计算资源,限制了在边缘设备和资源受限环境中的部署。我们提出了TernaryLM,一个具有13200万参数的transformer模型,原生使用三元量化({-1, 0, +1},有效精度约为log2(3) ~ 1.58位),实现了显著的内存减少而不牺牲语言建模能力。与后训练量化方法不同,TernaryLM通过直通估计器和自适应的每层缩放因子从头学习量化感知表示。我们的实验显示:(1)在TinyStories上验证 perplexity 为58.42,交叉种子标准差为±0.17 PPL,证实了稳定的优化;(2)在MRPC任务上表现出色,使用比DistilBERT少55倍的预训练数据,F1得分达到82.47%;(3)内存减少2.4倍(498 MB vs 1,197 MB for an FP32模型),延迟相等;(4)三元约束产生隐含的正则化效果,使训练/验证比例为1.05倍,而FP32基线为3.51倍,证明离散权重在小语料库上防止过拟合。我们提供了层间稀疏性分析,显示中间transformer层(L5-L9)达到60-62%的量化稀疏度,而边界层为45-55%,确立了非均匀精度分配的设计原则。我们的实现和训练模型在https://github.com/1nisharg/TernaryLM-Memory-Efficient-Language-Modeling上公开可用。
英文摘要:
Large language models (LLMs) achieve remarkable performance but demand substantial computational resources, limiting deployment on edge devices and resource-constrained environments. We present TernaryLM, a 132M-parameter transformer trained natively with ternary quantization {-1, 0, +1} (log2(3) ~ 1.58-bit effective precision), achieving significant memory reduction without sacrificing language modeling capability. Unlike post-training quantization approaches that quantize pre-trained full-precision models, TernaryLM learns quantization-aware representations from scratch using straight-through estimators and adaptive per-layer scaling factors.
Our experiments demonstrate: (1) validation perplexity of 58.42 on TinyStories with a cross-seed standard deviation of +/- 0.17 PPL, confirming stable optimization; (2) strong downstream transfer with 82.47% F1 on MRPC, surpassing DistilBERT despite using 55x less pretraining data; (3) 2.4x memory reduction (498 MB vs 1,197 MB for an FP32 model of identical architecture) with latency parity; and (4) an implicit regularization effect whereby the ternary constraint yields a train/val ratio of 1.05x versus 3.51x for the FP32 baseline, demonstrating that discrete weights prevent overfitting on small corpora.
We provide layer-wise sparsity analysis revealing that middle transformer layers (L5-L9) achieve 60-62% quantization sparsity versus 45-55% for boundary layers, establishing an actionable design principle for non-uniform precision allocation. Our implementation and trained models are publicly available at https://github.com/1nisharg/TernaryLM-Memory-Efficient-Language-Modeling.
AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));City University of Hong Kong(香港城市大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);The Hong Kong University of Science and Technology(香港科技大学);University of California, Merced(加州大学默塞德分校);Chongqing University(重庆大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学)
Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models
The University of Hong Kong(香港大学);Huazhong University of Science and Technology(华中科技大学);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生学院)
AMix-2: Establishing Protein as a Native Modality in Large Language Models
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Generative Symbolic Intelligence Lab (GenSI), Tsinghua University(生成符号智能实验室(GenSI),清华大学);Institute for AI Industry Research (AIR), Tsinghua University(人工智能产业研究院(AIR),清华大学)
Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models
University of Chinese Academy of Sciences(中国科学院大学);Chinese Information Processing Laboratory(中文信息处理实验室);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所)
Tencent HY Team(腾讯混元团队);ShanghaiTech University(上海科技大学);The Hong Kong University of Science and Technology(香港科技大学);University of California, San Diego(加州大学圣地亚哥分校)
LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Southwest University(西南大学);Dalian University of Technology(大连理工大学);University of Science and Technology Beijing(北京科技大学);University College London(伦敦大学学院)
Why Pretraining Fails to Share Cross-Lingual Knowledge
Weizmann Institute of Science(魏茨曼科学研究所);Bar-Ilan University(巴伊兰大学);Johns Hopkins University(约翰斯·霍普金斯大学);A*STAR(新加坡科技研究局);University of Washington(华盛顿大学);MIT(麻省理工学院);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室)
Just Let Linear States Forget the Distant Past: Prefix Caching via Suffix Replay for Hybrid LLMs
Institute of Artificial Intelligence, China Telecom (TeleAI)(中国电信(TeleAI)人工智能研究所);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Individual Researcher(独立研究者)