arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.17764cs.CLcs.LG
收藏

1比特LLM时代:所有大型语言模型皆为1.58比特

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

Microsoft Research(微软研究院);University of Chinese Academy of Sciences(中国科学院大学)

Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei

AI总结:

本文提出BitNet b1.58,将LLM参数三值化为{-1,0,1},在性能匹配FP16/BF16模型的同时大幅降低延迟、内存与能耗,并定义新的缩放法则与专用硬件范式。

摘要:

近期研究(如BitNet)正在为1比特大型语言模型(LLMs)的新时代铺平道路。在本工作中,我们提出了一种1比特LLM变体,即BitNet b1.58,其中LLM的每一个参数(或权重)均为三值{-1, 0, 1}。在相同模型规模和训练token数的情况下,它在困惑度和端任务性能上均与全精度(即FP16或BF16)Transformer LLM相当,同时在延迟、内存、吞吐量和能耗方面显著更具成本效益。更深层地,1.58比特LLM定义了一种新的缩放法则和训练新一代LLM的配方,使其兼具高性能与成本效益。此外,它还催生了一种新的计算范式,并为设计针对1比特LLM优化的专用硬件打开了大门。

英文摘要:

Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption. More profoundly, the 1.58-bit LLM defines a new scaling law and recipe for training new generations of LLMs that are both high-performance and cost-effective. Furthermore, it enables a new computation paradigm and opens the door for designing specific hardware optimized for 1-bit LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑