FBI-LLM：通过自回归蒸馏从零扩展全二值化大语言模型
FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation
AI总结：
本研究提出FBI-LLM，通过自回归蒸馏从零训练全二值化大语言模型，匹配全精度模型性能，验证预训练权重非必需，为1比特LLM硬件设计提供支撑并开源相关资源。
摘要：
本研究提出了全二值化大语言模型FBI-LLM，首次展示了如何从零训练大规模二值语言模型（而非类似BitNet b1.58的部分二值或三值LLM），使其在基于Transformer的LLM中匹配全精度模型（如FP16或BF16）的性能。该模型通过采用自回归蒸馏（AD）损失实现这一目标，保持与常规LLM预训练相当的模型维度（130M、1.3B、7B）和训练数据量，同时在困惑度和特定任务效果上取得具有竞争力的结果。有趣的是，通过分析训练轨迹，我们发现预训练权重对于从零训练二值化LLM并非必需。该研究提出了一种新的计算框架，或可推动未来专为全1比特LLM设计的专用硬件的发展。我们完全公开所有模型、代码和训练数据集以支持后续研究（代码：https://github.com/LiqunMa/FBI-LLM；模型：https://huggingface.co/LiqunMa/）。
英文摘要：
This work presents a Fully BInarized Large Language Model (FBI-LLM), demonstrating for the first time how to train a large-scale binary language model from scratch (not the partial binary or ternary LLM like BitNet b1.58) to match the performance of its full-precision counterparts (e.g., FP16 or BF16) in transformer-based LLMs. It achieves this by employing an autoregressive distillation (AD) loss with maintaining equivalent model dimensions (130M, 1.3B, 7B) and training data volume as regular LLM pretraining, while delivering competitive results in terms of perplexity and task-specific effectiveness. Intriguingly, by analyzing the training trajectory, we find that the pretrained weight is not necessary for training binarized LLMs from scratch. This research encourages a new computational framework and may facilitate the future design of specialized hardware tailored for fully 1-bit LLMs. We make all models, code, and training dataset fully accessible and transparent to support further research (Code: https://github.com/LiqunMa/FBI-LLM. Model: https://huggingface.co/LiqunMa/).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
