arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.09900cs.CL
收藏

GEB-1.3B:开放的轻量级大型语言模型

GEB-1.3B: Open Lightweight Large Language Model

GEB

Jie Wu, Yufeng Zhu, Lei Shen, Xuqing Lu

AI总结:

本文提出开源轻量级中英双语模型 GEB-1.3B,结合 ROPE、GQA、FlashAttention-2 与指令微调,在多项基准上优于同规模模型并提升 CPU 推理效率。

摘要:

近期开发的大型语言模型(LLM),如 ChatGPT、Claude 和 Llama,已展现出令人瞩目的能力,甚至在若干任务上超过人类水平。尽管取得了这些成功,这些模型对资源的密集需求——训练和推理都需要大量计算能力——将其部署限制在高性能服务器上。此外,模型的大量计算需求往往导致响应时间延迟增加。随着 LLM 在 CPU 上高效运行的需求不断增长,面向 CPU 推理优化的轻量级模型研究已经出现。在本工作中,我们推出 GEB-1.3B,这是一个使用 5500 亿个中英文 token 训练的轻量级 LLM。我们采用包括 ROPE、Group-Query-Attention 和 FlashAttention-2 在内的新型训练技术,在保持模型性能的同时加速训练。此外,我们使用 1000 万条指令数据样本对模型进行微调,以增强对齐。GEB-1.3B 在 MMLU、C-Eval 和 CMMLU 等通用基准上表现出色,优于 MindLLM-1.3B 和 TinyLLaMA-1.1B 等对比模型。值得注意的是,GEB-1.3B 的 FP32 版本在 CPU 上实现了值得称道的推理时间,同时我们正持续努力,通过先进的量化技术进一步提升速度。GEB-1.3B 作为开源模型发布,标志着对轻量级 LLM 发展的重要贡献,有望促进该领域的进一步研究与创新。

英文摘要:

Recently developed large language models (LLMs) such as ChatGPT, Claude, and Llama have demonstrated impressive abilities, and even surpass human-level performance in several tasks. Despite their success, the resource-intensive demands of these models, requiring significant computational power for both training and inference, limit their deployment to high-performance servers. Additionally, the extensive calculation requirements of the models often lead to increased latency in response times. With the increasing need for LLMs to operate efficiently on CPUs, research about lightweight models that are optimized for CPU inference has emerged. In this work, we introduce GEB-1.3B, a lightweight LLM trained on 550 billion tokens in both Chinese and English languages. We employ novel training techniques, including ROPE, Group-Query-Attention, and FlashAttention-2, to accelerate training while maintaining model performance. Additionally, we fine-tune the model using 10 million samples of instruction data to enhance alignment. GEB-1.3B exhibits outstanding performance on general benchmarks such as MMLU, C-Eval, and CMMLU, outperforming comparative models such as MindLLM-1.3B and TinyLLaMA-1.1B. Notably, the FP32 version of GEB-1.3B achieves commendable inference times on CPUs, with ongoing efforts to further enhance speed through advanced quantization techniques. The release of GEB-1.3B as an open-source model marks a significant contribution to the development of lightweight LLMs, promising to foster further research and innovation in the field.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑