Tele-FLM 技术报告
Tele-FLM Technical Report
AI总结：
本报告介绍52B开源多语言模型Tele-FLM，提出稳定高效预训练范式并增强事实判断能力，性能媲美更大FLOPs的开源模型，同时公开核心设计与工程实践以惠及学界业界。
摘要：
大型语言模型（LLMs）在语言理解和生成方面展现出深厚能力，促进了广泛应用。然而，关于以最小试错成本和计算资源高效扩展超过500亿参数的LLMs，目前缺乏详细的开源方法。本报告介绍Tele-FLM（又名FLM-2），这是一个52B开源多语言大型语言模型，具有稳定、高效的预训练范式和增强的事实判断能力。Tele-FLM在文本语料上以BPB衡量，展现出卓越的多语言建模能力。此外，在英文和中文基础模型评估中，它与涉及更大预训练FLOPs的强开源模型（如Llama2-70B和DeepSeek-67B）相当。除模型权重外，我们还分享核心设计、工程实践和训练细节，期望惠及学术界和工业界。
英文摘要：
Large language models (LLMs) have showcased profound capabilities in language understanding and generation, facilitating a wide array of applications. However, there is a notable paucity of detailed, open-sourced methodologies on efficiently scaling LLMs beyond 50 billion parameters with minimum trial-and-error cost and computational resources. In this report, we introduce Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that features a stable, efficient pre-training paradigm and enhanced factual judgment capabilities. Tele-FLM demonstrates superior multilingual language modeling abilities, measured by BPB on textual corpus. Besides, in both English and Chinese foundation model evaluation, it is comparable to strong open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B and DeepSeek-67B. In addition to the model weights, we share the core designs, engineering practices, and training details, which we expect to benefit both the academic and industrial communities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
