52 B到1 T：通过Tele-FLM系列学到的教训
52B to 1T: Lessons Learned via Tele-FLM Series
摘要：
大语言模型（LLM）代表了人工通用智能的一个重大进步。由于缩放定律强调了增加模型大小的潜力，学术界加强了对容量超过500亿个参数的LLM的研究。这份技术报告建立在我们之前使用Tele-FLM（也称为FLM-2）的工作基础上，这是一个公开的520亿参数模型。我们深入研究了两个主要领域：我们首先讨论了我们对Tele-FLM-52 B上的监督微调（SFT）的观察，该观察支持SFT数据构建的“少即是多”方法;其次，我们展示了我们的实验和分析，这些实验和分析是关于将模型从520亿逐步增长到1020亿，随后增长到1万亿参数的最佳实践。我们将开源一个1 T模型检查点，即Tele-FLM-1 T，以促进进一步的培训和研究。
英文摘要：
Large Language Models (LLMs) represent a significant stride toward Artificial General Intelligence. As scaling laws underscore the potential of increasing model sizes, the academic community has intensified its investigations into LLMs with capacities exceeding 50 billion parameters. This technical report builds on our prior work with Tele-FLM (also known as FLM-2), a publicly available 52-billion-parameter model. We delve into two primary areas: we first discuss our observation of Supervised Fine-tuning (SFT) on Tele-FLM-52B, which supports the "less is more" approach for SFT data construction; second, we demonstrate our experiments and analyses on the best practices for progressively growing a model from 52 billion to 102 billion, and subsequently to 1 trillion parameters. We will open-source a 1T model checkpoint, namely Tele-FLM-1T, to advance further training and research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
