揭开缩放定律的奥秘：第一部分
Unraveling the Mystery of Scaling Laws: Part I
AI总结：
本文验证 OpenAI 缩放定律可扩展至 33B 参数模型，并提出仅用 1M～60M 参数小模型估计公式常数的方法，以提前预测损失、训练步数、token 数、临界批量及完整损失轨迹。
摘要：
缩放定律原理表明，损失与模型规模、数据集规模以及训练期间使用的计算资源等变量之间存在幂律相关性。这些原理在优化模型预训练的各个方面发挥着至关重要的作用，最终促成了 GPT-4、Llama 和 Gemini 等大型语言模型的成功。然而，OpenAI 最初的缩放定律论文并未披露推导精确缩放定律公式所需的完整细节，而且其结论仅基于参数量最高达 15 亿的模型。尽管后续一些工作试图揭示这些细节并扩展到更大的模型，但它们往往忽略了学习率、上下文长度和批量大小等重要因素对训练的依赖性，导致无法建立可靠的公式来预测测试损失轨迹。在本技术报告中，我们确认，当模型规模扩展至 330 亿参数时，OpenAI 原始论文提出的缩放定律表述仍然有效，但这些公式中的常数系数会随实验设置发生显著变化。我们细致地识别了影响因素，并提供透明、分步的说明，仅通过训练参数量为 100 万至 6000 万的模型来估计缩放定律公式中的所有常数项。利用这些估计出的公式，我们展示了在参数量最高达 33B 的模型训练之前准确预测其多种属性的能力，包括：（1）可能达到的最低测试损失；（2）达到特定损失所需的最少训练步数和已处理 token 数；（3）在任意损失值下具有最优时间/计算权衡的临界批量大小；以及（4）任意批量大小下完整的测试损失轨迹。
英文摘要：
Scaling law principles indicate a power-law correlation between loss and variables such as model size, dataset size, and computational resources utilized during training. These principles play a vital role in optimizing various aspects of model pre-training, ultimately contributing to the success of large language models such as GPT-4, Llama and Gemini. However, the original scaling law paper by OpenAI did not disclose the complete details necessary to derive the precise scaling law formulas, and their conclusions are only based on models containing up to 1.5 billion parameters. Though some subsequent works attempt to unveil these details and scale to larger models, they often neglect the training dependency of important factors such as the learning rate, context length and batch size, leading to their failure to establish a reliable formula for predicting the test loss trajectory. In this technical report, we confirm that the scaling law formulations proposed in the original OpenAI paper remain valid when scaling the model size up to 33 billion, but the constant coefficients in these formulas vary significantly with the experiment setup. We meticulously identify influential factors and provide transparent, step-by-step instructions to estimate all constant terms in scaling-law formulas by training on models with only 1M~60M parameters. Using these estimated formulas, we showcase the capability to accurately predict various attributes for models with up to 33B parameters before their training, including (1) the minimum possible test loss; (2) the minimum required training steps and processed tokens to achieve a specific loss; (3) the critical batch size with an optimal time/computation trade-off at any loss value; and (4) the complete test loss trajectory with arbitrary batch size.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
