路由语言模型的统一缩放定律
Unified Scaling Laws for Routed Language Models
摘要：
一个语言模型的性能已被证明是有效的建模为幂律在其参数计数。在这里，我们研究的路由网络的缩放行为：架构，有条件地使用其参数的一个子集，而处理输入。对于这些模型，参数数量和计算需求形成了两个独立的轴，沿着这两个轴，参数数量的增加会导致更好的性能。在这项工作中，我们推导出并证明这两个变量上定义的缩放定律，概括了那些已知的标准语言模型，并描述了通过三种不同的技术训练的路由架构的广泛的性能。之后，我们提供了这些法律的两个应用程序：首先推导出一个有效的参数计数沿所有模型以相同的速率扩展，然后使用的缩放系数，给出一个定量比较的三个路由技术考虑。我们的分析来自对五个数量级的路由网络的广泛评估，包括数百名专家和数千亿参数的模型。
英文摘要：
The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: architectures that conditionally use only a subset of their parameters while processing an input. For these models, parameter count and computational requirement form two independent axes along which an increase leads to better performance. In this work we derive and justify scaling laws defined on these two variables which generalize those known for standard language models and describe the performance of a wide range of routing architectures trained via three different techniques. Afterwards we provide two applications of these laws: first deriving an Effective Parameter Count along which all models scale at the same rate, and then using the scaling coefficients to give a quantitative comparison of the three routing techniques considered. Our analysis derives from an extensive evaluation of Routing Networks across five orders of magnitude of size, including models with hundreds of experts and hundreds of billions of parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
