arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.05164cs.LGcs.AIcs.NE
收藏

一种面向神经缩放定律的资源模型

A Resource Model For Neural Scaling Law

Department of Physics, MIT(麻省理工学院物理系);IAIFI(人工智能与基础交互研究所)

Jinyeop Song, Ziming Liu, Max Tegmark, Jeff Gore

AI总结:

本研究提出一种神经缩放资源模型,基于复合任务可分解为竞争神经元资源的子任务的假设,通过玩具问题实证得出子任务损失与分配神经元成反比、资源占比随模型扩容保持恒定的规律,成功复现Chinchilla模型的缩放定律,为神经网络表征与诊断提供新工具。

摘要:

神经缩放定律刻画了模型性能随模型规模扩大而提升的规律。受实证观察启发,我们提出了一种神经缩放的资源模型。任务通常是复合性的,因此可分解为多个子任务,这些子任务会竞争资源(资源以分配给各子任务的神经元数量衡量)。在玩具问题上,我们通过实证发现:(1)子任务的损失与其分配的神经元数量成反比。(2)当复合任务中存在多个子任务时,随着模型变大,每个子任务获得的资源会均匀增长,且各子任务获得的资源占比保持恒定。我们假设这些发现具有普适性,并构建了一个模型来预测通用复合任务的神经缩放定律,该模型成功复现了arXiv:2203.15556中报道的Chinchilla模型的神经缩放定律。我们认为本文提出的资源概念将成为表征和诊断神经网络的有用工具。

英文摘要:

Neural scaling laws characterize how model performance improves as the model size scales up. Inspired by empirical observations, we introduce a resource model of neural scaling. A task is usually composite hence can be decomposed into many subtasks, which compete for resources (measured by the number of neurons allocated to subtasks). On toy problems, we empirically find that: (1) The loss of a subtask is inversely proportional to its allocated neurons. (2) When multiple subtasks are present in a composite task, the resources acquired by each subtask uniformly grow as models get larger, keeping the ratios of acquired resources constants. We hypothesize these findings to be generally true and build a model to predict neural scaling laws for general composite tasks, which successfully replicates the neural scaling law of Chinchilla models reported in arXiv:2203.15556. We believe that the notion of resource used in this paper will be a useful tool for characterizing and diagnosing neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑