arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.17563cs.LGcond-mat.dis-nnstat.ML
收藏

多任务稀疏奇偶性问题中涌现与缩放律的精确可解模型

An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem

Rudolf Peierls Centre for Theoretical Physics, University of Oxford(鲁道夫·皮尔士理论物理中心,牛津大学);Center for Quantum Structures in Modules and Spaces, Seoul National University(模块与空间中的量子结构中心,首尔国立大学)

Yoonsoo Nam, Nayara Fonseca, Seok Hyeong Lee, Chris Mingard, Ard A. Louis

AI总结:

本文提出一个将技能表示为基函数的精确可解多线性模型,推导了新技能涌现与损失缩放律的解析表达式,并成功用单一参数拟合了多任务稀疏奇偶性神经网络的涌现现象。

摘要:

深度学习模型在训练时间、训练数据或模型规模增加时,会表现出似乎是突然具备解决新问题能力的现象,这被称为涌现。本文提出了一个框架,其中每种新能力(即一项技能)表示为一个基函数。我们在该技能基下求解一个简单的多线性模型,找到了新技能涌现的解析表达式,以及损失函数随训练时间、数据规模、模型规模和最优计算量变化的缩放律。我们将详细计算与直接模拟结果进行了比较,模拟对象为在多任务稀疏奇偶性上训练的两层神经网络,其中数据集中的任务按幂律分布。我们的简单模型仅用单一拟合参数,就捕捉到了随着训练时间、数据规模或模型规模增加,神经网络中多项新技能呈现S型涌现的现象。

英文摘要:

Deep learning models can exhibit what appears to be a sudden ability to solve a new problem as training time, training data, or model size increases, a phenomenon known as emergence. In this paper, we present a framework where each new ability (a skill) is represented as a basis function. We solve a simple multi-linear model in this skill-basis, finding analytic expressions for the emergence of new skills, as well as for scaling laws of the loss with training time, data size, model size, and optimal compute. We compare our detailed calculations to direct simulations of a two-layer neural network trained on multitask sparse parity, where the tasks in the dataset are distributed according to a power-law. Our simple model captures, using a single fit parameter, the sigmoidal emergence of multiple new skills as training time, data size or model size increases in the neural network.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑