arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.10463cs.CLcs.AIcs.LG
收藏

语言模型的临界数据规模:从顿悟视角

Critical Data Size of Language Models from a Grokking Perspective

Shanghai Jiao Tong University(上海交通大学);University of Edinburgh(爱丁堡大学);Tsinghua University(清华大学)

Xuekai Zhu, Yao Fu, Bowen Zhou, Zhouhan Lin

AI总结:

本研究从顿悟视角探索语言模型的临界数据规模,提出数据效率假说,并通过实验验证了临界点随模型规模增大而增大,揭示数据在语言模型学习机制中的关键作用。

摘要:

我们探索了语言模型中的临界数据规模,这是一个标志着从快速记忆到缓慢泛化根本性转变的阈值。我们在顿悟配置下将相变形式化为数据效率假说,并识别出语言模型训练动态中的数据不足、充分和过剩状态。我们开发了一种顿悟配置,通过重新缩放初始化和权重衰减,在简单语言模型上稳定地重现顿悟现象。我们表明,只有当语言模型达到临界规模时,泛化才会发生。我们从样本维度和模型维度分析顿悟,验证了所提出的数据效率假说。我们的实验揭示了在语言数据集的临界数据集规模处发生的更平滑的相变。随着模型规模的增大,这一临界点也变大,表明更大的模型需要更多的数据。我们的结果加深了对语言模型训练的理解,为数据在语言模型学习机制中的作用提供了新颖视角。

英文摘要:

We explore the critical data size in language models, a threshold that marks a fundamental shift from quick memorization to slow generalization. We formalize the phase transition under the grokking configuration into the Data Efficiency Hypothesis and identify data insufficiency, sufficiency, and surplus regimes in language models training dynamics. We develop a grokking configuration to reproduce grokking on simplistic language models stably by rescaling initialization and weight decay. We show that generalization occurs only when language models reach a critical size. We analyze grokking across sample-wise and model-wise, verifying the proposed data efficiency hypothesis. Our experiments reveal smoother phase transitions occurring at the critical dataset size for language datasets. As the model size increases, this critical point also becomes larger, indicating that larger models require more data. Our results deepen the understanding of language model training, offering a novel perspective on the role of data in the learning mechanism of language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑