arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.15936cs.LGcs.AIcs.CLstat.ML
收藏

语言模型中复杂技能涌现的理论

A Theory for Emergence of Complex Skills in Language Models

Princeton University(普林斯顿大学);Google DeepMind(谷歌DeepMind)

Sanjeev Arora, Anirudh Goyal

AI总结:

针对大语言模型规模扩大时技能涌现的机制解释难题,本文基于Scaling Laws和统计框架开展分析,提出关联损失与基础技能的统计框架,发现“弹弓泛化”现象并验证k元组复杂技能与基础技能涌现规律一致。

摘要:

当今AI产品发展的一个主要驱动力是,当语言模型的参数规模和训练语料库扩大时,模型中会涌现出新技能。目前人们对这一现象的理解仍很有限,且通过对基于梯度的训练进行数学分析来给出机制层面的解释似乎难度很大。本文采用了不同的研究思路,利用著名的(且经实证验证的)大语言模型Scaling Laws(缩放定律)以及一个简单的统计框架来分析技能涌现现象。本文的贡献包括:(a)提出了一个统计框架,将大语言模型的交叉熵损失与语言任务所依赖的基础技能的胜任程度关联起来。(b)数学分析表明,Scaling Laws隐含着一种强形式的归纳偏置,能让预训练模型实现极高效率的学习。我们将其非正式地称为“弹弓泛化(slingshot generalization)”,因为从直观来看,它所带来的技能胜任水平似乎违背了常规的泛化理论。(c)给出了弹弓泛化的一个关键例证:执行涉及k元组技能组合的任务的胜任程度,其涌现的缩放阶段和速率与基础技能本身的胜任程度基本一致。

英文摘要:

A major driver of AI products today is the fact that new skills emerge in language models when their parameter set and training corpora are scaled up. This phenomenon is poorly understood, and a mechanistic explanation via mathematical analysis of gradient-based training seems difficult. The current paper takes a different approach, analysing emergence using the famous (and empirical) Scaling Laws of LLMs and a simple statistical framework. Contributions include: (a) A statistical framework that relates cross-entropy loss of LLMs to competence on the basic skills that underlie language tasks. (b) Mathematical analysis showing that the Scaling Laws imply a strong form of inductive bias that allows the pre-trained model to learn very efficiently. We informally call this {\em slingshot generalization} since naively viewed it appears to give competence levels at skills that violate usual generalization theory. (c) A key example of slingshot generalization, that competence at executing tasks involving $k$-tuples of skills emerges essentially at the same scaling and same rate as competence on the elementary skills themselves.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑