arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.11004cs.LG
收藏

统计归纳头的演化:上下文学习马尔可夫链

The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains

Harvard University(哈佛大学);University of Pennsylvania(宾夕法尼亚大学);NYU(纽约大学)

Benjamin L. Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, Nikolaos Tsilivis

AI总结:

通过引入马尔可夫链序列建模任务,研究大型语言模型中上下文学习能力的涌现机制,发现Transformer在训练中形成统计归纳头,经历从均匀预测到一元再到二元统计的快速相变,揭示了层间交互及简单解对复杂解形成的延迟影响。

摘要:

大型语言模型具有生成模仿其输入模式的文本的能力。我们引入了一个简单的马尔可夫链序列建模任务,以研究这种上下文学习(ICL)能力是如何涌现的。在我们的设置中,每个样本都是从马尔可夫链先验分布中抽取的马尔可夫链生成的。在此任务上训练的Transformers形成了“统计归纳头”,能够根据上下文的二元统计量计算准确的下一个token概率。在训练过程中,模型经历了多个阶段:在初始阶段预测是均匀的,随后模型学会次优地使用上下文单token统计量(一元语法)进行预测;接着,出现向正确的上下文二元语法解决方案的快速相变。我们对这一多阶段过程进行了实证和理论调查,展示了成功的学习如何源于transformer各层之间的交互,并揭示了较简单的一元语法解决方案的存在可能会延迟最终二元语法解决方案的形成。我们研究了改变马尔可夫链先验分布如何影响学习,并考虑了将上下文学习马尔可夫链(ICL-MC)任务推广到n>2的n元语法的情况。

英文摘要:

Large language models have the ability to generate text that mimics patterns in their inputs. We introduce a simple Markov Chain sequence modeling task in order to study how this in-context learning (ICL) capability emerges. In our setting, each example is sampled from a Markov chain drawn from a prior distribution over Markov chains. Transformers trained on this task form \emph{statistical induction heads} which compute accurate next-token probabilities given the bigram statistics of the context. During the course of training, models pass through multiple phases: after an initial stage in which predictions are uniform, they learn to sub-optimally predict using in-context single-token statistics (unigrams); then, there is a rapid phase transition to the correct in-context bigram solution. We conduct an empirical and theoretical investigation of this multi-phase process, showing how successful learning results from the interaction between the transformer's layers, and uncovering evidence that the presence of the simpler unigram solution may delay formation of the final bigram solution. We examine how learning is affected by varying the prior distribution over Markov chains, and consider the generalization of our in-context learning of Markov chains (ICL-MC) task to $n$-grams for $n > 2$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑