arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.03980cs.CLcs.AI
收藏

Transformer 实现自适应部分池化

Transformers perform adaptive partial pooling

University of Oregon(俄勒冈大学)

Vsevolod Kapatsinski

AI总结:

本文研究了Transformer模型在训练过程中自适应部分池化特性,发现其下一个词预测随着训练进行逐渐减少对外部观察的依赖,且受情境频率、数量和变化影响。

摘要:

因为语言具有创造性,任何合理的语言模型都必须具备泛化能力,在新情境中决定说什么,通过利用相似情境的信息做出决策。但当情境不是新情境而是仅仅不常见时又如何处理?在分层回归中,模型对情境行为的预测会受到其他相似情境观察的影响,程度取决于两个因素:1)当前情境不常见;2)不同情境行为相似。这被称为自适应部分证据池化。本文表明,Transformer(GPT2)的下一个词预测在训练过程中,随着训练轮次的增加,对当前情境外部观察的依赖性逐渐降低(池化程度随训练减少),并且池化的程度受情境频率、情境数量(类型频率)和情境变化的影响,与分层回归类似。这些Transformer学习的特征从理性与实证角度都被认为是现实的。

英文摘要:

Any language model must decide what to say in novel contexts based on information from similar contexts. But what about contexts that are not novel but merely infrequent? In hierarchical regression, the model's predictions for behavior in a context are affected by observations from similar contexts to the extent that 1) the current context is infrequent and 2) different contexts behave similarly. This is called adaptive partial pooling. This paper shows that next-word predictions of a transformer (GPT2) are affected by observations from outside the current context, but this pooling reduces with more training. Pooling is affected by context frequency, context number (type frequency) and context variability in a qualitatively similar way to hierarchical regression. However, there is a "sweet spot" in training at which the transformer best matches the behavior of hierarchical regression. This is the point at which the effect of context frequency on pooling is at maximum.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑