arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.10424cs.CLcs.AI
收藏

Pelican Soup框架:语言模型能力的一种理论框架

Pelican Soup Framework: A Theoretical Framework for Language Model Capabilities

University of Southern California(南加州大学)

Ting-Rui Chiang, Dani Yogatama

AI总结:

Pelican Soup框架通过引入知识库和参考意义关联,探讨语言模型在预训练和上下文学习中的能力,并推导出相关理论界。

摘要:

在本文中,我们提出了一个简单的理论框架,即Pelican Soup,旨在更好地理解预训练如何使大语言模型(LLMs)能够(1)泛化到未见过的指令,以及(2)在上下文学习中表现良好,即使在 verbalizers 与任务无关的情况下。为此,我们在我们的框架中引入了“知识库”和“参考意义关联”以及一种简单的自然语言处理任务形式化方法。我们的框架展示了语言学、心理学和哲学研究如何帮助我们理解语言模型,并与几种现有的理论结果相关联。作为我们框架使用的示例,我们利用该框架推导出上下文学习损失的界。最后,我们通过实验证实了我们的框架,并提出了可能的未来研究方向。

英文摘要:

In this work, we propose a simple theoretical framework, Pelican Soup, aiming to better understand how pretraining allows LLMs to (1) generalize to unseen instructions and (2) perform in-context learning, even when the verbalizers are irrelevant to the task. To this end, in our framework, we introduce the notion of "knowledge base" and "reference-sense association" and a simple formalism for natural language processing tasks. Our framework demonstrates how linguistic, psychology, and philosophy studies can inform our understanding of the language model and is connected to several other existing theoretical results. As an illustration of the usage of our framework, we derive a bound on in-context learning loss with our framework. Finally, we support our framework with empirical experiments and provide possible future research directions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑