arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.03016cs.LGcs.CL
收藏

通过学习学习离散函数来理解 Transformer 与 LLM 中的上下文学习

Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions

University of Oxford(牛津大学);Mila(米拉研究所);McGill University(麦吉尔大学);Cohere(Cohere公司)

Satwik Bhattamishra, Arkil Patel, Phil Blunsom, Varun Kanade

AI总结:

本文通过 Boolean 函数学习测试平台分析 Transformer 与 LLM 的上下文学习能力,揭示任务复杂度、无注意力模型、教学序列及算法自适应选择的影响,并验证预训练 LLM 的分布外预测竞争力。

摘要:

为了理解上下文学习现象,近期研究采用了一种程式化实验框架,并证明 Transformer 能够为各类实值函数学习基于梯度的学习算法。然而,Transformer 在实现学习算法方面的局限,以及它们学习其他形式算法的能力,尚未得到充分理解。此外,这些能力在多大程度上仅限于基于注意力的模型,也不清楚。进一步而言,从这些程式化设定中得出的洞见能否推广到预训练大语言模型(LLM),仍有待观察。在本工作中,我们通过展示以下结果,向回答这些问题迈出一步:(a)在包含多种 Boolean 函数类别的测试平台上,我们发现 Transformer 在较“简单”的任务上几乎可以达到最优学习算法的水平,而在更“复杂”的任务上性能会下降。此外,我们发现某些无注意力模型在一系列任务上的表现与 Transformer(几乎)相同。(b)当提供教学序列(即一组能够唯一确定某一函数类别中某个函数的示例)时,我们表明 Transformer 能以更高的样本效率进行学习。有趣的是,我们的结果表明,Transformer 可以学会实现两种不同算法来解决同一任务,并能根据上下文示例序列自适应地选择样本效率更高的算法。(c)最后,我们表明,现有 LLM(例如 LLaMA-2、GPT-4)在保证不会出现在其训练集中的预测任务上,能够与最近邻基线竞争。

英文摘要:

In order to understand the in-context learning phenomenon, recent works have adopted a stylized experimental framework and demonstrated that Transformers can learn gradient-based learning algorithms for various classes of real-valued functions. However, the limitations of Transformers in implementing learning algorithms, and their ability to learn other forms of algorithms are not well understood. Additionally, the degree to which these capabilities are confined to attention-based models is unclear. Furthermore, it remains to be seen whether the insights derived from these stylized settings can be extrapolated to pretrained Large Language Models (LLMs). In this work, we take a step towards answering these questions by demonstrating the following: (a) On a test-bed with a variety of Boolean function classes, we find that Transformers can nearly match the optimal learning algorithm for 'simpler' tasks, while their performance deteriorates on more 'complex' tasks. Additionally, we find that certain attention-free models perform (almost) identically to Transformers on a range of tasks. (b) When provided a teaching sequence, i.e. a set of examples that uniquely identifies a function in a class, we show that Transformers learn more sample-efficiently. Interestingly, our results show that Transformers can learn to implement two distinct algorithms to solve a single task, and can adaptively select the more sample-efficient algorithm depending on the sequence of in-context examples. (c) Lastly, we show that extant LLMs, e.g. LLaMA-2, GPT-4, can compete with nearest-neighbor baselines on prediction tasks that are guaranteed to not be in their training set.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑