arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.12837cs.CLcs.AI
收藏

反思演示的作用:是什么让上下文学习有效?

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

Meta AI;Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学)

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer

摘要:

大语言模型(LM)能够在上下文中学习--仅通过对一些输入-标签对(演示)进行条件化并对新输入进行预测来通过推理执行新任务。然而,人们对模型如何学习以及演示的哪些方面有助于最终任务性能的理解很少。在本文中,我们表明,地面真相演示实际上是不需要的-在演示中随机替换标签几乎不会影响一系列分类和多选择任务的性能,在包括GPT-3在内的12个不同模型中始终如此。相反,我们发现演示的其他方面是最终任务性能的关键驱动因素,包括它们提供了以下几个示例:(1)标签空间,(2)输入文本的分布,以及(3)序列的整体格式。总之,我们的分析提供了一种理解上下文学习如何以及为什么有效的新方法,同时也提出了一些新问题,即仅仅通过推理就可以从大语言模型中学到多少东西。

英文摘要:

Large language models (LMs) are able to in-context learn -- perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrations contribute to end task performance. In this paper, we show that ground truth demonstrations are in fact not required -- randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choce tasks, consistently over 12 different models including GPT-3. Instead, we find that other aspects of the demonstrations are the key drivers of end task performance, including the fact that they provide a few examples of (1) the label space, (2) the distribution of the input text, and (3) the overall format of the sequence. Together, our analysis provides a new way of understanding how and why in-context learning works, while opening up new questions about how much can be learned from large language models through inference alone.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑