arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2104.08786cs.CLcs.AI
收藏

奇妙的订单和在哪里找到它们:克服Few-Shot提示订单敏感性

Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity

University College London(伦敦大学学院);Mishcon de Reya LLP(米什康德雷亚律师事务所)

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, Pontus Stenetorp

摘要:

当只使用少量训练样本时,与完全监督的、微调的、大型的、预训练的语言模型相比,GPT-3等非常大的、预训练的语言模型表现出了竞争力。我们证明了提供样本的顺序可以在接近最先进的和随机猜测性能之间产生差异:本质上有些排列是“梦幻般的”,有些不是。我们详细分析了这一现象,确定:它存在于不同的模型大小(即使是当前最大的模型),它与特定的样本子集无关,并且一个模型的给定良好排列不能转移到另一个模型。虽然可以使用开发集来确定哪些排列是有效的,但这将偏离真正的Few-Shot设置,因为它需要额外的注释数据。相反,我们使用的生成性质的语言模型,构建一个人工的发展集和熵统计的基础上,这个集合上的候选排列,我们确定性能提示。我们的方法在11个不同的已建立的文本分类任务中对GPT家族模型产生了13%的相对改进。

英文摘要:

When primed with only a handful of training samples, very large, pretrained language models such as GPT-3 have shown competitive results when compared to fully-supervised, fine-tuned, large, pretrained language models. We demonstrate that the order in which the samples are provided can make the difference between near state-of-the-art and random guess performance: essentially some permutations are "fantastic" and some not. We analyse this phenomenon in detail, establishing that: it is present across model sizes (even for the largest current models), it is not related to a specific subset of samples, and that a given good permutation for one model is not transferable to another. While one could use a development set to determine which permutations are performant, this would deviate from the true few-shot setting as it requires additional annotated data. Instead, we use the generative nature of language models to construct an artificial development set and based on entropy statistics of the candidate permutations on this set, we identify performant prompts. Our method yields a 13% relative improvement for GPT-family models across eleven different established text classification tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑