arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.06133cs.CL
收藏

面向语义的大规模语言模型无标记启动

Semantic-Oriented Unlabeled Priming for Large-Scale Language Models

Technical University of Munich(慕尼黑工业大学);LMU Munich(慕尼黑大学)

Yanchen Liu, Timo Schick, Hinrich Schütze

摘要:

由于与微调大语言模型相关的高成本,最近的各种工作提出通过上下文学习使其适应特定任务而无需任何参数更新。不幸的是,对于上下文学习,目前还没有办法利用未标记的数据,这通常比标记的示例更容易获得大量数据。因此,在这项工作中,我们研究了如何利用未标记的示例来提高预训练语言模型的zero-shot性能,而无需任何微调:我们引入了面向语义的未标记启动(SOUP),这是一种通过检索语义相似的未标记示例来分类示例的方法,以zero-shot的方式为它们分配标签,然后将它们用于上下文学习。我们还提出了一个新的启动策略,即袋的上下文启动,更适合我们的设置,使更多的例子比适合到上下文窗口的使用。

英文摘要:

Due to the high costs associated with finetuning large language models, various recent works propose to adapt them to specific tasks without any parameter updates through in-context learning. Unfortunately, for in-context learning there is currently no way to leverage unlabeled data, which is often much easier to obtain in large quantities than labeled examples. In this work, we therefore investigate ways to make use of unlabeled examples to improve the zero-shot performance of pretrained language models without any finetuning: We introduce Semantic-Oriented Unlabeled Priming (SOUP), a method that classifies examples by retrieving semantically similar unlabeled examples, assigning labels to them in a zero-shot fashion, and then using them for in-context learning. We also propose bag-of-contexts priming, a new priming strategy that is more suitable for our setting and enables the usage of more examples than fit into the context window.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑