arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.13586cs.CLcs.AIcs.LG
收藏

预训练、提示和预测:自然语言处理中的预训练方法系统综述

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Carnegie Mellon University(卡内基梅隆大学);National University of Singapore(新加坡国立大学)

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, Graham Neubig

摘要:

本文综述和组织了自然语言处理中一种新范式的研究工作,我们称之为“基于语义的学习”。与传统的监督学习不同,传统的监督学习训练模型接受输入x并预测输出y为P(y| x),基于概率的学习基于直接对文本的概率建模的语言模型。为了使用这些模型来执行预测任务,使用模板将原始输入x修改为具有一些未填充槽的文本字符串提示x',然后使用语言模型来概率性地填充未填充信息以获得最终字符串x,从该最终字符串x可以导出最终输出y。这个框架功能强大,具有吸引力,原因有很多:它允许语言模型在大量原始文本上进行预训练,并且通过定义新的提示函数,模型能够执行Few-Shot甚至零样本学习,适应具有很少或没有标记数据的新场景。在本文中,我们介绍了这个有前途的范例的基础知识,描述了一组统一的数学符号,可以覆盖各种现有的工作,并沿着几个维度组织现有的工作,例如选择预先训练的模型,提示和调整策略。为了使感兴趣的初学者更容易进入该领域,我们不仅对现有的作品进行了系统的回顾,并对基于概念的概念进行了高度结构化的类型学,而且还发布了其他资源,例如,一个网站http://pretrain.nlpedia.ai/,包括不断更新的调查,和paperlist。

英文摘要:

This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning". Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P(y|x), prompt-based learning is based on language models that model the probability of text directly. To use these models to perform prediction tasks, the original input x is modified using a template into a textual string prompt x' that has some unfilled slots, and then the language model is used to probabilistically fill the unfilled information to obtain a final string x, from which the final output y can be derived. This framework is powerful and attractive for a number of reasons: it allows the language model to be pre-trained on massive amounts of raw text, and by defining a new prompting function the model is able to perform few-shot or even zero-shot learning, adapting to new scenarios with few or no labeled data. In this paper we introduce the basics of this promising paradigm, describe a unified set of mathematical notations that can cover a wide variety of existing work, and organize existing work along several dimensions, e.g.the choice of pre-trained models, prompts, and tuning strategies. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured typology of prompt-based concepts, but also release other resources, e.g., a website http://pretrain.nlpedia.ai/ including constantly-updated survey, and paperlist.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑