arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.04332cs.CL
收藏

PPT:用于少样本学习的预先训练的提示微调

PPT: Pre-trained Prompt Tuning for Few-shot Learning

Tsinghua University(清华大学);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院)

Yuxian Gu, Xu Han, Zhiyuan Liu, Minlie Huang

摘要:

预训练语言模型(PLM)的测试通过弥合预训练任务和各种下游任务之间的差距,表现出显着的性能。在这些方法中,提示微调冻结PLM,只调整软提示,为大规模PLM适应下游任务提供了一个高效的解决方案。然而,提示微调还有待充分探索。在我们的试点实验中,我们发现,及时调整执行与传统的全模型微调时,下游数据是足够的,而它的表现差得多的少样本学习设置,这可能会阻碍提示微调在实践中的应用。我们将这种低性能归因于初始化软提示的方式。因此,在这项工作中,我们建议通过在预训练阶段添加软提示来预训练提示,以获得更好的初始化。我们将此预训练提示微调框架命名为“PPT”。为了保证PPT的通用性,我们将相似的分类任务制定为统一的任务形式,并为这个统一的任务预训练软提示。大量的实验表明,在全数据和Few-Shot设置下,为下游任务调整预训练的提示可以达到甚至超过全模型微调。我们的方法是有效和高效的,在实践中使用大型PLM。

英文摘要:

Prompts for pre-trained language models (PLMs) have shown remarkable performance by bridging the gap between pre-training tasks and various downstream tasks. Among these methods, prompt tuning, which freezes PLMs and only tunes soft prompts, provides an efficient and effective solution for adapting large-scale PLMs to downstream tasks. However, prompt tuning is yet to be fully explored. In our pilot experiments, we find that prompt tuning performs comparably with conventional full-model fine-tuning when downstream data are sufficient, whereas it performs much worse under few-shot learning settings, which may hinder the application of prompt tuning in practice. We attribute this low performance to the manner of initializing soft prompts. Therefore, in this work, we propose to pre-train prompts by adding soft prompts into the pre-training stage to obtain a better initialization. We name this Pre-trained Prompt Tuning framework "PPT". To ensure the generalization of PPT, we formulate similar classification tasks into a unified task form and pre-train soft prompts for this unified task. Extensive experiments show that tuning pre-trained prompts for downstream tasks can reach or even outperform full-model fine-tuning under both full-data and few-shot settings. Our approach is effective and efficient for using large-scale PLMs in practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑