arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.05313cs.CLcs.AI
收藏

面向Few-Shot文本分类的统一提示微调

Towards Unified Prompt Tuning for Few-shot Text Classification

East China Normal University(华东师范大学);Alibaba Group(阿里巴巴集团);Zhejiang Lab(之江实验室);Ant Group(蚂蚁集团)

Jianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan, Minghui Qiu, Fei Yang, Qiuhui Shi, Songfang Huang, Ming Gao

摘要:

基于任务的微调通过采用特定于任务的提示,提高了预训练语言模型(PLM)在Few-Shot文本分类上的性能。然而,PLM在预训练期间不熟悉机器学习风格的表达,这限制了下游任务的少样本学习性能。如果模型能够在适应特定的NLP任务之前获得一些提示性知识,那将是可取的。我们提出了统一提示微调(UPT)框架,通过显式捕获非目标NLP数据集的提示语义,为BERT风格的模型提供更好的Few-Shot文本分类。在UPT中,提出了一种新的范式,用于跨不同NLP任务的联合提示学习,迫使PLM捕获任务不变的提示知识。我们进一步设计了一个名为知识增强的选择性掩蔽语言建模的自监督任务,以提高PLM的泛化能力,以准确适应以前看不见的任务。在跨多个任务进行多任务学习之后,PLM可以更好地针对低资源设置中的任何不同目标任务进行自动调整。对各种NLP任务的实验表明,UPT在基于神经网络的微调方面始终优于最先进的技术。

英文摘要:

Prompt-based fine-tuning has boosted the performance of Pre-trained Language Models (PLMs) on few-shot text classification by employing task-specific prompts. Yet, PLMs are unfamiliar with prompt-style expressions during pre-training, which limits the few-shot learning performance on downstream tasks. It would be desirable if the models can acquire some prompting knowledge before adaptation to specific NLP tasks. We present the Unified Prompt Tuning (UPT) framework, leading to better few-shot text classification for BERT-style models by explicitly capturing prompting semantics from non-target NLP datasets. In UPT, a novel paradigm Prompt-Options-Verbalizer is proposed for joint prompt learning across different NLP tasks, forcing PLMs to capture task-invariant prompting knowledge. We further design a self-supervised task named Knowledge-enhanced Selective Masked Language Modeling to improve the PLM's generalization abilities for accurate adaptation to previously unseen tasks. After multi-task learning across multiple tasks, the PLM can be better prompt-tuned towards any dissimilar target tasks in low-resourced settings. Experiments over a variety of NLP tasks show that UPT consistently outperforms state-of-the-arts for prompt-based fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑