arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.04824cs.CL
收藏

AdaPrompt:基于自适应的NLP模型训练

AdaPrompt: Adaptive Model Training for Prompt-based NLP

Microsoft Research(微软研究院);Zhejiang University(浙江大学);Westlake University(西湖大学);Westlake Institute for Advanced Study(西湖高等研究院)

Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, Yue Zhang

摘要:

基于机器学习的NLP方法以其能够处理zero-shot和Few-Shot的NLP任务而受到了广泛的关注。其主要思想是通过将NLP下游任务映射到自然语言提示中,然后由预训练的语言模型(PLM)填充,从而弥合NLP下游任务和语言建模(LM)之间的差距。然而,对于快速学习,NLP任务和预训练之间仍然存在两个明显的差距。首先,提示信息不一定充分存在于LM预训练期间。第二,特定于任务的数据在预训练期间不一定得到很好的表示。我们解决这两个问题,提出AdaPrompt,自适应地检索外部数据,利用任务和提示的特点,不断预训练的PLM。此外,我们利用自然语言推理模型中的知识来获得自适应的动词。在五个NLP基准测试上的实验结果表明,AdaPrompt在Few-Shot设置中可以比标准PLM有所改进。此外,在zero-shot设置,我们的方法优于标准的基于小波变换的方法高达26.35\%的相对误差减少。

英文摘要:

Prompt-based learning, with its capability to tackle zero-shot and few-shot NLP tasks, has gained much attention in community. The main idea is to bridge the gap between NLP downstream tasks and language modeling (LM), by mapping these tasks into natural language prompts, which are then filled by pre-trained language models (PLMs). However, for prompt learning, there are still two salient gaps between NLP tasks and pretraining. First, prompt information is not necessarily sufficiently present during LM pretraining. Second, task-specific data are not necessarily well represented during pretraining. We address these two issues by proposing AdaPrompt, adaptively retrieving external data for continual pretraining of PLMs by making use of both task and prompt characteristics. In addition, we make use of knowledge in Natural Language Inference models for deriving adaptive verbalizers. Experimental results on five NLP benchmarks show that AdaPrompt can improve over standard PLMs in few-shot settings. In addition, in zero-shot settings, our method outperforms standard prompt-based methods by up to 26.35\% relative error reduction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑