基于预训练语言模型的原型提示动词化知识获取
Eliciting Knowledge from Pretrained Language Models for Prototypical Prompt Verbalizer
摘要：
最近的进展，快速调整铸造Few-Shot分类任务作为一个掩蔽的语言建模问题。通过将输入包装到模板中并使用在标签空间和标签词空间之间构造映射的动词化器，自动调优可以在zero-shot和Few-Shot场景中获得优异的结果。然而，典型的自动调优需要手动设计的语言化器，这需要领域专业知识和人工努力。标签空间不足可能会给结果带来相当大的偏差。在本文中，我们专注于从预训练的语言模型中获取知识，并提出了一个原型提示动词化器的自动调优。标签由特征空间中的原型嵌入表示，而不是由离散的单词表示。在输入的掩蔽位置处的嵌入和原型嵌入之间的距离被用作分类标准。对于zero-shot设置，通过手动设计的模板从预训练的语言模型中引出知识以形成初始原型嵌入。对于Few-Shot设置，调整模型以学习有意义和可解释的原型嵌入。我们的方法通过对比学习优化模型。在几个低资源环境下的多类文本分类数据集上的实验结果表明，与其他动词化器构造方法相比，该方法是有效的。我们的实现可以在https://github.com/Ydongd/prototypical-prompt-verbalizer上获得。
英文摘要：
Recent advances on prompt-tuning cast few-shot classification tasks as a masked language modeling problem. By wrapping input into a template and using a verbalizer which constructs a mapping between label space and label word space, prompt-tuning can achieve excellent results in zero-shot and few-shot scenarios. However, typical prompt-tuning needs a manually designed verbalizer which requires domain expertise and human efforts. And the insufficient label space may introduce considerable bias into the results. In this paper, we focus on eliciting knowledge from pretrained language models and propose a prototypical prompt verbalizer for prompt-tuning. Labels are represented by prototypical embeddings in the feature space rather than by discrete words. The distances between the embedding at the masked position of input and prototypical embeddings are used as classification criterion. For zero-shot settings, knowledge is elicited from pretrained language models by a manually designed template to form initial prototypical embeddings. For few-shot settings, models are tuned to learn meaningful and interpretable prototypical embeddings. Our method optimizes models by contrastive learning. Extensive experimental results on several many-class text classification datasets with low-resource settings demonstrate the effectiveness of our approach compared with other verbalizer construction methods. Our implementation is available at https://github.com/Ydongd/prototypical-prompt-verbalizer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
