arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2203.03235cs.CLcs.AI
收藏

作为Few-Shot学习者的预先训练的Token替换检测模型

Pre-trained Token-replaced Detection Model as Few-shot Learner

Soochow University(苏州大学)

Zicheng Li, Shoushan Li, Guodong Zhou

摘要:

预训练的掩蔽语言模型已经证明了作为Few-Shot学习者的卓越能力。在本文中,作为一种替代方案,我们提出了一种新的方法来少样本学习与预先训练的令牌替换检测模型,如ELECTRA。在这种方法中,我们将分类或回归任务重新表述为标记替换检测问题。具体来说,我们首先为每个任务定义一个模板和标签描述词,并将它们放入输入中,形成一个自然语言提示。然后,我们采用预先训练的标记替换检测模型来预测哪个标签描述词是最原始的(即,最少替换)。在16个数据集上的系统评估表明,我们的方法在单句和两句学习任务中的性能优于使用预先训练的掩蔽语言模型的Few-Shot学习者。

英文摘要:

Pre-trained masked language models have demonstrated remarkable ability as few-shot learners. In this paper, as an alternative, we propose a novel approach to few-shot learning with pre-trained token-replaced detection models like ELECTRA. In this approach, we reformulate a classification or a regression task as a token-replaced detection problem. Specifically, we first define a template and label description words for each task and put them into the input to form a natural language prompt. Then, we employ the pre-trained token-replaced detection model to predict which label description word is the most original (i.e., least replaced) among all label description words in the prompt. A systematic evaluation on 16 datasets demonstrates that our approach outperforms few-shot learners with pre-trained masked language models in both one-sentence and two-sentence learning tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑