arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2207.08141cs.CL
收藏

ELECTRA也是Zero-Shot学习者

ELECTRA is a Zero-Shot Learner, Too

National Cheng Kung University(国立成功大学)

Shiwen Ni, Hung-Yu Kao

摘要:

近年来,对于Few-Shot甚至零样本学习,“预训练、提示、预测”的新范式与“预训练、微调”范式相比,取得了显著的成就。在基于XML的GPT-3成功之后,一系列基于掩码语言模型(MLM)的(例如,BERT,RoBERTa)的快速学习方法变得流行并被广泛使用。然而,另一种有效的预训练判别模型ELECTRA可能被忽视了。在本文中,我们试图完成几个自然语言处理任务,在zero-shot的情况下,使用一种新的我们提出的替代令牌检测(RTD)为基础的提示学习方法。实验结果表明,基于RTD提示学习的ELECTRA模型实现了令人惊讶的最先进的zero-shot性能。从数字上看,与MLM-RoBERTa-large和MLM-BERT-large相比,我们的RTD-ELECTRA-large在所有15个任务上的平均改进分别约为8.4%和13.7%。特别是在SST-2任务中,我们的RTD-ELECTRA-large在没有任何训练数据的情况下达到了惊人的90.1%的准确率。总体而言,与预训练的掩蔽语言模型相比,预训练的替换标记检测模型在零样本学习中表现更好。源代码可在https://github.com/nishiwen1214/RTD-ELECTRA上获得。

英文摘要:

Recently, for few-shot or even zero-shot learning, the new paradigm "pre-train, prompt, and predict" has achieved remarkable achievements compared with the "pre-train, fine-tune" paradigm. After the success of prompt-based GPT-3, a series of masked language model (MLM)-based (e.g., BERT, RoBERTa) prompt learning methods became popular and widely used. However, another efficient pre-trained discriminative model, ELECTRA, has probably been neglected. In this paper, we attempt to accomplish several NLP tasks in the zero-shot scenario using a novel our proposed replaced token detection (RTD)-based prompt learning method. Experimental results show that ELECTRA model based on RTD-prompt learning achieves surprisingly state-of-the-art zero-shot performance. Numerically, compared to MLM-RoBERTa-large and MLM-BERT-large, our RTD-ELECTRA-large has an average of about 8.4% and 13.7% improvement on all 15 tasks. Especially on the SST-2 task, our RTD-ELECTRA-large achieves an astonishing 90.1% accuracy without any training data. Overall, compared to the pre-trained masked language models, the pre-trained replaced token detection model performs better in zero-shot learning. The source code is available at: https://github.com/nishiwen1214/RTD-ELECTRA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑