WARP:字级对抗ReProgramming
WARP: Word-level Adversarial ReProgramming
摘要:
从预训练的语言模型进行迁移学习最近成为解决许多NLP任务的主要方法。用于多个任务的迁移学习的最大化参数共享的常见方法在语言模型之上训练一个或多个任务特定层。在本文中,我们提出了一种基于对抗性重编程的替代方法,该方法扩展了自动提示生成的早期工作。对抗性重编程试图学习特定于任务的词嵌入,当这些词嵌入连接到输入文本时,指示语言模型解决指定的任务。每个任务使用多达25 K个可训练参数,这种方法在GLUE基准测试的公共排行榜上具有多达25 M个可训练参数,优于所有现有方法。我们的方法初始化了特定于任务的人类可读提示,也适用于Few-Shot设置,在两个SuperGLUE任务上的表现优于GPT-3,只有32个训练样本。
英文摘要:
Transfer learning from pretrained language models recently became the dominant approach for solving many NLP tasks. A common approach to transfer learning for multiple tasks that maximize parameter sharing trains one or more task-specific layers on top of the language model. In this paper, we present an alternative approach based on adversarial reprogramming, which extends earlier work on automatic prompt generation. Adversarial reprogramming attempts to learn task-specific word embeddings that, when concatenated to the input text, instruct the language model to solve the specified task. Using up to 25K trainable parameters per task, this approach outperforms all existing methods with up to 25M trainable parameters on the public leaderboard of the GLUE benchmark. Our method, initialized with task-specific human-readable prompts, also works in a few-shot setting, outperforming GPT-3 on two SuperGLUE tasks with just 32 training samples.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。