arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.07602cs.CL
收藏

P-Tuning v2:提示微调可以与跨尺度和任务的通用微调相媲美

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

Tsinghua University(清华大学);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院);Shanghai Qi Zhi Institute(上海期智研究院)

Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, Jie Tang

摘要:

提示微调仅使用冻结的语言模型来调整连续提示,这大大减少了训练时的每任务存储和内存使用。然而,在NLU的背景下,先前的工作表明,提示微调对于正常大小的预训练模型表现不佳。我们还发现,现有的提示微调方法不能处理硬序列标记任务,表明缺乏通用性。我们提出了一个新的经验发现,即适当优化的提示微调可以在广泛的模型规模和NLU任务中普遍有效。它匹配微调的性能,同时只有0.1%-3%的调整参数。我们的方法P-Tuning v2是针对NLU优化和调整的深度提示微调\cite{li2021prefix,qin2021learning}的实现。鉴于P-Tuning v2的通用性和简单性,我们相信它可以作为微调的替代方案,并为未来的研究提供强大的基线。我们的代码和数据发布在https://github.com/THUDM/P-tuning-v2上。

英文摘要:

Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU, prior work reveals that prompt tuning does not perform well for normal-sized pretrained models. We also find that existing methods of prompt tuning cannot handle hard sequence labeling tasks, indicating a lack of universality. We present a novel empirical finding that properly optimized prompt tuning can be universally effective across a wide range of model scales and NLU tasks. It matches the performance of finetuning while having only 0.1%-3% tuned parameters. Our method P-Tuning v2 is an implementation of Deep Prompt Tuning \cite{li2021prefix,qin2021learning} optimized and adapted for NLU. Given the universality and simplicity of P-Tuning v2, we believe it can serve as an alternative to finetuning and a strong baseline for future research.Our code and data are released at https://github.com/THUDM/P-tuning-v2.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑