Prompts for pre-trained language models (PLMs) have shown remarkable performance by bridging the gap between pre-training tasks and various downstream tasks. Among these methods, prompt tuning, which freezes PLMs and only tunes soft prompts, provides an efficient and effective solution for adapting large-scale PLMs to downstream tasks. However, prompt tuning is yet to be fully explored. In our pilot experiments, we find that prompt tuning performs comparably with conventional full-model fine-tuning when downstream data are sufficient, whereas it performs much worse under few-shot learning settings, which may hinder the application of prompt tuning in practice. We attribute this low performance to the manner of initializing soft prompts. Therefore, in this work, we propose to pre-train prompts by adding soft prompts into the pre-training stage to obtain a better initialization. We name this Pre-trained Prompt Tuning framework "PPT". To ensure the generalization of PPT, we formulate similar classification tasks into a unified task form and pre-train soft prompts for this unified task. Extensive experiments show that tuning pre-trained prompts for downstream tasks can reach or even outperform full-model fine-tuning under both full-data and few-shot settings. Our approach is effective and efficient for using large-scale PLMs in practice.
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);Alibaba Cloud Computing(阿里巴巴云计算);Shanghai Jiao Tong University(上海交通大学);College of Software, Zhejiang University(浙江大学软件学院)
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Singapore Management University(新加坡管理大学);Southwest Jiaotong University(西南交通大学);Singapore University of Technology and Design(新加坡科技设计大学);East China Normal University(华东师范大学)
Zero-Shot Chain-of-Thought Reasoning Guided by Evolutionary Algorithms in Large Language Models
School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);National Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);Key Laboratory of Machine Perceptron (MOE), Peking University(北京大学机器感知教育部重点实验室)
Metacognition-Enhanced Few-Shot Prompting With Positive Reinforcement
Institute of AI for Education, East China Normal University(华东师范大学教育人工智能研究院);School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Shanghai Changning Mental Health Center(上海市长宁区精神卫生中心)
STPrompt: Semantic-guided and Task-driven prompts for Effective Few-shot Classification
University of Chinese Academy of Sciences(中国科学院大学);Institute of Information Engineering(信息工程研究所);Beijing Institute of Technology(北京理工大学);Guangzhou University(广州大学)
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Institute for Clarity in Documentation(文档清晰性研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗克库尔中心);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室)
Evaluating Prompts Across Multiple Choice Tasks In a Zero-Shot Setting
大语言模型已经表明,通过自然语言提示可以实现令人印象深刻的zero-shot性能(Radford et al.,2019; Brown等人,2020; Sanh等人,2021年)。然而,创建一个有效的提示需要大量的尝试和错误。这就向\textit{prompts}提出了一个问题:提示的质量如何影响它的表现?为此,我们从各种任务中收集并标准化提示,以用于它们不是为之设计的任务。然后,我们在固定的多项选择数据集上评估这些提示,以定量分析提示的某些属性如何影响性能。我们发现,包括选择和使用提示不使用在预培训提供了显着的改善。所有的实验和代码都可以在https://github.com/gabeorlanski/zero-shot-cross-task上找到。
LAMPER: LanguAge Model and Prompt EngineeRing for zero-shot time series classification
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学清华大学深圳国际研究生院);School of Automation, Wuhan University of Technology(武汉理工大学自动化学院)
PromptFix: Few-shot Backdoor Removal via Adversarial Prompt Tuning
School of Information Science & Technology, Pennsylvania State University(宾夕法尼亚州立大学信息科学与技术学院);Department of Computer Science, Stony Brook University(石溪大学计算机科学系)
Bidirectional Language Models Are Also Few-shot Learners
University of Pennsylvania(宾夕法尼亚大学);Microsoft(微软);Google Research(谷歌研究院);UNC Chapel Hill(北卡罗来纳大学教堂山分校)
大语言模型,例如GPT-3(Brown等人,2020)可以执行任意任务,而无需在仅使用几个标记示例进行提示后进行微调。一个任意的任务可以被重新表述为一个自然语言提示,一个语言模型可以被要求生成完成,间接地执行任务的范式称为基于语义的学习。到目前为止,基于突现的学习能力主要是针对单向语言模型来证明的。然而,在去噪目标上预训练的双向语言模型(例如掩蔽语言建模)为迁移学习产生更强的学习表示。这激发了促进双向模式的可能性,但它们的预培训目标使它们在很大程度上与现有的激励范式不相容。我们提出了SAP(顺序自回归拟合),一种技术,使双向模型的提示。利用机器翻译任务作为案例研究,我们提出了双向mT5模型(Xue et al.,2021)与SAP进行比较,并证明其Few-Shot和zero-shot转换优于单向模型(如GPT-3和XGLM)的Few-Shot转换(Lin et al.,2021),尽管mT5的参数减少了约50%。我们进一步表明SAP在问答和摘要方面是有效的。这是第一次,我们的研究结果表明,基于语义的学习是一个更广泛的一类语言模型,而不仅仅是单向模型的新兴属性。
Diverge to Induce Prompting: Multi-Rationale Induction for Zero-Shot Reasoning
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan(国家台湾大学计算机科学与信息工程系);Institute of Information Science, Academia Sinica, Taiwan(学术院信息科学研究所);AI Research Center (AINTU), National Taiwan University, Taiwan(国家台湾大学人工智能研究中心(AINTU))