Prompt tuning prepends a soft prompt to the input embeddings or hidden states and only optimizes the prompt to adapt pretrained models (PTMs) to downstream tasks. The previous work manually selects prompt layers which are far from optimal and failed to exploit the potential of prompt tuning. In this work, we propose a novel framework, \underline{S}elective \underline{P}rompt \underline{T}uning (SPT), that learns to select the proper prompt layers by inserting a prompt controlled by a learnable probabilistic gate at each intermediate layer. We further propose a novel bi-level optimization framework, SPT-DARTS, that can better optimize the learnable gates and improve the final prompt tuning performances of the learned prompt layer settings. We conduct extensive experiments with ten benchmark datasets under the full-data and few-shot scenarios. The results demonstrate that our SPT framework can perform better than the previous state-of-the-art PETuning baselines with comparable or fewer tunable parameters.
Decomposed Prompting: Probing Multilingual Linguistic Structure Knowledge in Large Language Models
Center for Information and Language Processing (CIS)(信息与语言处理中心);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);ScaDS.AI and TU Dresden(ScaDS.AI 和 梅克伦堡-前波美拉尼亚技术大学);Department of Statistics, LMU Munich(统计学系,慕尼黑大学);University of Maryland, College Park(马里兰大学 College Park 分校)
Contextual Dynamic Prompting for Response Generation in Task-oriented Dialog Systems
AWS AI Labs(AWS人工智能实验室);University of Chicago(芝加哥大学)
响应生成是面向任务的对话系统的关键组成部分之一。现有的研究表明,大型预训练的语言模型可以适应这项任务。适应这种极其大的大语言模型的典型范例是通过对下游任务进行微调,这不仅耗时,而且涉及大量资源和对微调数据的访问。修正(Schick and Schütze,2020)已经成为许多NLP任务中微调的替代方案。在我们的工作中,我们探索的想法,使用提示响应生成面向任务的对话系统。具体来说,我们提出了一种方法,执行上下文动态提示的提示是从对话上下文学习。我们的目标是从对话上下文中提取有用的提示信号。在使用MultiWOZ 2.2数据集的实验中(Zang等人,2020),我们表明,上下文动态提示改善反应生成的综合得分(Mehri等人,2019年)由3个绝对点,和一个巨大的20点时,对话状态的合并。此外,这些对话的人类注释发现,结合上下文的代理人比香草前缀调整的代理人更受欢迎。
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Singapore Management University(新加坡管理大学);Southwest Jiaotong University(西南交通大学);Singapore University of Technology and Design(新加坡科技设计大学);East China Normal University(华东师范大学)
No One Fits All: From Fixed Prompting to Learned Routing in Multilingual LLMs
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan(国立台湾大学计算机科学与资讯工程学系);Institute of Information Science, Academia Sinica, Taiwan(台湾“中央研究院”信息科学研究所);AI Research Center (AINTU), National Taiwan University, Taiwan(国立台湾大学人工智能研究中心)
HPSS: Heuristic Prompting Strategy Search for LLM Evaluators
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University(清华大学对话人工智能(CoAI)小组);University of Electronic Science and Technology of China(电子科技大学);Zhipu AI(智谱AI);The Knowledge Engineering Group (KEG), Tsinghua University(清华大学知识工程小组(KEG))
Transfer-Prompting: Enhancing Cross-Task Adaptation in Large Language Models via Dual-Stage Prompts Optimization
School of Artificial intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University(吉林大学知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际中心)
Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting
University of Bayreuth(拜罗伊特大学)
我们提出了一个概念上的观点提示大语言模型(LLM),区分(1)diegetic提示(叙事的一部分,例如:“从前,我看到一只狐狸.),以及(2)非diegetic提示(外部,例如“写关于狐狸的冒险。").有了这个镜头,我们研究了129多产人群工作者如何用不同的用户界面(1 vs 3建议,有/没有非diegetic提示;实现GPT-3)写短文本:当界面提供多个建议,并提供了一个选项,非diegetic提示,参与者更喜欢从多个建议中选择,而不是通过非diegetic提示控制它们。当参与者提供非diegetic提示,这是要求灵感,主题或事实。特别是单一的建议,指导与diegetic和非diegetic信息。这项工作通过揭示(1)编写非diegetic提示需要努力,(2)人们结合diegetic和非diegetic提示,(3)他们使用他们的草稿(即diegetic信息)和建议时间来战略性地指导LLM,从而告知人类与人工智能的交互。