关于条件与组合式语言模型可微提示的研究
On Conditional and Compositional Language Model Differentiable Prompting
AI总结：
本文提出提示产生式系统PRopS，将任务指令或输入元数据转换为连续提示，通过模块化神经规则实现组合迁移与少样本学习，在组合泛化、可控摘要和多语言翻译上以更少可训练参数超越其他PLM适应方法。
摘要：
提示已被证明是使冻结的预训练语言模型（PLM）适应下游任务并取得良好表现的有效方法。提示可以由人工设计的词序列表示，也可以由学习到的连续嵌入表示。本文研究了条件式和组合式可微提示。我们提出了一种新模型——提示产生式系统（PRopS），该模型学习将任务指令或输入元数据转换为连续提示，从而从PLM中引出任务特定的输出。我们的模型采用基于产生式系统神经化表述的模块化网络结构，使模型能够学习离散规则——即学习专门用于转换特定提示输入模式的神经函数，从而适用于组合迁移学习和少样本学习。我们进行了广泛的实证和理论分析，结果表明PRopS在组合泛化任务、可控摘要和多语言翻译上始终优于其他PLM适应技术，并且常常超过完全微调的模型，同时所需的可训练参数更少。
英文摘要：
Prompts have been shown to be an effective method to adapt a frozen Pretrained Language Model (PLM) to perform well on downstream tasks. Prompts can be represented by a human-engineered word sequence or by a learned continuous embedding. In this work, we investigate conditional and compositional differentiable prompting. We propose a new model, Prompt Production System (PRopS), which learns to transform task instructions or input metadata, into continuous prompts that elicit task-specific outputs from the PLM. Our model uses a modular network structure based on our neural formulation of Production Systems, which allows the model to learn discrete rules -- neural functions that learn to specialize in transforming particular prompt input patterns, making it suitable for compositional transfer learning and few-shot learning. We present extensive empirical and theoretical analysis and show that PRopS consistently surpasses other PLM adaptation techniques, and often improves upon fully fine-tuned models, on compositional generalization tasks, controllable summarization and multilingual translation, while needing fewer trainable parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
