ADePT: 适应性分解提示微调用于参数高效微调
ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning
AI总结:
ADePT通过引入适应性分解提示微调方法,改进了参数高效微调的性能,实现了更优的适应能力和更高效的推理速度。
摘要:
提示微调(PT)通过优化少量软虚拟标记来使预训练大语言模型(PLMs)适应下游任务,这些标记被前置到输入标记嵌入中。最近,分解提示微调(DePT)通过将软提示分解为更短的软提示和一对低秩矩阵,展示了更强的适应能力。这对低秩矩阵的乘积被加到输入标记嵌入中以抵消它们。此外,由于更短的软提示,DePT相比PT实现了更快的推理速度。然而,在本文中,我们发现DePT基于位置的标记嵌入抵消限制了其在多样化模型输入中的泛化能力,且许多标记嵌入的共享嵌入抵消导致了次优化。为了解决这些问题,我们引入了适应性分解提示微调(ADePT),它由一个短软提示和一个浅层标记共享前馈神经网络组成。ADePT利用标记共享前馈神经网络学习每个标记的嵌入抵消,使嵌入抵消能够根据模型输入变化,从而更好地优化标记嵌入抵消。这使ADePT能够在不增加推理时间和额外可训练参数的情况下实现更优的适应性能。在涵盖23种自然语言处理任务和4种不同规模的典型PLM的全面实验中,ADePT一致超越了其他领先的参数高效微调方法,并在某些场景中甚至超过了全微调。我们还提供了对ADePT的理论分析。代码可在https://github.com/HungerPWAY/ADePT上获取。
英文摘要:
Prompt Tuning (PT) enables the adaptation of Pre-trained Large Language Models (PLMs) to downstream tasks by optimizing a small amount of soft virtual tokens, which are prepended to the input token embeddings. Recently, Decomposed Prompt Tuning (DePT) has demonstrated superior adaptation capabilities by decomposing the soft prompt into a shorter soft prompt and a pair of low-rank matrices. The product of the pair of low-rank matrices is added to the input token embeddings to offset them. Additionally, DePT achieves faster inference compared to PT due to the shorter soft prompt. However, in this paper, we find that the position-based token embedding offsets of DePT restrict its ability to generalize across diverse model inputs, and that the shared embedding offsets across many token embeddings result in sub-optimization. To tackle these issues, we introduce Adaptive Decomposed Prompt Tuning (ADePT), which is composed of a short soft prompt and a shallow token-shared feed-forward neural network. ADePT utilizes the token-shared feed-forward neural network to learn the embedding offsets for each token, enabling adaptive embedding offsets that vary according to the model input and better optimization of token embedding offsets. This enables ADePT to achieve superior adaptation performance without requiring more inference time or additional trainable parameters compared to vanilla PT and its variants. In comprehensive experiments across 23 natural language processing tasks and 4 typical PLMs of different scales, ADePT consistently surpasses the other leading parameter-efficient fine-tuning methods, and even outperforms the full fine-tuning in certain scenarios. We also provide a theoretical analysis towards ADePT. Code is available at https://github.com/HungerPWAY/ADePT.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。