arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.19486cs.CLcs.AIcs.ETcs.LGeess.SP
收藏

LoPT:面向参数高效语言模型的低秩提示调优

LoPT: Low-Rank Prompt Tuning for Parameter Efficient Language Models

Microsoft AI(微软人工智能)

Shouchang Guo, Sonam Damani, Keng-hao Chang

AI总结:

本文提出低秩提示调优LoPT,通过低秩模型优化提示,在保持与全参数提示调优相近效果的同时将可训练参数量减少5倍,并优于需多10-20倍参数的现有方法。

摘要:

在提示调优中,会在提示中添加前缀或后缀文本,并优化前缀/后缀的嵌入(软提示)或标记索引(硬提示),以在特定任务上获得对语言模型更强的控制。该方法无需手工设计提示工程或显式微调模型。提示调优比模型微调显著更参数高效,因为它只优化语言模型的部分输入以产生期望输出。在本工作中,我们旨在进一步减少语言模型在特定任务上表现良好所需的可训练参数量。我们提出低秩提示调优(LoPT),这是一种用于提示的低秩模型,能够实现高效的提示优化。所提方法在将可训练参数量减少5倍的同时,展现出与全参数提示调优相似的结果。与需要多10到20倍参数的最新方法相比,它也提供了有前景的结果。

英文摘要:

In prompt tuning, a prefix or suffix text is added to the prompt, and the embeddings (soft prompts) or token indices (hard prompts) of the prefix/suffix are optimized to gain more control over language models for specific tasks. This approach eliminates the need for hand-crafted prompt engineering or explicit model fine-tuning. Prompt tuning is significantly more parameter-efficient than model fine-tuning, as it involves optimizing partial inputs of language models to produce desired outputs. In this work, we aim to further reduce the amount of trainable parameters required for a language model to perform well on specific tasks. We propose Low-rank Prompt Tuning (LoPT), a low-rank model for prompts that achieves efficient prompt optimization. The proposed method demonstrates similar outcomes to full parameter prompt tuning while reducing the number of trainable parameters by a factor of 5. It also provides promising results compared to the state-of-the-art methods that would require 10 to 20 times more parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑