利用自注意力实现大语言模型中依赖输入的软提示
Leveraging Self-Attention for Input-Dependent Soft Prompting in LLMs
AI总结：
针对大语言模型微调成本高的问题，提出ID-SPAM，利用自注意力根据输入生成软提示，以少量可训练参数提升任务表现与零样本领域迁移能力。
摘要：
大语言模型在特定领域任务中的表现需要进行微调，而微调在计算上代价高昂且在技术上具有挑战性。本文聚焦于使用软提示的参数高效微调，这是一种很有前景的方法，通过学习少量参数使预训练模型适应下游任务。我们提出一种新颖的带有自注意力机制的依赖输入软提示技术（ID-SPAM），该技术基于输入词元生成软提示，并以不同的重要性关注不同词元。我们的方法简单且高效，同时保持较少的可训练参数数量。我们在各类任务上展示了所提方法相比最先进技术的优势，并展示了其提升的零样本领域迁移能力。
英文摘要：
The performance of large language models in domain-specific tasks necessitates fine-tuning, which is computationally expensive and technically challenging. This paper focuses on parameter-efficient fine-tuning using soft prompting, a promising approach that adapts pre-trained models to downstream tasks by learning a small set of parameters. We propose a novel Input Dependent Soft Prompting technique with a self-Attention Mechanism (ID-SPAM) that generates soft prompts based on the input tokens and attends different tokens with varying importance. Our method is simple and efficient, keeping the number of trainable parameters small. We show the merits of the proposed approach compared to state-of-the-art techniques on various tasks and show the improved zero shot domain transfer capability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
