超低维提示调优
Ultra-Low-Dimensional Prompt Tuning via Random Projection
AI总结：
本文提出超低维提示调优方法，通过在低维空间优化提示并在冻结随机矩阵上进行上投影，实现训练参数减少98%且保持性能，适用于大规模LLM定制。
摘要：
大型语言模型虽性能优异但微调成本高。提示调优是一种参数高效的方法，但其嵌入通常受限于模型隐藏维度，限制了参数节省。本文提出超低维提示调优（ULPT），通过在低维空间（如2D）优化提示并使用冻结随机矩阵进行上投影。ULPT相比传统提示调优可减少98%的训练参数，同时保持性能。我们在超过20个NLP任务上的广泛实验表明，ULPT在显著更少的参数下优于近期的参数高效调优方法，使其成为大规模LLM定制的存储高效框架。
英文摘要：
Large language models achieve state-of-the-art performance but are increasingly costly to fine-tune. Prompt tuning is a parameter-efficient fine-tuning method that addresses parameter-efficiency by learning prompt embeddings, but these embeddings are typically tied to the model's hidden dimensionality, limiting parameter saving. In this paper, we propose Ultra-Low-dimensional Prompt Tuning (ULPT), a simple yet effective method that optimizes prompts in a low-dimensional space (e.g., 2D) and uses a frozen random matrix for up-projection. ULPT can achieve 98% reduction in the training parameters compared to vanilla prompt tuning while preserving performance. Our extensive experiments across over 20 NLP tasks demonstrate that ULPT consistently outperforms recent parameter-efficient tuning methods using significantly fewer parameters, making it well-suited as a storage-efficient framework for massive LLM customization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
