arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.14036cs.LGcs.AIcs.CL
收藏

走向 Transformer 中的无限长前缀

Towards Infinite-Long Prefix in Transformer

The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);The Simons Institute for the Theory of Computing(西蒙斯理论计算研究所);University of California, Berkeley(加州大学伯克利分校);Sun Yat-sen University(中山大学)

Yingyu Liang, Zhenmei Shi, Zhao Song, Chiwun Yang

AI总结:

针对 Prefix Learning 理论理解有限的问题,本文基于 NTK 框架为超长前缀训练提供收敛保证,并设计仅需少量额外参数即可近似无限长前缀注意力的算法,在多领域实验中表现优异。

摘要:

提示和基于上下文的微调方法(我们称之为 Prefix Learning)已被提出,用于增强语言模型在各种下游任务上的性能。它们在经验上是高效且有效的,能够匹敌全参数微调的性能,但理论理解仍然有限。在本文中,我们旨在通过从前缀长度的角度研究它们的能力来解决这一局限性。特别地,我们使用 Neural Tangent Kernel (NTK) 框架,在一种程式化设置下为训练超长前缀提供了收敛性保证。基于这一强理论保证,我们设计并实现了一种算法,该算法只需在 transformer 的每一层中引入并微调少量额外的可训练参数,而不是无限长的前缀,并且能将近似前缀注意力的误差保证在多项式级别的小范围内。在视觉、自然语言和数学数据上的初步实验结果表明,与全参数微调、P-Tuning V2 和 LoRA 等现有方法相比,我们的方法取得了更优或具竞争力的性能。这表明我们的方法在参数高效微调方面具有前景。

英文摘要:

Prompting and context-based fine-tuning methods, which we call Prefix Learning, have been proposed to enhance the performance of language models on various downstream tasks. They are empirically efficient and effective, matching the performance of full parameter fine-tuning, but the theoretical understandings are limited. In this paper, we aim to address this limitation by studying their ability from the perspective of prefix length. In particular, we provide a convergence guarantee for training an ultra-long prefix in a stylized setting using the Neural Tangent Kernel (NTK) framework. Based on this strong theoretical guarantee, we design and implement an algorithm that only needs to introduce and fine-tune a few extra trainable parameters instead of an infinite-long prefix in each layer of a transformer, and can approximate the prefix attention to a guaranteed polynomial-small error. Preliminary experimental results on vision, natural language, and math data show that our method achieves superior or competitive performance compared to existing methods like full parameters fine-tuning, P-Tuning V2, and LoRA. This demonstrates our method is promising for parameter-efficient fine-tuning. Our code can be found at \url{https://github.com/ChristianYang37/chiwun/tree/main/src/NTK-Attention}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑