arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.17044cs.CL
收藏

Transformer的长度外推:从位置编码视角的综述

Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding

Harbin Institute of Technology(哈尔滨工业大学);The University of Hong Kong(香港大学);Peng Cheng Laboratory(鹏城实验室);Du Xiaoman Financial (Beijing)(度小满金融(北京))

Liang Zhao, Xiachong Feng, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin, Ting Liu

AI总结:

该综述针对Transformer模型长度外推能力不足的问题,从位置编码视角系统梳理了可外推位置编码及基于其的位置插值、随机位置等外推方法,并提出了该领域的挑战与未来方向。

摘要:

基于Transformer构建的大语言模型(LLM)凭借其卓越能力受到全球关注。然而,包括LLM在内的所有基于Transformer的模型都存在预设长度限制,难以从短训练序列泛化到更长的推理序列,即无法进行长度外推以处理长序列,这严重阻碍了其在法律或科学文档等需要长输入序列的场景中的应用。因此,大量旨在增强Transformer长度外推能力的方法相继涌现。尽管研究投入颇多,但目前仍缺乏系统性的综述。为填补这一空白,我们从位置编码(PE)这一被认为是影响长度外推的核心因素的角度,采用统一符号体系深入梳理了这些研究进展。具体而言,我们首先介绍了在该研究领域占主导地位的可外推位置编码;随后深入探讨了基于这类位置编码的外推方法,涵盖位置插值和随机位置方法;最后,我们指出了该领域存在的若干挑战与未来研究方向。通过本综述,我们旨在帮助读者深入理解现有方法,并为未来研究提供启发。

英文摘要:

Built upon the Transformer, large language models (LLMs) have captured worldwide attention due to their remarkable abilities. Nevertheless, all Transformer-based models including LLMs suffer from a preset length limit and can hardly generalize from short training sequences to longer inference ones, namely, they cannot perform length extrapolation to handle long sequences, which severely hinders their application in scenarios demanding long input sequences such as legal or scientific documents. Thus, numerous methods have emerged to enhance the length extrapolation of Transformers. Despite the great research efforts, a systematic survey is still lacking. To fill this gap, we delve into these advances in a unified notation from the perspective of positional encoding (PE), as it has been considered the primary factor on length extrapolation. Specifically, we begin with extrapolatable PEs that have dominated this research field. Then, we dive into extrapolation methods based on them, covering position interpolation and randomized position methods. Finally, several challenges and future directions in this area are highlighted. Through this survey, we aim to enable the reader to gain a deep understanding of existing methods and provide stimuli for future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑