arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.18009cs.CLcs.LG
收藏

通过分解位置向量探索大型语言模型的上下文窗口

Exploring Context Window of Large Language Models via Decomposed Positional Vectors

Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Baichuan Inc.(百川公司);School of Information, Renmin University of China(中国人民大学信息学院);Department of Computer Science, National University of Singapore(新加坡国立大学计算机科学系)

Zican Dong, Junyi Li, Xin Men, Wayne Xin Zhao, Bingbing Wang, Zhen Tian, Weipeng Chen, Ji-Rong Wen

AI总结:

本研究通过基于均值的分解方法分离LLMs隐藏状态中的位置向量,分析其在上下文窗口内外的形成与注意力影响,进而设计了位置向量替换和注意力窗口扩展两种无需训练的方法,有效实现了上下文窗口的扩展。

摘要:

基于Transformer的大型语言模型(LLMs)通常具有有限的上下文窗口,导致在处理超出上下文窗口长度的文本时性能显著下降。已有大量研究提出扩展上下文窗口并实现LLMs的长度外推,但对这些方法的深入解释仍然缺乏。在本研究中,我们探索上下文窗口内外的位置信息,以解读LLMs的底层机制。通过使用基于均值的分解方法,我们从LLMs的隐藏状态中分离出位置向量,并分析其形成及对注意力的影响。此外,当文本超出上下文窗口时,我们在两种设置下分析位置向量的变化,即直接外推和上下文窗口扩展。基于我们的发现,我们设计了两种无需训练的上下文窗口扩展方法:位置向量替换和注意力窗口扩展。实验结果表明,我们的方法能够有效扩展上下文窗口长度。

英文摘要:

Transformer-based large language models (LLMs) typically have a limited context window, resulting in significant performance degradation when processing text beyond the length of the context window. Extensive studies have been proposed to extend the context window and achieve length extrapolation of LLMs, but there is still a lack of in-depth interpretation of these approaches. In this study, we explore the positional information within and beyond the context window for deciphering the underlying mechanism of LLMs. By using a mean-based decomposition method, we disentangle positional vectors from hidden states of LLMs and analyze their formation and effect on attention. Furthermore, when texts exceed the context window, we analyze the change of positional vectors in two settings, i.e., direct extrapolation and context window extension. Based on our findings, we design two training-free context window extension methods, positional vector replacement and attention window extension. Experimental results show that our methods can effectively extend the context window length.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑