arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.03170cs.CLcs.AIcs.LG
收藏

聚焦Transformer:用于上下文扩展的对比训练

Focused Transformer: Contrastive Training for Context Scaling

IDEAS NCBR(IDEAS国家数字能力研究中心);University of Warsaw(华沙大学);Google DeepMind(谷歌DeepMind);Institute of Mathematics, Polish Academy of Sciences(波兰科学院数学研究所);xAI

Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, Piotr Miłoś

AI总结:

本文提出聚焦Transformer(FoT),通过对比学习式训练优化(键,值)空间结构,缓解长上下文中的分心问题,并微调OpenLLaMA得到LongLLaMA,支持高达256k上下文的passkey检索。

摘要:

大型语言模型具有以上下文方式纳入新信息的卓越能力。然而,这种方法的全部潜力往往因有效上下文长度的限制而受到制约。解决该问题的一种方案是使注意力层能够访问由(键,值)对组成的外部记忆。然而,随着文档数量的增加,相关键与不相关键的比例会下降,导致模型更多地关注不相关键。我们发现了一个重大挑战,称为分心问题,即与不同语义值相关联的键可能发生重叠,使它们难以区分。为了解决这个问题,我们引入了聚焦Transformer(Focused Transformer,FoT),这是一种采用受对比学习启发的训练过程的技术。这种新颖方法增强了(键,值)空间的结构,从而能够扩展上下文长度。我们的方法允许对已有的大规模模型进行微调,以延长其有效上下文。我们通过对$3B$和$7B$ OpenLLaMA检查点进行微调证明了这一点。由此得到的模型,我们将其命名为LongLLaMA,在需要长上下文的任务中表现出进步。我们进一步展示了我们的LongLLaMA模型能够熟练地管理$256k$上下文长度以完成passkey检索。

英文摘要:

Large language models have an exceptional capability to incorporate new information in a contextual manner. However, the full potential of such an approach is often restrained due to a limitation in the effective context length. One solution to this issue is to endow an attention layer with access to an external memory, which comprises of (key, value) pairs. Yet, as the number of documents increases, the proportion of relevant keys to irrelevant ones decreases, leading the model to focus more on the irrelevant keys. We identify a significant challenge, dubbed the distraction issue, where keys linked to different semantic values might overlap, making them hard to distinguish. To tackle this problem, we introduce the Focused Transformer (FoT), a technique that employs a training process inspired by contrastive learning. This novel approach enhances the structure of the (key, value) space, enabling an extension of the context length. Our method allows for fine-tuning pre-existing, large-scale models to lengthen their effective context. This is demonstrated by our fine-tuning of $3B$ and $7B$ OpenLLaMA checkpoints. The resulting models, which we name LongLLaMA, exhibit advancements in tasks requiring a long context. We further illustrate that our LongLLaMA models adeptly manage a $256 k$ context length for passkey retrieval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑