arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.00509cs.CL
收藏

LongRecipe:大语言模型中高效长上下文泛化的配方

LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models

National University of Singapore(新加坡国立大学);Nanjing University(南京大学);University of Toronto(多伦多大学);Mila, Québec AI Institute(魁北克人工智能研究所(Mila));Université de Montréal(蒙特利尔大学);Tencent Inc(腾讯公司);Baidu Inc(百度公司);Nanyang Technological University(南洋理工大学)

Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang, Yan Wang, Wei Shen, Qing Gu, Anh Tuan Luu, See-Kiong Ng, Zhiwei Jiang, Bryan Hooi

AI总结:

LongRecipe通过token分析、位置索引变换和训练优化,仅需目标上下文窗口30%的序列长度和不到15%的计算资源,即可将LLM有效上下文从8k扩展到128k,性能接近GPT-4。

摘要:

大语言模型(LLMs)在处理长上下文任务时面临重大挑战,因为它们在预训练期间的有效上下文窗口大小有限,这限制了它们对扩展序列的泛化能力。同时,通过后预训练来扩展LLMs的上下文窗口是高度资源密集型的。为了解决这个问题,我们引入了LongRecipe,这是一种用于扩展LLMs上下文窗口的高效训练策略,包括有影响力的token分析、位置索引变换和训练优化策略。它模拟长序列输入,同时保持训练效率,并显著提高模型对长距离依赖的理解。在三种类型的LLMs上的实验表明,LongRecipe可以利用长序列,同时只需要目标上下文窗口大小的30%,并且与全序列训练相比,计算训练资源减少了85%以上。此外,LongRecipe还保留了原始LLM在通用任务上的能力。最终,我们可以将开源LLMs的有效上下文窗口从8k扩展到128k,仅使用一块80G内存的GPU进行一天的专门训练,就能达到接近GPT-4的性能。我们的代码发布在https://github.com/zhiyuanhubj/LongRecipe。

英文摘要:

Large language models (LLMs) face significant challenges in handling long-context tasks because of their limited effective context window size during pretraining, which restricts their ability to generalize over extended sequences. Meanwhile, extending the context window in LLMs through post-pretraining is highly resource-intensive. To address this, we introduce LongRecipe, an efficient training strategy for extending the context window of LLMs, including impactful token analysis, position index transformation, and training optimization strategies. It simulates long-sequence inputs while maintaining training efficiency and significantly improves the model's understanding of long-range dependencies. Experiments on three types of LLMs show that LongRecipe can utilize long sequences while requiring only 30% of the target context window size, and reduces computational training resource over 85% compared to full sequence training. Furthermore, LongRecipe also preserves the original LLM's capabilities in general tasks. Ultimately, we can extend the effective context window of open-source LLMs from 8k to 128k, achieving performance close to GPT-4 with just one day of dedicated training using a single GPU with 80G memory. Our code is released at https://github.com/zhiyuanhubj/LongRecipe.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑