arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.02660cs.CLcs.LG
收藏

如何有效训练长上下文语言模型

How to Train Long-Context Language Models (Effectively)

Princeton Language and Intelligence(普林斯顿语言与智能)

Tianyu Gao, Alexander Wettig, Howard Yen, Danqi Chen

AI总结:

ProLong-8B 通过有效利用长上下文数据,实现了在长上下文任务上的卓越性能,尽管训练数据量仅为 Llama-3.1-8B-Instruct 的 5%。

摘要:

我们研究了语言模型(LM)的持续训练和监督微调(SFT),以有效利用长上下文信息。我们首先建立了一个可靠的评估协议来指导模型开发——而不是困惑度或简单的针在 haystack 中的测试,我们使用了一组广泛的长上下文下游任务,并在 SFT 之后评估模型,这能更好地揭示长上下文能力。在我们的稳健评估支持下,我们进行了彻底的实验,以决定持续预训练的数据混合、指令微调数据集以及许多其他设计选择,如位置外推。我们发现(1)代码库和书籍是长数据的优秀来源,但必须将其与高质量的短上下文数据结合;(2)训练序列长度超过评估长度可以提升长上下文性能;(3)对于 SFT,仅使用短指令数据集在长上下文任务上表现强劲。我们的最终模型 ProLong-8B,源自 Llama-3 并在 40B 个 token 上进行训练,展示了在同样大小的模型中 128K 长度下的最新长上下文性能。尽管在长上下文训练中只使用了 5% 的 token,ProLong 在大多数长上下文任务上仍优于 Llama-3.1-8B-Instruct。此外,ProLong 可以有效处理高达 512K 个 token,这是目前公开可用语言模型中最长的上下文窗口之一。

英文摘要:

We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information. We first establish a reliable evaluation protocol to guide model development -- instead of perplexity or simple needle-in-a-haystack (NIAH) tests, we use a broad set of long-context downstream tasks, and we evaluate models after SFT as this better reveals long-context abilities. Supported by our robust evaluations, we run thorough experiments to decide the data mix for continued pre-training, the instruction tuning dataset, and many other design choices such as position extrapolation. We find that (1) code repositories and books are excellent sources of long data, but it is crucial to combine them with high-quality short-context data; (2) training with a sequence length beyond the evaluation length boosts long-context performance; (3) for SFT, using only short instruction datasets yields strong performance on long-context tasks. Our final model, ProLong-8B, which is initialized from Llama-3 and trained on 40B tokens, demonstrates state-of-the-art long-context performance among similarly sized models at a length of 128K. ProLong outperforms Llama-3.1-8B-Instruct on the majority of long-context tasks despite using only 5% as many tokens during long-context training. Additionally, ProLong can effectively process up to 512K tokens, one of the longest context windows of publicly available LMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑