解开绳结:一种用于语言模型长上下文预训练的高效数据增强策略
Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models
AI总结:
针对长上下文训练数据稀缺、短文本性能下降及注意力效率低的问题,本文提出 UtK 数据增强策略,通过打乱文档块构造交织长文本并训练模型定位相关片段,在 7B/72B 模型上显著提升 RULER 表现。
摘要:
大型语言模型(LLM)一直优先扩展上下文窗口,使模型能够纳入更多信息。然而,训练模型处理长上下文面临重大挑战,包括高质量自然长上下文数据稀缺、短上下文任务性能可能下降,以及注意力机制导致训练效率降低。本文提出 Untie the Knots(UtK),这是一种在继续预训练阶段采用的新型数据增强策略,旨在高效赋予 LLM 长上下文能力,而无需修改现有数据混合。具体而言,我们将文档分块、打乱分块,并创建复杂且如绳结般交织的长文本结构;随后训练 LLM 解开这些绳结,并在看似混乱的 token 序列中识别相关片段。该方法通过让模型在长上下文中准确关注相关信息,大幅提升性能,同时训练效率也显著提高。我们在 7B 和 72B 参数规模的模型上进行了大量实验,训练使用 200 亿个 token,结果表明 UtK 在 128K 上下文长度的 RULER 上分别达到 75% 和 84.5% 的准确率,显著优于其他长上下文策略。训练后的模型将开源,以供进一步研究。
英文摘要:
Large language models (LLM) have prioritized expanding the context window from which models can incorporate more information. However, training models to handle long contexts presents significant challenges. These include the scarcity of high-quality natural long-context data, the potential for performance degradation on short-context tasks, and the reduced training efficiency associated with attention mechanisms. In this paper, we introduce Untie the Knots (\textbf{UtK}), a novel data augmentation strategy employed during the continue pre-training phase, designed to efficiently enable LLMs to gain long-context capabilities without the need to modify the existing data mixture. In particular, we chunk the documents, shuffle the chunks, and create a complex and knotted structure of long texts; LLMs are then trained to untie these knots and identify relevant segments within seemingly chaotic token sequences. This approach greatly improves the model's performance by accurately attending to relevant information in long context and the training efficiency is also largely increased. We conduct extensive experiments on models with 7B and 72B parameters, trained on 20 billion tokens, demonstrating that UtK achieves 75\% and 84.5\% accurracy on RULER at 128K context length, significantly outperforming other long context strategies. The trained models will open-source for further research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。