arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.00692cs.LGcs.AIcs.CL
收藏

用于状态空间模型超长上下文训练的伴随分片技术

Adjoint sharding for very long context training of state space models

Amazon(亚马逊);Duke University(杜克大学)

Xingzi Xu, Amir Tavanaei, Kavosh Asadi, Karim Bouyarmane

AI总结:

本文提出伴随分片技术,通过在训练期间分片梯度计算大幅降低内存需求,使状态空间大模型能在超长上下文上进行训练,并在 1.27B 参数模型上将最大上下文长度从 35K 提升至超过 100K token。

摘要:

尽管进展非常迅速,但在超长上下文中高效训练大语言模型(LLMs)仍然具有挑战性。现有方法退回到使用短上下文训练大语言模型(训练时最多几千个 token),并在评估长上下文(推理时超过 1M token 的上下文窗口)时使用推理时技术。与长上下文推理相反,在超长上下文输入提示上进行训练很快受到 GPU 内存可用性的限制,以及在最先进的硬件上所需的过长训练时间的限制。与此同时,许多现实生活中的应用不仅需要推理,还需要在特定任务上使用长上下文进行训练/微调。此类应用包括,例如,用各种原始参考信息来源扩充上下文以进行事实提取、事实摘要或事实核对任务。我们提出了伴随分片(adjoint sharding),这是一种新技术,包括在训练期间对梯度计算进行分片,以将内存需求降低数个数量级,从而使超长上下文训练在计算上变得可行。伴随分片基于伴随方法,并计算与反向传播等效的梯度。我们还提出了截断伴随分片,以在保持性能的同时加速算法。我们提供了分布式版本和并行版本的伴随分片,以进一步加速训练。实证结果表明,所提出的伴随分片算法在 1M 上下文长度训练下,使用 1.27B 参数的大语言模型将内存使用量降低了最多 3 倍。这使得在由五个 AWS P4 实例组成的训练基础设施上,将 1.27B 参数模型在训练或微调期间的最大上下文长度从 35K token 增加到超过 100K token。

英文摘要:

Despite very fast progress, efficiently training large language models (LLMs) in very long contexts remains challenging. Existing methods fall back to training LLMs with short contexts (a maximum of a few thousands tokens in training) and use inference time techniques when evaluating on long contexts (above 1M tokens context window at inference). As opposed to long-context-inference, training on very long context input prompts is quickly limited by GPU memory availability and by the prohibitively long training times it requires on state-of-the-art hardware. Meanwhile, many real-life applications require not only inference but also training/fine-tuning with long context on specific tasks. Such applications include, for example, augmenting the context with various sources of raw reference information for fact extraction, fact summarization, or fact reconciliation tasks. We propose adjoint sharding, a novel technique that comprises sharding gradient calculation during training to reduce memory requirements by orders of magnitude, making training on very long context computationally tractable. Adjoint sharding is based on the adjoint method and computes equivalent gradients to backpropagation. We also propose truncated adjoint sharding to speed up the algorithm while maintaining performance. We provide a distributed version, and a paralleled version of adjoint sharding to further speed up training. Empirical results show the proposed adjoint sharding algorithm reduces memory usage by up to 3X with a 1.27B parameter large language model on 1M context length training. This allows to increase the maximum context length during training or fine-tuning of a 1.27B parameter model from 35K tokens to above 100K tokens on a training infrastructure composed of five AWS P4 instances.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑