E²-LLM:大语言模型的高效与极限长度扩展
E^2-LLM: Efficient and Extreme Length Extension of Large Language Models
AI总结:
本文提出E²-LLM方法,仅用4k短上下文数据训练一次,并通过RoPE位置嵌入的尺度和位置索引增强,实现大语言模型在推理时对任意长上下文的低成本高效扩展。
摘要:
通常,训练具有长上下文大小的大语言模型在计算上是昂贵的,需要大量的训练时间和GPU资源。现有的长上下文扩展方法通常需要额外的训练过程来支持相应的长上下文窗口,其中需要长上下文训练数据(例如32k),并假定存在高昂的GPU训练成本。为了解决上述问题,我们提出了一种面向大语言模型的高效且极限长度扩展方法,称为E²-LLM,该方法仅需一次训练过程并大幅降低计算成本,同时也无需收集长上下文数据。具体而言,首先,我们的E²-LLM的训练数据仅需要较短的长度(例如4k),这极大地降低了调优成本。其次,在短训练上下文窗口上的训练过程只执行一次,并且我们可以在推理时支持不同的评估上下文窗口。第三,在E²-LLM中,基于RoPE位置嵌入,我们在训练中针对不同样本引入了两种不同的增强方法,分别作用于尺度和位置索引参数。其目的是使模型在推理时直接插值任意上下文长度时,对不同相对差异更加鲁棒。在多个基准数据集上的综合实验结果表明,我们的E²-LLM在具有挑战性的长上下文任务上是有效的。
英文摘要:
Typically, training LLMs with long context sizes is computationally expensive, requiring extensive training hours and GPU resources. Existing long-context extension methods usually need additional training procedures to support corresponding long-context windows, where the long-context training data (e.g., 32k) is needed, and high GPU training costs are assumed. To address the aforementioned issues, we propose an Efficient and Extreme length extension method for Large Language Models, called E 2 -LLM, with only one training procedure and dramatically reduced computation cost, which also removes the need to collect long-context data. Concretely, first, the training data of our E 2 -LLM only requires a short length (e.g., 4k), which reduces the tuning cost greatly. Second, the training procedure on the short training context window is performed only once time, and we can support different evaluation context windows at inference. Third, in E 2 - LLM, based on RoPE position embeddings, we introduce two different augmentation methods on the scale and position index parameters for different samples in training. It aims to make the model more robust to the different relative differences when directly interpolating the arbitrary context length at inference. Comprehensive experimental results on multiple benchmark datasets demonstrate the effectiveness of our E 2 -LLM on challenging long-context tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。