arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.12665cs.CLcs.AIcs.LG
收藏

超越下一 token 预测:面向大语言模型的 patch 级训练

Beyond Next Token Prediction: Patch-Level Training for Large Language Models

Pattern Recognition Center, WeChat AI, Tencent Inc(模式识别中心、微信AI、腾讯公司)

Chenze Shao, Fandong Meng, Jie Zhou

AI总结:

本文提出 LLM 的 patch 级训练,用多 token 聚合的 patch 预测降低大部分数据处理成本,再经 token 级训练对齐推理,在 370M–2.7B 模型上将总成本降至 0.5×且保持性能。

摘要:

大语言模型(LLM)高昂的训练成本已成为下一代 LLM 发展中的重要瓶颈。本文表明,可以在不牺牲性能的情况下显著降低 LLM 的训练成本。具体而言,我们提出了面向 LLM 的 patch 级训练:将多个 token 聚合成一个信息密度更高的单元,称为“patch”,并将其作为训练 LLM 的基本文本单元。在 patch 级训练期间,我们向语言模型输入更短的 patch 序列,并训练其预测下一个 patch,从而以显著降低的成本处理大部分训练数据。随后,模型在剩余训练数据上继续进行 token 级训练,以与推理模式保持一致。在多种模型(370M–2.7B 参数)上的实验表明,patch 级训练可将总体训练成本降至 0.5×,同时与 token 级训练相比不会损害模型性能。源代码:https://github.com/shaochenze/PatchTrain。

英文摘要:

The prohibitive training costs of Large Language Models (LLMs) have emerged as a significant bottleneck in the development of next-generation LLMs. In this paper, we show that it is possible to significantly reduce the training costs of LLMs without sacrificing their performance. Specifically, we introduce patch-level training for LLMs, in which multiple tokens are aggregated into a unit of higher information density, referred to as a `patch', to serve as the fundamental text unit for training LLMs. During patch-level training, we feed the language model shorter sequences of patches and train it to predict the next patch, thereby processing the majority of the training data at a significantly reduced cost. Following this, the model continues token-level training on the remaining training data to align with the inference mode. Experiments on a diverse range of models (370M-2.7B parameters) demonstrate that patch-level training can reduce the overall training costs to 0.5$\times$, without compromising the model performance compared to token-level training. Source code: https://github.com/shaochenze/PatchTrain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑