arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.22018cs.CL
收藏

通过无缝数据打包改进持续预训练

Improving Continual Pre-training Through Seamless Data Packing

School of Computer Science, Fudan University(复旦大学计算机学院)

Ruicheng Yin, Xuan Gao, Changze Lv, Xiaohua Wang, Xiaoqing Zheng, Xuanjing Huang

AI总结:

针对持续预训练中传统数据打包导致的过度截断与上下文不连续问题,提出无缝打包(SP)策略,结合滑动窗口与 First-Fit-Decreasing 算法保留上下文并减少填充,在 99% 的设置中优于基线方法。

摘要:

持续预训练在提升模型性能方面展现了巨大潜力,尤其是在特定领域场景中。在持续预训练前打包数据最常见的方法是将输入文本拼接并切分为固定长度的序列。虽然这种方法简单高效,但常导致过度截断和上下文不连续,从而阻碍模型性能。为解决这些问题,我们探索了数据工程提升持续预训练的潜力,特别是其对模型性能和效率的影响。我们提出了 Seamless Packing (SP),一种新型数据打包策略,旨在更有效地保留上下文信息并提升模型性能。该方法在第一阶段采用滑动窗口技术,同步连续序列间重叠的 token,确保更好的连续性和上下文连贯性。在第二阶段,我们采用 First-Fit-Decreasing 算法,将较短文本打包进略大于目标序列长度的容器中,从而最小化填充和截断。在多种模型架构和语料领域上的实证评估证明了该方法的有效性,在 99% 的所有设置中优于基线方法。代码可在 https://github.com/Infernus-WIND/Seamless-Packing 获取。

英文摘要:

Continual pre-training has demonstrated significant potential in enhancing model performance, particularly in domain-specific scenarios. The most common approach for packing data before continual pre-training involves concatenating input texts and splitting them into fixed-length sequences. While straightforward and efficient, this method often leads to excessive truncation and context discontinuity, which can hinder model performance. To address these issues, we explore the potential of data engineering to enhance continual pre-training, particularly its impact on model performance and efficiency. We propose Seamless Packing (SP), a novel data packing strategy aimed at preserving contextual information more effectively and enhancing model performance. Our approach employs a sliding window technique in the first stage that synchronizes overlapping tokens across consecutive sequences, ensuring better continuity and contextual coherence. In the second stage, we adopt a First-Fit-Decreasing algorithm to pack shorter texts into bins slightly larger than the target sequence length, thereby minimizing padding and truncation. Empirical evaluations across various model architectures and corpus domains demonstrate the effectiveness of our method, outperforming baseline method in 99% of all settings. Code is available at https://github.com/Infernus-WIND/Seamless-Packing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑