数据受限语言模型预训练的数据增强
Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining
AI总结：
针对数据受限下标准自回归预训练严重过拟合的问题，提出三类数据增强方法（token级噪声、序列排列、目标偏移预测），有效降低验证损失并支持数百epoch训练。
摘要：
随着AI实验室接近数据天花板，计算能力超过新高质量文本生成速率，语言模型预训练正转向数据受限、计算充裕的体制，需要在固定语料库上进行高效的多轮训练。标准自回归（AR）预训练在此设置下严重过拟合，早期达到最优然后持续恶化。我们研究数据增强作为正则化器来缓解过拟合，并在相同数据上实现数百轮的有效训练。我们为AR预训练引入了三类正交的增强：token级噪声（掩码、随机替换）、序列排列（从右到左预测、Fill-in-the-Middle）以及目标偏移预测（$x_{t+i}$，$i > 1$）。通过系统消融实验，我们发现单个增强相对于基线延迟了过拟合并降低了验证损失，其中随机token替换在单个方法中实现了最佳最小损失。组合增强类别进一步降低了最小验证损失。我们的实验表明，数据增强缓解了AR预训练的数据低效问题，并为数据受限体制提供了有前景的解决方案。所有代码和数据可在https://github.com/michaelchen-lab/data-augmentations-for-pretraining获取。
英文摘要：
As AI labs approach a data ceiling where compute capacity outpaces the rate of new high-quality text generation, language model pretraining is shifting toward a data-constrained, compute-abundant regime that demands productive multi-epoch training on fixed corpora. Standard autoregressive (AR) pretraining overfits severely in this setting, reaching its optimum early and then continuously deteriorating. We investigate training-time data augmentation as a regularizer to mitigate this overfitting and enable productive training for hundreds of epochs on the same data. We introduce three orthogonal categories of augmentation for AR pretraining: token-level noise (masking, random replacement), sequence permutations (right-to-left prediction, Fill-in-the-Middle), and target offset prediction ($x_{t+i}$ for $i > 1$). Through systematic ablations, we find that individual augmentations delay overfitting and lower validation loss relative to the baseline, with random token replacement achieving the best minimum loss among individual methods. Combining augmentation categories further lowers the minimum validation loss. Our experiments demonstrate that data augmentations mitigate AR pretraining's data inefficiency and offer a promising solution to the data-constrained regime~\footnote{All code and data are available at https://github.com/ michaelchen-lab/ data-augmentations-for-pretraining.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
