合成重写作为质量乘数:来自葡萄牙持续预训练的证据
Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining
AI总结:
该研究通过控制实验探讨合成重写在葡萄牙持续预训练中对数据质量的影响,发现高质量数据通过重写可提升模型性能,而低质量数据效果有限。
摘要:
通过文档重写生成合成数据已成为提升语言模型预训练的有前景技术,但大多数研究集中在英语且未系统控制源数据质量。本文基于标注有STEM和教育质量评分的葡萄牙语语料ClassiCC-PT,构建了两个不同质量级别的10B-token子集,并使用7B指令微调模型将每个子集重写为四种风格,生成约40B tokens的合成数据。在葡萄牙语基准PoETa V2上训练了两个英语中心基础模型,并发现7B规模下,重写高质量数据可提升3.4 NPM,而低质量数据提升仅0.5 NPM。1.1B规模下,低质量数据未修改效果与重写高质量数据相当。结果表明合成重写主要作为质量乘数而非数据整理的替代品,且效果依赖于规模。
英文摘要:
Synthetic data generation through document rewriting has emerged as a promising technique for improving language model pretraining, yet most studies focus on English and do not systematically control for the quality of the source data being rewritten. We present a controlled study of how synthetic rewriting interacts with source data quality in the context of Portuguese continued pretraining. Starting from ClassiCC-PT, a Portuguese corpus annotated with STEM and Educational quality scores, we construct two 10B-token subsets at different quality levels and rewrite each into four styles using a 7B instruction-tuned model, producing approximately 40B tokens of synthetic data per condition. We train two English-centric base models (1.1B and 7B parameters) on each condition and evaluate on PoETa V2, a comprehensive 44-task Portuguese benchmark. At the 7B scale, rewriting high-quality data yields a +3.4 NPM gain over the same data unmodified, while rewriting low-quality data provides only +0.5 NPM. At the 1.1B scale, this interaction is weaker, with unmodified low-quality data performing comparably to rewritten high-quality data. Our results demonstrate that synthetic rewriting acts primarily as a quality multiplier rather than a substitute for data curation, and that this effect is scale-dependent.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。