合成数据训练下模型行为表征：跨规模与混合比例的实证研究
Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
AI总结：
本研究通过Pythia模型套件在五个任务上的受控实验，系统考察了合成数据比例对模型性能、校准和输出特征的影响，发现20%以内性能稳定、超过30%退化加速，且大模型更鲁棒，为合成数据预算提供了实用指导。
摘要：
大型语言模型生成的合成数据已成为现代NLP训练流程中不可或缺的一部分，从引导推理能力到增强指令遵循数据集。虽然近期工作展示了在保持高外部数据比例下成功应用的案例，但关于合成数据比例如何在不同规模上影响模型行为的系统性理解仍然有限。本文提出了一项受控实证研究，考察在训练中使用不同合成数据与外部数据比例时，模型的性能、校准和输出特征。利用Pythia模型套件（参数规模410M-12B）在五个不同任务上，我们评估了经过一到三轮训练迭代、合成数据比例从0%到50%的模型。我们的主要发现包括：模型在合成数据比例高达20%时保持稳定性能，但超过30%后性能下降加速；较大模型（6.9B-12B）对合成数据的鲁棒性优于较小模型（410M-1.4B）；校准退化先于准确率损失，提供了早期预警信号；任务特性也很重要，在合成数据训练下，推理任务比检索任务退化更快。重要的是，我们发现当前的最佳实践，例如STaR和Self-Instruct系统中采用的高于80%外部数据的做法，在我们的实验所识别的安全范围内运行良好。我们为从业者提供了基于模型规模和任务要求的合成数据预算的实用指导，并与同期工作（包括Shumailov等人的模型崩溃发现）进行了详细比较。
英文摘要：
Synthetic data generated by large language models has become integral to modern NLP training pipelines, from bootstrapping reasoning capabilities to augmenting instruction-following datasets. While recent work demonstrates successful applications maintaining high external data ratios, systematic understanding of how synthetic data proportion affects model behavior across different scales remains limited. This paper presents a controlled empirical study examining model performance, calibration, and output characteristics when trained on varying synthetic-to-external data ratios. Using the Pythia model suite (410M-12B parameters) across five diverse tasks, we evaluate models after one to three training iterations with synthetic data proportions ranging from 0-50\%. Our key findings include: models maintain stable performance with up to 20\% synthetic data, but degradation accelerates beyond 30\%; larger models (6.9B-12B) show greater robustness to synthetic data than smaller models (410M-1.4B); calibration degradation precedes accuracy loss, providing an early warning signal; and task characteristics matter, with reasoning tasks degrading faster than retrieval tasks under synthetic data training. Importantly, we find that current best practices, such as those employed in STaR and Self-Instruct systems that maintain greater than 80\% external data, operate well within safe regimes identified by our experiments. We provide practical guidance for practitioners on synthetic data budgets based on model scale and task requirements, alongside detailed comparison with concurrent work including Shumailov et al.'s model collapse findings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
