明智花费:在迭代式合成数据自举中最大化训练后收益
Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Bootstrapping
AI总结:
本研究提出理论框架分析迭代合成数据自举中的预算分配,证明指数增长策略优于恒定策略,并在图像去噪和数学推理实验中验证其稳定优势。
摘要:
现代基础模型在训练后阶段通常经历迭代式“自举”过程:模型生成合成数据,外部验证器过滤掉低质量样本,然后使用高质量子集进行进一步微调。经过多轮迭代,模型性能得到提升,这引出了一个关键问题:如何在各轮迭代之间分配生成和训练的总预算,以最大化最终性能?在本工作中,我们开发了一个用于分析预算分配策略的理论框架。具体而言,我们证明了恒定策略以高概率无法收敛,而递增策略——尤其是指数增长策略——展现出显著的理论优势。在基于扩散概率模型的图像去噪和基于大语言模型的数学推理实验中,指数增长和多项式增长策略均持续优于恒定策略,其中指数策略通常提供更稳定的性能。
英文摘要:
Modern foundation models often undergo iterative ``bootstrapping'' in their post-training phase: a model generates synthetic data, an external verifier filters out low-quality samples, and the high-quality subset is used for further fine-tuning. Over multiple iterations, the model performance improves, raising a crucial question: How should the total budget for generation and training be allocated across iterations to maximize final performance? In this work, we develop a theoretical framework for analyzing budget allocation strategies. Specifically, we show that constant policies fail to converge with high probability, while increasing policies -- particularly exponential growth policies -- exhibit significant theoretical advantages. Experiments on image denoising with diffusion probabilistic models and math reasoning with large language models show that both exponential and polynomial growth policies consistently outperform constant policies, with exponential policies often providing more stable performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。