自我改进预训练：利用后训练模型来预训练更好的模型
Self-Improving Pretraining: using post-trained models to pretrain better models
AI总结：
本文提出一种新的预训练和中训练方法，通过利用已有的强后训练模型来重写预训练数据并判断策略模型的 rollout，从而在早期训练中引入安全、事实性、生成质量和推理能力等关键行为，提升模型整体性能。
摘要：
大型语言模型通常分阶段训练：先在原始文本上预训练，然后进行后训练以实现指令跟随和推理。然而，这种分离导致了一个根本性限制：许多期望的行为，如安全、事实性、整体生成质量以及推理能力，只在后期阶段添加，尽管早期学习的模式强烈影响模型的能力。为了解决这个问题，我们引入了一种新的预训练和中训练方法，该方法在早期训练中整合这些行为。我们利用一个已有的强大后训练模型来重写预训练数据并判断策略模型的 rollout，从而在训练早期使用强化学习。在我们的实验中，我们展示了这种做法可以显著提升质量、安全、事实性和推理能力。
英文摘要：
Large language models are classically trained in stages: pretraining on raw text followed by post-training for instruction following and reasoning. However, this separation creates a fundamental limitation: many desirable behaviors such as safety, factuality, overall generation quality, and reasoning ability are only added at a late stage, even though the patterns learned earlier strongly shape a model's capabilities. To tackle this issue, we introduce a new way to pretrain and mid-train models that incorporates these behaviors earlier. We utilize an existing strong, post-trained model to both rewrite pretraining data and to judge policy model rollouts, thus using reinforcement earlier in training. In our experiments, we show this can give strong gains in quality, safety, factuality and reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
