训练后阶段对数据污染的影响
The Impact of Post-training on Data Contamination
AI总结：
研究发现数据污染在训练后阶段会引发性能波动，但通过SFT和GRPO方法可缓解，且模型规模越大，污染影响越显著。
摘要：
我们对数据污染如何与大型语言模型训练管道中现在标准的训练后阶段相互作用进行了受控研究。从Qwen2.5（0.5B/1.5B）和Gemma3（1B/4B）的干净检查点开始，我们将五个GSM8K和MBPP测试项目注入到一个原本为25B token扩展预训练数据集的前2B token中。然后，我们比较了受污染和干净的模型，在预训练后立即以及再次在两种流行的训练后方法（监督微调（SFT）和强化学习（RL）与群体相对策略优化（GRPO））之后进行比较。应用的训练后步骤没有污染。在数学和编程基准测试中，我们发现三个一致的模式：（i）污染导致性能激增，这种激增随着持续的预训练逐渐减弱。即使在25B token之后，污染的明显性能膨胀可以接近零。（ii）SFT和GRPO都恢复了泄漏的信息，但有不同的外部有效性：SFT只在受污染的任务上提高分数，而GRPO还提高了在未受污染的对应任务（GSMPlus、HumanEval）上的表现。（iii）模型规模放大了这些倾向，更大的监督微调模型记忆更多，而更大的GRPO模型将泄漏转化为更可推广的能力。我们的结果强调了在训练后进行污染审计的必要性，并表明基于强化学习的训练后方法，虽然不完全免疫，但有助于缓解与污染相关的过估计问题。
英文摘要：
We present a controlled study of how dataset contamination interacts with the post-training stages now standard in large language model training pipelines. Starting from clean checkpoints of Qwen2.5 (0.5B/1.5B) and Gemma3 (1B/4B), we inject five copies of GSM8K and MBPP test items into the first 2B tokens of an otherwise 25B token extended pre-training dataset. We then compare the contaminated and clean models both immediately after pre-training and again after two popular post-training methods: supervised fine-tuning (SFT) and reinforcement learning (RL) with group relative policy optimization (GRPO). The applied post-training steps do not have any contamination. Across math and coding benchmarks, we find three consistent patterns: (i) Contamination causes performance spikes that are gradually diminished with continued pre-training. After even 25B tokens the apparent performance inflation of contamination can become close to zero. (ii) Both SFT and GRPO resurface the leaked information, but with different external validity: SFT inflates scores only on the contaminated tasks, whereas GRPO also inflates performance on uncontaminated counterparts (GSMPlus, HumanEval). (iii) Model scale amplifies these tendencies, larger Supervised Fine Tuned models memorize more, while larger GRPO models translate leakage into more generalizable capabilities. Our results underscore the need for contamination audits \emph{after} post-training and suggest that RL-based post-training, although not immune, can help alleviate contamination-related over-estimation problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
