发表机构
Shandong University（山东大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究通过固定递归污染协议，发现13个语言模型检查点的崩溃脆弱性差异显著且为检查点固有属性，并证明基于自身输出迭代可低成本预测脆弱性，收紧top-p能有效干预崩溃。
AI 中文摘要
模型生成的文本正重新进入训练语料库，大量证据表明，反复在此类数据上训练会导致输出多样性崩溃。已有研究探讨了这一现象本身：哪些协议和哪些数据混合会导致崩溃。但不同模型在同一过程中表现差异巨大。我们固定一种递归污染协议，让13个公开发布的检查点形成一个共享同一语料库、持续五代的生态系统。五代后，各检查点的唯一4-gram结果从0.187到0.940不等，相差约五倍：有些模型几乎不受影响，另一些则退化为重复片段。改变共享池的组成或混入人类文本，排序的Spearman相关性保持在0.91至0.97；改变随机种子，相关性保持在0.93至0.98。因此，模型在递归训练下是否容易崩溃，是检查点本身的一种属性，而这一属性在很大程度上未被研究。仅参数规模无法解释这一点，因为同一家族内三个规模的阶梯在规模上并非单调，且我们测试的静态指标均无法预测。有效的方法成本低廉：让模型在自己的输出上迭代两到三代，其在更大生态系统中的脆弱性即可由此推断。崩溃速度也响应干预。收紧top-p（在生成时截断低概率尾部）几乎能在三代内阻止崩溃，并使跨越整个谱系的六个检查点同时稳定，而数据端过滤则减缓崩溃但无法阻止。
英文摘要
How fast does a language model degrade when trained on its own outputs? Theory traces it to gradually accumulating errors, while experiments report repeated phrases within ten generations. Under a fixed sampling seed in vLLM, the fast loss of lexical diversity comes from the sampler. When vLLM serves a batch from one seeded sampling configuration, every request receives the same random draws, and a fixed seed replays them every generation. Fine-tuning raises the tokens that won, and the replayed draws let them win by more. Sharing across requests and replay across generations matter only together. Remove either one, by changing the shared seed every generation or by giving each request its own seed that repeats every generation, and the unique-4-gram fraction of two StableLM checkpoints stays near its starting value of about 0.98 through generation 3. Keep both, and the replayed shared seed takes seven checkpoints from five families to between 0.045 and 0.38 by then. Three generations of replay write the favoured phrases into the weights: decoded with one seed per request, the generation-3 weights of the replayed StableLM-2-1.6B chain recover most of their diversity, yet the phrase that filled every sample under the shared seed still opens 46% of them. Without replay, five checkpoints drift slowly, consistent with the gradual accumulation that theory describes, and three turn incoherent though their diversity scores stay high. One peer-reviewed model-collapse pipeline that fine-tunes Gemma-2-27B samples identical prompts under one seeded configuration, and three quarters of the rows it released for one iteration repeat nearly as often as one such batch copies them. A seed per request restores the fresh sample that stability analyses assume.
Commentsthe diversity loss reported in v1 is traced to a replayed vLLM sampling seed; substantially revised, new title
