在微调后输出多样性为何会崩溃？
Where does output diversity collapse in post-training?
AI总结：
研究探讨了微调后输出多样性的崩溃原因，发现数据组成和训练方法共同影响多样性，揭示多样性崩溃发生在训练阶段而非推理阶段。
摘要：
微调后的语言模型产生的输出多样性低于基础模型。这种多样性崩溃会影响依赖多样化样本的推理时间扩展方法，并在创造性和价值相关的任务中导致模型输出同质化。先前的工作将崩溃归因于特定的微调方法，但未区分训练数据组成与方法，或生成格式与模型权重的作用。我们追踪了三个平行的微调线谱（Olmo 3、Think（链式推理蒸馏）、Instruct（广泛多源数据）和RL-Zero）在15个任务和四个文本多样性度量标准下的输出多样性。我们发现多样性崩溃的位置与数据组成相关：Think线谱在监督微调中失去大部分语义多样性，DPO在Instruct中的影响大于Think。在Think模型中抑制推理时的链式推理会降低在困难任务上的准确性，但保持答案层面的多样性不变，表明多样性崩溃嵌入在模型权重中，而非由生成格式引起。将六个可验证任务上的多样性损失分解为质量控制组件（去除错误输出）和残差组件（正确输出间的缩小）发现，这种分割是任务依赖的，尽管Think模型在总体上崩溃更多，但保留了更多正确答案的多样性。我们的结果表明，多样性崩溃由训练期间的数据组成决定，无法仅在推理阶段解决。
英文摘要：
Post-trained language models produce less varied outputs than their base counterparts. This output diversity collapse undermines inference-time scaling methods that rely on varied samples, and risks homogenizing model outputs on creative and value-laden tasks. Prior work attributes collapse to specific post-training methods, without separating the role of training data composition from the method, or the generation format from the model weights. We trace output diversity through three parallel post-training lineages of Olmo 3, Think (chain-of-thought distillation), Instruct (broad multi-source data), and RL-Zero, across 15 tasks and four text diversity metrics. We find that the location of collapse co-varies with data composition: the Think lineage loses most semantic diversity at supervised fine-tuning, and the effect of DPO is larger in Instruct than in Think. Suppressing chain-of-thought reasoning at inference in Think models drops accuracy on hard tasks, yet leaves answer-level diversity unchanged, showing that the collapse is embedded in the model weights by training data, not imposed by the generation format. Decomposing diversity loss on six verifiable tasks into a quality-control component (removal of incorrect outputs) and a residual component (genuine narrowing among correct outputs) reveals that the split is task-dependent, and Think models retain more correct-answer diversity than Instruct despite collapsing more in aggregate. Our results indicate that diversity collapse is determined during training by data composition and cannot be addressed at inference time alone.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
