开源推理模型缺失的一环：一个用于缓解强化学习中短思维链大语言模型冷启动问题的数据集
One Missing Piece for Open-Source Reasoning Models: A Dataset to Mitigate Cold-Starting Short CoT LLMs in RL
AI总结：
本文提出Long CoT Collection数据集，利用短CoT大语言模型构建10万条长思维链推理数据，以缓解开源推理模型对R1蒸馏的依赖，并为强化学习提供更强的冷启动基础。
摘要：
随着R1这一公开可用的大型推理模型（LRM）的发布，研究者通常通过在大语言模型上训练R1的长思维链（CoT）推理来构建新的LRM。尽管先前工作表明，LRM的能力可以通过直接蒸馏来复现，但对现有模型（如R1）的持续依赖仍然是推动该领域发展的关键局限。作为迈向独立开发LRM的第一步，本文探索了利用未针对推理时扩展进行训练的大语言模型构建长CoT数据集的可能性。为此，我们提出了Long CoT Collection，这是一个包含10万条CoT推理依据的数据集，使用现有的短CoT大语言模型进行标注。我们开发了一套流程，将o1的新颖推理策略引入短CoT大语言模型，使其能够进行更长时间的思考，并引入对思考预算的可控性，以更好地管理过度思考问题。我们的广泛分析验证了该数据集的质量与R1相当——或略低于R1。此外，我们的实验表明，在该数据集上进行训练不仅能增强通用推理能力，还能为强化学习提供坚实基础——以我们的数据初始化的模型通过RLVR获得了2到3倍的更大提升。
英文摘要：
With the release of R1, a publicly available large reasoning model (LRM), researchers commonly train new LRMs by training language models on R1's long chain-of-thought (CoT) inferences. While prior works show that LRMs' capabilities can be reproduced through direct distillation, the continued reliance on the existing models (e.g., R1) remains a critical limitation in advancing the field. As a first step toward independent LRM development, this paper explores the possibility of constructing a long CoT dataset with LLMs that are not trained for inference-time scaling. To this end, we present the Long CoT Collection, a dataset of 100K CoT rationales annotated using existing short CoT LLMs. We develop a pipeline that induces o1's novel reasoning strategies into short CoT LLMs, enabling them to think longer and introducing controllability over the thought budget to better manage the overthinking problem. Our extensive analyses validate that our dataset achieves quality comparable to--or slightly below--R1. Furthermore, our experiments demonstrate that training on our dataset not only strengthens general reasoning skills, but also provides a strong foundation for reinforcement learning--models initialized on our data achieve 2-3x larger gains with RLVR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
