Select2Reason: 为长链推理实现高效的指令微调数据选择
Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoning
AI总结：
Select2Reason通过高效数据选择提升长链推理性能，实验证明其在多个基准测试中表现优异。
摘要：
一种实用的方法是通过在由强大推理模型如DeepSeek-R1合成的指令数据集上进行监督微调，以激活预训练大语言模型的长链推理能力，这为强化学习提供了一个成本效益更高的替代方案。然而，具有超过10万样本的大型指令集会带来显著的训练开销，而有效的自动长链推理指令选择策略仍然未被探索。在本工作中，我们提出Select2Reason，一种新颖且高效的长链推理指令微调数据选择框架。从重新思考行为如自我纠正和回溯的出现角度出发，我们调查了可能决定长链推理指令质量的常见指标。Select2Reason利用一个量化器来估计问题的难度，并通过加权方案联合采用基于推理轨迹长度的启发式方法进行排名，以优先选择高实用性的示例。在OpenR1-Math-220k上的实验证明，仅在Select2Reason所选数据的10%上微调LLM，其性能可与或优于全数据微调和开源基线OpenR1-Qwen-7B在三个竞赛级别和六个综合数学基准测试中。进一步的实验突显了在不同数据规模下的可扩展性、推理过程中的效率以及其对其他指令池的适应性，成本极低。
英文摘要：
A practical approach to activate long chain-of-thoughts reasoning ability in pre-trained large language models is to perform supervised fine-tuning on instruction datasets synthesized by strong Large Reasoning Models such as DeepSeek-R1, offering a cost-effective alternative to reinforcement learning. However, large-scale instruction sets with more than 100k samples incur significant training overhead, while effective strategies for automatic long-CoT instruction selection still remain unexplored. In this work, we propose Select2Reason, a novel and efficient instruction-tuning data selection framework for long-CoT reasoning. From the perspective of emergence of rethinking behaviors like self-correction and backtracking, we investigate common metrics that may determine the quality of long-CoT reasoning instructions. Select2Reason leverages a quantifier to estimate difficulty of question and jointly incorporates a reasoning trace length-based heuristic through a weighted scheme for ranking to prioritize high-utility examples. Empirical results on OpenR1-Math-220k demonstrate that fine-tuning LLM on only 10% of the data selected by Select2Reason achieves performance competitive with or superior to full-data tuning and open-source baseline OpenR1-Qwen-7B across three competition-level and six comprehensive mathematical benchmarks. Further experiments highlight the scalability in varying data size, efficiency during inference, and its adaptability to other instruction pools with minimal cost.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
