VCORE: 基于方差控制的优化重加权用于链式推理监督
VCORE: Variance-Controlled Optimization-based Reweighting for Chain-of-Thought Supervision
AI总结：
VCORE通过将链式推理监督转化为约束优化问题，实现对token的自适应监督分配，提升大语言模型的推理泛化能力，在数学和编程基准测试中表现突出。
摘要：
监督微调（SFT）在长链式推理（CoT）轨迹上的应用已成为增强大语言模型（LLMs）推理能力的关键技术。然而，标准交叉熵损失将所有token视为同等重要，忽略了推理轨迹中token的异质贡献。这种统一处理导致监督分配不当和泛化能力弱，尤其是在复杂、长形式推理任务中。为此，我们引入了基于方差控制的优化重加权（VCORE）框架，将CoT监督重新表述为约束优化问题。通过采用优化理论视角，VCORE实现了对token的原理性和自适应监督分配，使训练目标更贴近稳健推理泛化的目标。实证评估表明，VCORE在整体平均性能上表现最强，尤其在低容量模型上表现出明显优势。在域内和域外设置中，VCORE在数学和编程基准测试中实现了显著性能提升，使用Qwen3系列（4B、8B、32B）和LLaMA-3.1-8B-Instruct模型。此外，我们展示了VCORE作为后续强化学习更有效的初始化方法，为提升LLMs推理能力奠定了更坚实的基础。代码将在https://github.com/coder-gx/VCORE发布。
英文摘要：
Supervised fine-tuning (SFT) on long chain-of-thought (CoT) trajectories has emerged as a crucial technique for enhancing the reasoning abilities of large language models (LLMs). However, the standard cross-entropy loss treats all tokens equally, ignoring their heterogeneous contributions across a reasoning trajectory. This uniform treatment leads to misallocated supervision and weak generalization, especially in complex, long-form reasoning tasks. To address this, we introduce \textbf{V}ariance-\textbf{C}ontrolled \textbf{O}ptimization-based \textbf{RE}weighting (VCORE), a principled framework that reformulates CoT supervision as a constrained optimization problem. By adopting an optimization-theoretic perspective, VCORE enables a principled and adaptive allocation of supervision across tokens, thereby aligning the training objective more closely with the goal of robust reasoning generalization. Empirical evaluations demonstrate that VCORE achieves the strongest overall average performance, with especially clear gains on lower-capacity models. Across both in-domain and out-of-domain settings, VCORE achieves substantial performance gains on mathematical and coding benchmarks, using models from the Qwen3 series (4B, 8B, 32B) and LLaMA-3.1-8B-Instruct. Moreover, we show that VCORE serves as a more effective initialization for subsequent reinforcement learning, establishing a stronger foundation for advancing the reasoning capabilities of LLMs. The Code will be released at https://github.com/coder-gx/VCORE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
