通过弱监督激励强推理
Incentivizing Strong Reasoning from Weak Supervision
AI总结：
本文研究如何在无需高成本示范和强化学习的情况下，通过弱监督模型激励大语言模型的推理能力，发现弱监督能有效提升推理性能，接近强化学习的效果，且成本更低。
摘要：
大型语言模型（LLMs）在需要推理的任务上表现出色，但增强其推理能力通常依赖于强化学习（RL）或监督微调（SFT），这两种方法成本高昂。本文研究了一种新的问题：如何在不使用昂贵高质量示范和强化学习的情况下，通过弱监督模型激励LLMs的推理能力。我们探讨了LLMs的推理能力是否可以通过显著弱于自身的模型的监督来有效提升，并分析了何时以及为何弱监督能促使更强的模型产生推理能力。我们的发现表明，来自显著弱于自身的推理者的监督可以显著提升学生推理性能，恢复接近强化学习94%的收益，成本仅为后者的一小部分。在多样化的基准和模型架构上的实验表明，弱推理者能有效激励更强的学生模型，在广泛的推理任务中一致提升性能。我们的结果表明，这种简单的弱到强范式是激励LLMs在推理时间实现强推理能力的一种有前途且可推广的替代方法。代码可在https://github.com/yuanyige/w2sr公开获取。
英文摘要：
Large language models (LLMs) have demonstrated impressive performance on reasoning-intensive tasks, but enhancing their reasoning abilities typically relies on either reinforcement learning (RL) with verifiable signals or supervised fine-tuning (SFT) with high-quality long chain-of-thought (CoT) demonstrations, both of which are expensive. In this paper, we study a novel problem of incentivizing the reasoning capacity of LLMs without expensive high-quality demonstrations and reinforcement learning. We investigate whether the reasoning capabilities of LLMs can be effectively incentivized via supervision from significantly weaker models. We further analyze when and why such weak supervision succeeds in eliciting reasoning abilities in stronger models. Our findings show that supervision from significantly weaker reasoners can substantially improve student reasoning performance, recovering close to 94% of the gains of expensive RL at a fraction of the cost. Experiments across diverse benchmarks and model architectures demonstrate that weak reasoners can effectively incentivize reasoning in stronger student models, consistently improving performance across a wide range of reasoning tasks. Our results suggest that this simple weak-to-strong paradigm is a promising and generalizable alternative to costly methods for incentivizing strong reasoning capabilities at inference-time in LLMs. The code is publicly available at https://github.com/yuanyige/w2sr.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
