arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.12643cs.CLcs.AI
收藏

推理模式至关重要:无需人工推理依据的学习推理

Reasoning Pattern Matters: Learning to Reason without Human Rationales

Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所智能信息处理重点实验室)

Chaoxu Pang, Yixuan Cao, Ping Luo

AI总结:

本文证明在模式化推理任务中,推理模式而非推理依据数量决定性能,提出PARO框架用LLM自动生成推理依据,无需人工标注,性能媲美10倍规模人工数据。

摘要:

大型语言模型(LLMs)在广泛采用的SFT+RLVR范式下展现出了卓越的推理能力,该范式首先在人工标注的推理轨迹(推理依据)上进行监督微调(SFT)以建立初始推理行为,然后应用带可验证奖励的强化学习(RLVR),利用可验证信号优化模型,而无需黄金推理依据。然而,为SFT阶段标注高质量的推理依据仍然成本高昂。本文研究了在何种情况下以及如何在不损害推理性能的前提下大幅降低推理依据的标注成本。我们识别出一大类问题,称为模式化推理任务,其中推理遵循一种固定的、跨实例一致的程序性策略。尽管实例在内容上有所不同,如领域知识、事实信息或数值,但解决方案源于应用共享的推理模式。我们认为,SFT+RLVR在此类任务上的成功主要源于其使模型内化这些推理模式的能力。以数值语义匹配作为代表性任务,我们提供了因果和行为两方面的证据,表明推理模式而非推理依据的数量或质量是性能的关键决定因素。基于这些见解,我们提出了模式感知LLM作为推理依据标注器(PARO),这是一个简单而有效的框架,使LLM能够生成与任务特定推理模式对齐的推理依据,而无需人工推理依据标注。实验表明,PARO生成的推理依据在SFT+RLVR性能上可与10倍规模的人工推理依据相媲美。这些结果表明,大规模人工推理依据标注可以被基于LLM的自动标注所取代,只需对推理模式进行有限的人工监督。

英文摘要:

Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities under the widely adopted SFT+RLVR paradigm, which first performs Supervised Fine-Tuning (SFT) on human-annotated reasoning trajectories (rationales) to establish initial reasoning behaviors, then applies Reinforcement Learning with Verifiable Rewards (RLVR) to optimize the model using verifiable signals without golden rationales. However, annotating high-quality rationales for the SFT stage remains prohibitively expensive. This paper investigates when and how rationale annotation costs can be substantially reduced without compromising reasoning performance. We identify a broad class of problems, termed patterned reasoning tasks, where reasoning follows a fixed, procedural strategy consistent across instances. Although instances vary in content such as domain knowledge, factual information, or numeric values, the solution derives from applying a shared reasoning pattern. We argue that the success of SFT+RLVR on such tasks primarily stems from its ability to enable models to internalize these reasoning patterns. Using numerical semantic matching as a representative task, we provide both causal and behavioral evidence showing that reasoning patterns rather than the quantity or quality of rationales are the key determinant of performance. Building on these insights, we propose Pattern-Aware LLMs as Rationale AnnOtators (PARO), a simple yet effective framework that enables LLMs to generate rationales aligned with task-specific reasoning patterns without requiring human rationale annotations. Experiments show that PARO-generated rationales achieve comparable SFT+RLVR performance to human rationales that are 10 times larger. These results suggest that large-scale human rationale annotations can be replaced with LLM-based automatic annotations requiring only limited human supervision over reasoning patterns.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑