从“顿悟时刻”到可控思考:通过解耦推理与控制实现大型推理模型中的元认知推理
From "Aha Moments" to Controllable Thinking: Toward Meta-Cognitive Reasoning in Large Reasoning Models via Decoupled Reasoning and Control
AI总结:
针对大型推理模型过度思考问题,提出元认知推理框架MERA,通过解耦推理与控制、构建推理-控制交替序列训练,结合控制段策略优化,提升推理效率与准确性。
摘要:
大型推理模型(LRMs)能够展示逐步推理、反思和回溯,但这些行为往往缺乏调控,导致过度思考。结果,LRMs即使在得出高置信度结论后仍继续生成冗余推理,增加了推理成本和延迟,限制了实际部署。根本原因在于缺乏内在机制来监控推理状态并决定何时继续、回溯或停止。我们提出MERA,一个元认知推理框架,将推理与控制解耦,以实现控制策略的独立优化。MERA通过基于接管的数据构建高质量推理-控制监督数据,并将长程轨迹转化为结构化的推理-控制交替序列用于训练。模型通过监督微调内化结构化分离,并进一步通过控制段策略优化(CSPO)优化,该优化结合了段级GRPO与控制掩码,聚焦于控制段的学习。在推理基准上的实验表明,MERA同时提高了效率和准确性。
英文摘要:
Large Reasoning Models (LRMs) can exhibit step-by-step reasoning, reflection, and backtracking, but these behaviors are often unregulated, leading to overthinking. As a result, LRMs continue generating redundant reasoning even after reaching high-confidence conclusions. This increases inference cost and latency, limiting practical deployment. The root cause is the absence of an intrinsic mechanism to monitor the reasoning state and decide when to continue, backtrack, or stop. We propose MERA, a meta-cognitive reasoning framework that decouples reasoning from control to enable independent optimization of control strategies. MERA constructs high-quality reasoning-control supervision data via a takeover-based pipeline, and transforms long-horizon traces into structured reasoning-control alternating sequences for training. The model is trained with supervised fine-tuning to internalize the structured separation, and further optimized with Control-Segment Policy Optimization (CSPO), which combines segment-wise GRPO with control masking to focus learning on control segments. Experiments across reasoning benchmarks show that MERA improves both efficiency and accuracy.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。