如何通过强化学习解锁几何交错推理中的顿悟时刻
How RL Unlocks the Aha Moment in Geometric Interleaved Reasoning
AI总结:
本文探讨了在几何交错推理中,通过强化学习框架Faire克服传统监督微调的局限性,实现更深层次的因果对齐,从而提升推理性能。
摘要:
解决复杂几何问题本质上需要交错推理:紧密交替地构建图表和进行逻辑推断。尽管最近的多模态大语言模型(MLLMs)在视觉生成和绘图方面表现出色,但我们发现了一个反直觉且未被充分探索的现象。简单地对交错绘图-解题数据进行监督微调(SFT)会导致推理性能显著下降,相比纯文本基线。我们主张这种失败源于SFT的根本限制,即主要诱导分布对齐:模型学会复制交错绘图的表面格式,但未能内部化生成绘图与推理步骤之间的因果依赖。为克服这一限制,我们提出了Faire(用于交错推理的功能对齐)框架,该框架强制实施三个因果约束,以超越表面模仿,实现功能对齐。大量实验表明,Faire在模型行为上产生了质的转变,其中绘图被有效内部化,从而在具有挑战性的几何推理基准上取得竞争性表现。
英文摘要:
Solving complex geometric problems inherently requires interleaved reasoning: a tight alternation between constructing diagrams and performing logical deductions. Although recent Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities in visual generation and plotting, we identify a counter-intuitive and underexplored phenomenon. Naively applying Supervised Fine-Tuning (SFT) on interleaved plot-solution data leads to a substantial degradation in reasoning performance compared to text-only baselines. We argue that this failure stems from a fundamental limitation of SFT, which primarily induces distributional alignment: the model learns to reproduce the surface format of interleaved plotting but fails to internalize the causal dependency between the generated plot and reasoning steps. To overcome this limitation, we propose Faire (Functional alignment for interleaved reasoning), a reinforcement learning framework that enforces three casual constraints to move beyond superficial imitation toward functional alignment. Extensive experiments show that Faire induces a qualitative shift in model behavior in which the plotting is effectively internalized, yielding competitive performance on challenging geometric reasoning benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。