arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.13680cs.CLcs.AI
收藏

通过检索增强强化微调进行类比推理学习

Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning

Meta Superintelligence Labs(Meta超级智能实验室);Rice University(莱斯大学)

Zilin Xiao, Qi Ma, Chun-cheng Jason Chen, Xintao Chen, Avinash Atreya, Hanjie Chen, Vicente Ordonez

AI总结:

提出RA-RFT框架,通过黄金相关性蒸馏训练检索器,并结合强化微调利用类比推理轨迹,提升数学推理性能。

摘要:

检索增强生成(RAG)已成为将语言模型锚定于外部知识的标准机制,然而基于词汇或语义相似性的传统检索难以适用于复杂推理任务:语义相似的问题可能要求完全不同的解决策略,而表面不同的问题可能共享相同的底层推理模式。我们提出检索增强强化微调(RA-RFT),一种事后训练框架,教导语言模型通过类比进行推理。RA-RFT使用黄金相关性蒸馏训练检索器,该检索器根据预期推理收益而非语义重叠对上下文进行排序,然后通过强化微调方法利用检索到的类比演示对策略模型进行微调,使模型学会在可验证的结果奖励下利用推理轨迹。我们进一步分析了检索上下文的多样性,发现推理感知检索揭示了互补的解决策略,为个别问题提供了不同的推理支架。在具有挑战性的数学推理基准上,RA-RFT始终优于标准强化微调方法。例如,在AIME 2025上,对于Qwen3-1.7B和Qwen3-4B,RA-RFT的平均@32准确率分别比GRPO提高了7.1和2.8个百分点——这表明推理感知检索是一个互补的改进轴,与奖励设计或训练课程的进步正交。

英文摘要:

Retrieval-augmented generation (RAG) has become a standard mechanism for grounding language models in external knowledge, yet conventional retrieval based on lexical or semantic similarity is poorly suited for complex reasoning tasks: a semantically similar problem may demand an entirely different solution strategy, while a superficially different problem may share the same underlying reasoning pattern. We propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that teaches language models to reason by analogy. RA-RFT uses gold-relevance distillation to train a retriever that ranks contexts by expected reasoning benefit rather than semantic overlap, and then fine-tunes the policy model via reinforcement fine-tuning methods with retrieved analogous demonstrations, so the model learns to leverage reasoning traces under verifiable outcome rewards. We further analyze the diversity of retrieved contexts and find that reasoning-aware retrieval surfaces complementary solution strategies that provide distinct reasoning scaffolds for individual problems. Across challenging mathematical reasoning benchmarks, RA-RFT consistently outperforms standard reinforcement fine-tuning methods. For example, it improves AIME 2025 average@32 accuracy by 7.1 and 2.8 points over GRPO for Qwen3-1.7B and Qwen3-4B respectively -- suggesting that reasoning-aware retrieval is a complementary axis of improvement and orthogonal to advances in reward design or training curricula.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑