arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.10224cs.LGcs.AI
收藏

将元经验内化到记忆中以指导大语言模型的强化学习

Internalizing Meta-Experience into Memory for Guided Reinforcement Learning in Large Language Models

Shiting Huang, Zecheng Li, Yu Zeng, Qingnan Ren, Zhen Fang, Qisheng Su, Kou Shi, Lin Chen, Zehui Chen, Feng Zhao

AI总结:

通过将元经验内化到记忆中,MEL提升了大语言模型的强化学习效果,实现了更有效的知识重用和推理能力增强。

摘要:

可验证奖励的强化学习(RLVR)已经发展为增强大语言模型(LLMs)推理能力的有效方法。尽管其有效性,RLVR面临元学习瓶颈:它缺乏误差归因和经验内化机制,这些机制是人类学习周期中超越实践和验证之外固有的,从而限制了细粒度信用分配和可重用知识的形成。我们称从过去错误中衍生出的可重用知识表示为元经验。基于这一见解,我们提出了元经验学习(MEL),一种新的框架,该框架将自我蒸馏的元经验纳入模型的参数记忆中。在标准RLVR的基础上,我们引入了额外的设计,利用LLM的自我验证能力,对正确和错误轨迹进行对比分析,识别出推理错误出现的精确分叉点,并将其总结为可推广的元经验。元经验进一步被内化到LLM的参数记忆中,通过最小化负对数似然,这会诱导出一种语言建模的奖励信号,连接正确和错误的推理轨迹,并促进有效的知识重用。实验结果表明,MEL在基准测试上实现了持续的改进,跨不同模型规模获得3.92%至4.73%的Pass@1提升。

英文摘要:

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for enhancing the reasoning capabilities of Large Language Models (LLMs). Despite its efficacy, RLVR faces a meta-learning bottleneck: it lacks mechanisms for error attribution and experience internalization intrinsic to the human learning cycle beyond practice and verification, thereby limiting fine-grained credit assignment and reusable knowledge formation. We term such reusable knowledge representations derived from past errors as meta-experience. Based on this insight, we propose Meta-Experience Learning (MEL), a novel framework that incorporates self-distilled meta-experience into the model's parametric memory. Building upon standard RLVR, we introduce an additional design that leverages the LLM's self-verification capability to conduct contrastive analysis on paired correct and incorrect trajectories, identify the precise bifurcation points where reasoning errors arise, and summarize them into generalizable meta-experience. The meta-experience is further internalized into the LLM's parametric memory by minimizing the negative log-likelihood, which induces a language-modeled reward signal that bridges correct and incorrect reasoning trajectories and facilitates effective knowledge reuse. Experimental results demonstrate that MEL achieves consistent improvements on benchmarks, yielding 3.92%--4.73% Pass@1 gains across varying model sizes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑