发表机构
University of California, Merced（加州大学默塞德分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对大语言模型持续知识注入中监督微调泛化不足的问题，提出三阶段自学习框架GRIN，其核心为混合策略RL算法Golden-GRPO，引入两个基准验证，结果表明GRIN性能优于SFT及混合策略RL基线。
AI 中文摘要
持续知识注入是让大语言模型跟上快速发展世界的关键。现有方法依赖监督微调（SFT），该方法以训练格式记忆注入的事实，但无法在转述、文档组合和推理中泛化。为解决此问题，我们提出Golden-GRPO注入（GRIN），这是一个用于持续知识注入的三阶段自学习框架。Golden-GRPO是一种专门为知识注入设计的混合策略强化学习算法，即使在策略内回滚对新事实失效时，也会注入标准答案以提供学习信号。我们进一步引入Blank和Counter两个文档级基准，分别针对新事实获取和反事实覆盖，每个基准均评估单事实召回、多源检索和推理。我们的实验确立了明确的经验结论：混合策略强化学习能够实现超越监督微调的知识吸收，GRIN在更难的问题类型上显著优于SFT和混合策略RL基线，在基础事实召回上与它们表现相当。
英文摘要
Continual knowledge injection is essential for keeping large language models up-to-date in a fast-evolving world. Existing methods rely on supervised fine-tuning (SFT), which memorizes injected facts in their training format but fails to generalize across paraphrasing, document combinations, and reasoning. To address this, we propose Golden-GRPO Injection (GRIN), a three-stage self-learning framework for continual knowledge injection. Golden-GRPO is a mixed-policy reinforcement learning algorithm designed specifically for knowledge injection, which injects a golden answer to provide learning signal even when on-policy rollouts fail on novel facts. We further introduce Blank and Counter, two document-level benchmarks targeting novel acquisition and counterfactual overwrite respectively, each evaluating single-fact recall, multi-source retrieval, and inferential reasoning. Our experiments establish a clear empirical claim: mixed-policy reinforcement learning enables knowledge absorption beyond what supervised fine-tuning can achieve. GRIN substantially outperforms SFT and mixed-policy RL baselines on the harder question types while matching them on basic fact recall.
