知识图谱是隐式奖励模型：路径衍生信号促进组合推理
Knowledge Graphs are Implicit Reward Models: Path-Derived Signals Enable Compositional Reasoning
AI总结：
本文提出利用知识图谱作为隐式奖励模型，通过路径衍生信号提升模型在复杂推理任务中的组合推理能力。
摘要：
大型语言模型在结构推理领域如数学和编程中已实现了接近专家水平的性能，但其在专业科学领域进行组合多跳推理的能力仍然有限。我们提出了一种自底向上的学习范式，其中模型基于公理化领域事实并将其组合以解决复杂且未见过的任务。为此，我们提出了一种基于监督微调和强化学习（RL）结合的后训练流程，其中知识图谱作为隐式奖励模型。通过从知识图谱路径中推导出新的奖励信号，我们提供了可验证、可扩展和基于事实的监督，鼓励模型在RL过程中组合中间公理，而不是仅优化最终答案。我们验证了这种方法在医疗领域的应用，训练了一个14B模型在短跳推理路径（1-3跳）上，并评估其零样本泛化能力以应对复杂的多跳查询（4-5跳）。我们的实验表明，路径衍生的奖励起到一种
英文摘要：
Large language models have achieved near-expert performance in structured reasoning domains like mathematics and programming, yet their ability to perform compositional multi-hop reasoning in specialized scientific fields remains limited. We propose a bottom-up learning paradigm in which models are grounded in axiomatic domain facts and compose them to solve complex, unseen tasks. To this end, we present a post-training pipeline, based on a combination of supervised fine-tuning and reinforcement learning (RL), in which knowledge graphs act as implicit reward models. By deriving novel reward signals from knowledge graph paths, we provide verifiable, scalable, and grounded supervision that encourages models to compose intermediate axioms rather than optimize only final answers during RL. We validate this approach in the medical domain, training a 14B model on short-hop reasoning paths (1-3 hops) and evaluating its zero-shot generalization to complex multi-hop queries (4-5 hops). Our experiments show that path-derived rewards act as a "compositional bridge", enabling our model to significantly outperform much larger models and frontier systems like GPT-5.2 and Gemini 3 Pro, on the most difficult reasoning tasks. Furthermore, we demonstrate the robustness of our approach to adversarial perturbations against option-shuffling stress tests. This work suggests that grounding the reasoning process in structured knowledge is a scalable and efficient path toward intelligent reasoning. Our code is publicly available at: https://github.com/jha-lab/kg-implicit-reward-compositional-rl/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
