arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.21728cs.CL
收藏

在知识图谱上探索:通过路径细化奖励建模激励大语言模型自主探索

Explore-on-Graph: Incentivizing Autonomous Exploration of Large Language Models on Knowledge Graphs with Path-refined Reward Modeling

Zhongguancun Laboratory, Beijing, China(中关村实验室,北京,中国);Department of Electronic Engineering, Tsinghua University, Beijing, China(电子工程系,清华大学,北京,中国);Institute for Network Sciences and Cyberspace, Tsinghua University, Beijing, China(网络科学与网络空间研究院,清华大学,北京,中国);Ant International, Ant Group, Hangzhou, Zhejiang, China(蚂蚁集团,杭州,浙江,中国)

Shiqi Yan, Yubo Chen, Ruiqi Zhou, Zhengxi Yao, Shuai Chen, Tianyi Zhang, Shijie Zhang, Wei Qiang Zhang, Yongfeng Huang, Haixin Duan, Yunqi Zhang

AI总结:

本文提出Explore-on-Graph框架,通过强化学习和路径细化奖励建模,使大语言模型在知识图谱上自主探索更广泛的推理空间,实现对分布外推理问题的高效泛化。

摘要:

大语言模型(LLMs)的推理过程在问答任务中常受到幻觉和缺失事实的困扰。一个有前途的解决方案是将LLMs的答案建立在可验证的知识源上,例如知识图谱(KGs)。现有的KG增强方法通常通过在生成过程中强制规则或模仿固定演示路径集中的路径来限制LLM的推理。然而,它们自然将LLM的推理模式限制在先前经验或微调数据的范围内,限制了其对分布外图推理问题的泛化能力。为了解决这个问题,本文提出了一种名为Explore-on-Graph(EoG)的新框架,该框架鼓励LLMs在KGs上自主探索更广泛的推理空间。为了激励探索和发现新的推理路径,我们提出在训练过程中引入强化学习,其奖励是推理路径最终答案的正确性。为了提高探索的效率和意义,我们提出将路径信息作为额外的奖励信号来细化探索过程,以减少无效的努力。在五个KGQA基准数据集上的大量实验表明,到目前为止,我们的方法在性能上达到了最先进的水平,不仅优于开源的LLMs,甚至优于闭源的LLMs。

英文摘要:

The reasoning process of Large Language Models (LLMs) is often plagued by hallucinations and missing facts in question-answering tasks. A promising solution is to ground LLMs' answers in verifiable knowledge sources, such as Knowledge Graphs (KGs). Prevailing KG-enhanced methods typically constrained LLM reasoning either by enforcing rules during generation or by imitating paths from a fixed set of demonstrations. However, they naturally confined the reasoning patterns of LLMs within the scope of prior experience or fine-tuning data, limiting their generalizability to out-of-distribution graph reasoning problems. To tackle this problem, in this paper, we propose Explore-on-Graph (EoG), a novel framework that encourages LLMs to autonomously explore a more diverse reasoning space on KGs. To incentivize exploration and discovery of novel reasoning paths, we propose to introduce reinforcement learning during training, whose reward is the correctness of the reasoning paths' final answers. To enhance the efficiency and meaningfulness of the exploration, we propose to incorporate path information as additional reward signals to refine the exploration process and reduce futile efforts. Extensive experiments on five KGQA benchmark datasets demonstrate that, to the best of our knowledge, our method achieves state-of-the-art performance, outperforming not only open-source but also even closed-source LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑