课程引导的强化学习用于高效多跳检索增强生成
Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generation
AI总结：
本文提出EVO-RAG，一种课程引导的强化学习框架，通过动态奖励调度优化多跳RAG的查询重写，在四个基准上提升精确匹配达4.6点并减少15%检索深度。
摘要：
检索增强生成（RAG）将大型语言模型（LLMs）建立在最新的外部证据之上，然而现有的多跳RAG流程仍会发出冗余的子查询、探索过浅或漫游于过长的搜索链中。我们引入了EVO-RAG，一个课程引导的强化学习框架，它将查询重写智能体从广泛的早期探索演变为简洁的后期精炼。EVO-RAG将七因素、步骤级奖励向量（涵盖相关性、冗余性、效率和答案正确性）与时变调度器相结合，该调度器在情节展开时重新加权这些信号。智能体通过多头部奖励模型上的直接偏好优化进行训练，使其能够学习何时搜索、回溯、回答或弃权（不执行）。在四个多跳问答基准（HotpotQA、2WikiMultiHopQA、MuSiQue和Bamboogle）上，EVO-RAG在强RAG基线上将精确匹配提高了最多4.6个百分点，同时将平均检索深度削减了15%。消融研究证实了课程阶段划分和动态奖励调度的互补作用。因此，EVO-RAG为构建可靠、成本效益高的多跳RAG系统提供了一种通用方案。
英文摘要：
Retrieval-augmented generation (RAG) grounds large language models (LLMs) in up-to-date external evidence, yet existing multi-hop RAG pipelines still issue redundant subqueries, explore too shallowly, or wander through overly long search chains. We introduce EVO-RAG, a curriculum-guided reinforcement learning framework that evolves a query-rewriting agent from broad early-stage exploration to concise late-stage refinement. EVO-RAG couples a seven-factor, step-level reward vector (covering relevance, redundancy, efficiency, and answer correctness) with a time-varying scheduler that reweights these signals as the episode unfolds. The agent is trained with Direct Preference Optimization over a multi-head reward model, enabling it to learn when to search, backtrack, answer, or refuse. Across four multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle), EVO-RAG boosts Exact Match by up to 4.6 points over strong RAG baselines while trimming average retrieval depth by 15 %. Ablation studies confirm the complementary roles of curriculum staging and dynamic reward scheduling. EVO-RAG thus offers a general recipe for building reliable, cost-effective multi-hop RAG systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
