arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.03645cs.LG
收藏

强化微调用于历史感知密集检索器在RAG中

Reinforcement Fine-Tuning for History-Aware Dense Retriever in RAG

College of Computer Science and Technology, Zhejiang University, Hangzhou, China(浙江大学计算机科学与技术学院);Ningbo Global Innovation Center, Zhejiang University, Ningbo, China(浙江大学宁波全球创新中心);Department of Computer Science, National University of Singapore, Singapore(新加坡国立大学计算机科学系)

Yicheng Zhang, Zhen Qin, Zhaomin Wu, Wenqi Zhang, Shuiguang Deng

AI总结:

本文提出通过强化学习优化RAG中的检索器,通过引入随机采样和历史状态缓解状态别名问题,提升检索性能。

摘要:

检索增强生成(RAG)使大型语言模型(LLMs)能够生成基于证据的响应,其性能取决于检索器与LLMs之间的匹配程度。检索器优化已成为一种高效的LLMs微调替代方案。然而,现有解决方案在检索器优化目标与RAG流程目标之间存在目标不匹配的问题。强化学习(RL)提供了一种有希望的解决方案来解决这一限制,但将RL应用于检索器优化会引入两个根本性挑战:1)确定性检索与RL公式不兼容;2)多跳推理中的查询-only检索导致状态别名。为了解决这些挑战,我们用确定性检索替换为随机采样,并将RAG建模为马尔可夫决策过程,使检索器可通过RL进行优化。进一步,我们在每次检索步骤中将检索历史纳入状态以缓解状态别名。在各种RAG流程、数据集和检索器规模上进行的广泛实验展示了我们方法在RAG性能上的持续改进。

英文摘要:

Retrieval-augmented generation (RAG) enables large language models (LLMs) to produce evidence-based responses, and its performance hinges on the matching between the retriever and LLMs. Retriever optimization has emerged as an efficient alternative to fine-tuning LLMs. However, existing solutions suffer from objective mismatch between retriever optimization and the goal of RAG pipeline. Reinforcement learning (RL) provides a promising solution to address this limitation, yet applying RL to retriever optimization introduces two fundamental challenges: 1) the deterministic retrieval is incompatible with RL formulations, and 2) state aliasing arises from query-only retrieval in multi-hop reasoning. To address these challenges, we replace deterministic retrieval with stochastic sampling and formulate RAG as a Markov decision process, making retriever optimizable by RL. Further, we incorporate retrieval history into the state at each retrieval step to mitigate state aliasing. Extensive experiments across diverse RAG pipelines, datasets, and retriever scales demonstrate consistent improvements of our approach in RAG performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑