RAG-RL：通过强化学习和课程学习推进检索增强生成
RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learning
AI总结：
RAG-RL通过强化学习和课程学习提升检索增强生成性能，使生成模型能识别并引用相关信息，提高样本效率和泛化能力，实验显示在多跳问答任务中准确率显著提升。
摘要：
检索增强生成（RAG）系统依赖检索模型识别相关上下文并利用这些上下文生成答案。然而，检索器的召回率和精确度有限，限制了下游性能。我们引入RAG-RL，一种训练不仅生成答案还能够识别并引用从更大检索上下文中相关信息的生成模型，将部分识别相关文档的任务从检索器转移到生成器。我们的方法采用课程学习，模型首先在仅包含相关上下文的简单示例上进行训练。实验表明，这些训练样本使模型能以更高的样本效率和泛化能力获得引用和推理技能，即使在无关段落数量增加时仍表现出强性能。我们将在三个开放领域多跳问答数据集上基准测试我们的方法，并报告在答案和引用准确率上的显著提升。我们的实验提供了实证见解，说明更简单的训练样本如何为模型学习特定技能（如引用生成）提供更强信号，以及不同后训练组件（如训练集构建、基于规则的奖励、训练样本排序等）如何影响最终模型性能。
英文摘要：
Retrieval-augmented generation (RAG) systems rely on retrieval models for identifying relevant contexts and answer generation models for utilizing those contexts. However, retrievers exhibit imperfect recall and precision, limiting downstream performance. We introduce RAG-RL, an answer generation model trained not only to produce answers but also to identify and cite relevant information from larger sets of retrieved contexts, shifting some of the burden of identifying relevant documents from the retriever to the answer generator. Our approach uses curriculum learning, where the model is first trained on easier examples that include only relevant contexts. Our experiments show that these training samples enable models to acquire citation and reasoning skills with greater sample efficiency and generalizability, demonstrating strong model performance even as the number of irrelevant passages increases. We benchmark our methods on three open-domain multi-hop question answering datasets and report significant gains in answer and citation accuracy. Our experiments provide empirical insights into how easier training samples can give models stronger signals for learning specific skills (e.g., citation generation) and how different components of post-training (e.g., training set construction, rule-based rewards, training sample ordering, etc.) impact final model performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
