RAG-DDR：使用可微数据奖励优化检索增强生成
RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards
AI总结：
本文提出可微数据奖励（DDR）方法，通过rollout评估扰动影响来对齐RAG模块间数据偏好，端到端优化RAG系统，显著优于SFT方法并提升小模型知识提取与冲突缓解能力。
摘要：
检索增强生成（RAG）已证明其通过从外部资源检索知识来缓解大型语言模型（LLMs）幻觉的有效性。为了使LLMs适应RAG系统，当前方法使用指令微调来优化LLMs，提高其利用检索知识的能力。这种监督微调（SFT）方法侧重于使用不同指令使LLMs能够处理各种RAG任务。然而，它训练RAG模块过拟合训练信号，并忽略了RAG系统内智能体之间不同的数据偏好。在本文中，我们提出了一种可微数据奖励（DDR）方法，通过对齐不同RAG模块之间的数据偏好来端到端地训练RAG系统。DDR的工作原理是使用rollout方法收集奖励以优化RAG系统中的每个智能体，该方法提示智能体采样一些潜在响应作为扰动，评估这些扰动对整个RAG系统的影响，随后优化智能体以产生能提高RAG系统性能的输出。我们在各种知识密集型任务上的实验表明，DDR显著优于SFT方法，特别是对于更依赖检索知识的较小规模参数的LLMs。此外，DDR展现出更强的对齐RAG模块间数据偏好的能力。DDR方法使生成模块在从文档中提取关键信息以及缓解参数记忆与外部知识之间的冲突方面更加有效。所有代码可在 https://github.com/OpenMatch/RAG-DDR 获取。
英文摘要：
Retrieval-Augmented Generation (RAG) has proven its effectiveness in mitigating hallucinations in Large Language Models (LLMs) by retrieving knowledge from external resources. To adapt LLMs for the RAG systems, current approaches use instruction tuning to optimize LLMs, improving their ability to utilize retrieved knowledge. This supervised fine-tuning (SFT) approach focuses on equipping LLMs to handle diverse RAG tasks using different instructions. However, it trains RAG modules to overfit training signals and overlooks the varying data preferences among agents within the RAG system. In this paper, we propose a Differentiable Data Rewards (DDR) method, which end-to-end trains RAG systems by aligning data preferences between different RAG modules. DDR works by collecting the rewards to optimize each agent in the RAG system with the rollout method, which prompts agents to sample some potential responses as perturbations, evaluates the impact of these perturbations on the whole RAG system, and subsequently optimizes the agent to produce outputs that improve the performance of the RAG system. Our experiments on various knowledge-intensive tasks demonstrate that DDR significantly outperforms the SFT method, particularly for LLMs with smaller-scale parameters that depend more on the retrieved knowledge. Additionally, DDR exhibits a stronger capability to align the data preference between RAG modules. The DDR method makes the generation module more effective in extracting key information from documents and mitigating conflicts between parametric memory and external knowledge. All codes are available at https://github.com/OpenMatch/RAG-DDR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
