arXiv 2412.145102024-12-19cs.CLcs.AI

PA-RAG：通过多视角偏好优化实现RAG对齐

PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization

School of Data Science and Engineering, East China Normal University（华东师范大学数据科学与工程学院）；Chinese Academy of Sciences（中国科学院）；Baidu Inc（百度公司）；Peking University（北京大学）

Jiayi Wu, Hengyi Cai, Lingyong Yan, Hao Sun, Xiang Li, Shuaiqiang Wang, Dawei Yin, Ming Gao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出PA-RAG方法，通过构建多视角偏好数据并结合SFT和DPO优化RAG生成器，在保持端到端形式的同时显著提升了响应信息量、鲁棒性和引用质量。

摘要：

检索增强生成（RAG）的出现缓解了大型语言模型（LLMs）生成中内容过时和幻觉的问题，但仍暴露出诸多局限。当通用LLM作为RAG生成器时，常出现响应信息量不足、响应鲁棒性差及引用质量低的问题。以往解决这些局限的方法，无论是通过在生成响应之外引入额外步骤，还是通过监督微调（SFT）优化生成器，都未能彻底与RAG需求对齐。因此，在保持端到端LLM形式的同时，从多个偏好视角优化RAG生成器仍是一项挑战。为弥补这一空白，我们提出了针对检索增强生成的多视角偏好对齐方法（PA-RAG），这是一种优化RAG系统生成器以全面符合RAG需求的方法。具体而言，我们通过在不同提示文档质量场景下从生成器采样不同质量的响应，构建了高质量的指令微调数据和多视角偏好数据。随后，我们使用SFT和直接偏好优化（DPO）优化生成器。在四个问答数据集和三个LLMs上进行的广泛实验表明，PA-RAG能显著提升RAG生成器的性能。我们的代码和数据集可在 https://github.com/wujwyi/PA-RAG 获取。

英文摘要：

The emergence of Retrieval-augmented generation (RAG) has alleviated the issues of outdated and hallucinatory content in the generation of large language models (LLMs), yet it still reveals numerous limitations. When a general-purpose LLM serves as the RAG generator, it often suffers from inadequate response informativeness, response robustness, and citation quality. Past approaches to tackle these limitations, either by incorporating additional steps beyond generating responses or optimizing the generator through supervised fine-tuning (SFT), still failed to align with the RAG requirement thoroughly. Consequently, optimizing the RAG generator from multiple preference perspectives while maintaining its end-to-end LLM form remains a challenge. To bridge this gap, we propose Multiple Perspective Preference Alignment for Retrieval-Augmented Generation (PA-RAG), a method for optimizing the generator of RAG systems to align with RAG requirements comprehensively. Specifically, we construct high-quality instruction fine-tuning data and multi-perspective preference data by sampling varied quality responses from the generator across different prompt documents quality scenarios. Subsequently, we optimize the generator using SFT and Direct Preference Optimization (DPO). Extensive experiments conducted on four question-answer datasets across three LLMs demonstrate that PA-RAG can significantly enhance the performance of RAG generators. Our code and datasets are available at https://github.com/wujwyi/PA-RAG.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。