RAGferee：为检索增强生成构建上下文奖励模型
RAGferee: Building Contextual Reward Models for Retrieval-Augmented Generation
AI总结：
针对现有奖励模型在RAG场景评判能力不足、相关专用数据集和模型缺失的问题，提出RAGferee方法将QA数据集改造为偏好对，训练出的小参数RAG专用奖励模型在ContextualJudgeBench上超越大参数通用模型，性能提升15.5%。
摘要：
现有奖励模型（Reward Models, RMs）通常基于通用偏好数据训练，在检索增强生成（Retrieval Augmented Generation, RAG）场景中表现不佳——该场景需要判断回复对检索上下文的忠实度、与用户查询的相关性、上下文不足时的恰当弃权（不执行）、信息的完整性与简洁性。针对公开可用的以RAG为核心的偏好数据集及专用RM缺失的问题，我们提出RAGferee方法，将问答（question-answering, QA）数据集改造为偏好对，优先考虑事实 groundedness（ groundedness：基于事实的一致性）而非风格特征，从而训练出更适合评判RAG回复的上下文RM。借助RAGferee，我们整理了一个含4K样本的小型偏好数据集，并微调了参数规模从7B到24B的RM。我们的以RAG为核心的RM在ContextualJudgeBench上取得了当前最优性能，超过了在规模大得多（多达240万样本）的通用语料上训练的现有70B+ RM，绝对提升幅度达+15.5%。
英文摘要：
Existing Reward Models (RMs), typically trained on general preference data, struggle in Retrieval Augmented Generation (RAG) settings, which require judging responses for faithfulness to retrieved context, relevance to the user query, appropriate refusals when context is insufficient, completeness and conciseness of information. To address the lack of publicly available RAG-centric preference datasets and specialised RMs, we introduce RAGferee, a methodology that repurposes question-answering (QA) datasets into preference pairs that prioritise groundedness over stylistic features, enabling the training of contextual RMs better suited to judging RAG responses. Using RAGferee, we curate a small preference dataset of 4K samples and fine-tune RMs ranging from 7B to 24B parameters. Our RAG-centric RMs achieve state-of-the-art performance on ContextualJudgeBench, surpassing existing 70B+ RMs trained on much larger (up to 2.4M samples) general corpora, with an absolute improvement of +15.5%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
