RAG-RewardBench：面向偏好对齐的检索增强生成中奖励模型基准测试
RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment
AI总结：
为解决检索增强语言模型与人类偏好对齐中缺乏可靠奖励模型评估标准的问题，提出首个RAG奖励模型基准RAG-RewardBench，设计多跳推理等四个特定场景，全面评估45个模型并揭示现有方法在对齐上的不足。
摘要：
尽管现有的检索增强语言模型（RALMs）在提供可信响应和基于可靠来源方面取得了显著进展，但它们往往忽略与人类偏好的有效对齐。在对齐过程中，奖励模型（RMs）作为人类价值观的关键代理来指导优化。然而，目前尚不清楚如何评估和选择适用于RALMs偏好对齐的可靠RM。为此，我们提出RAG-RewardBench，这是首个在RAG设置下评估RMs的基准。首先，我们设计了四个关键且具挑战性的RAG特定场景来评估RMs，包括多跳推理、细粒度引用、适当的弃权（不执行）和冲突鲁棒性。然后，我们整合了18个RAG子集、6个检索器和24个RALMs以增加数据源的多样性。最后，我们采用LLM-as-a-judge方法提升偏好标注的效率与有效性，其与人类标注表现出强相关性。基于RAG-RewardBench，我们对45个RMs进行了全面评估，并揭示了它们在RAG场景中的局限性。此外，我们还发现现有的经过训练的RALMs在偏好对齐方面几乎没有改善，凸显了向偏好对齐训练转变的必要性。我们在 https://huggingface.co/datasets/jinzhuoran/RAG-RewardBench/ 公开发布了基准和代码供未来研究使用。
英文摘要：
Despite the significant progress made by existing retrieval augmented language models (RALMs) in providing trustworthy responses and grounding in reliable sources, they often overlook effective alignment with human preferences. In the alignment process, reward models (RMs) act as a crucial proxy for human values to guide optimization. However, it remains unclear how to evaluate and select a reliable RM for preference alignment in RALMs. To this end, we propose RAG-RewardBench, the first benchmark for evaluating RMs in RAG settings. First, we design four crucial and challenging RAG-specific scenarios to assess RMs, including multi-hop reasoning, fine-grained citation, appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG subsets, six retrievers, and 24 RALMs to increase the diversity of data sources. Finally, we adopt an LLM-as-a-judge approach to improve preference annotation efficiency and effectiveness, exhibiting a strong correlation with human annotations. Based on the RAG-RewardBench, we conduct a comprehensive evaluation of 45 RMs and uncover their limitations in RAG scenarios. Additionally, we also reveal that existing trained RALMs show almost no improvement in preference alignment, highlighting the need for a shift towards preference-aligned training.We release our benchmark and code publicly at https://huggingface.co/datasets/jinzhuoran/RAG-RewardBench/ for future work.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
