通过强化学习优化RAG重排序器与LLM反馈
Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning
AI总结：
本文提出RRPO框架，通过强化学习将重排序与LLM生成质量对齐，消除了对昂贵人工标注的依赖，实验显示其在知识密集型基准上优于基线模型。
摘要：
重排序器在精炼检索结果方面起着关键作用。然而，当前重排序模型通常仅基于静态人工标注的相关性标签进行优化，与下游生成过程脱节。这种脱节导致检索指标识别为相关文档的文档往往无法提供LLM所需的实际效用。为此，我们引入了重排序偏好优化（RRPO），一种强化学习框架，通过将重排序视为序列决策过程，利用LLM反馈优化上下文效用，从而消除对昂贵人工标注的需求。为确保训练稳定性，我们进一步引入了参考锚定的确定性基线。在知识密集型基准上的广泛实验表明，RRPO显著优于包括强大列表式重排序器RankZephyr在内的基线模型。进一步分析突显了我们框架的通用性：它能够无缝泛化到多样化的读者（例如GPT-4o），与查询扩展模块如Query2Doc集成，并且即使在训练时使用嘈杂的监督者时也保持稳健。
英文摘要：
Rerankers play a pivotal role in refining retrieval results for Retrieval-Augmented Generation. However, current reranking models are typically optimized on static human annotated relevance labels in isolation, decoupled from the downstream generation process. This isolation leads to a fundamental misalignment: documents identified as topically relevant by information retrieval metrics often fail to provide the actual utility required by the LLM for precise answer generation. To bridge this gap, we introduce ReRanking Preference Optimization (RRPO), a reinforcement learning framework that directly aligns reranking with the LLM's generation quality. By formulating reranking as a sequential decision-making process, RRPO optimizes for context utility using LLM feedback, thereby eliminating the need for expensive human annotations. To ensure training stability, we further introduce a reference-anchored deterministic baseline. Extensive experiments on knowledge-intensive benchmarks demonstrate that RRPO significantly outperforms strong baselines, including the powerful list-wise reranker RankZephyr. Further analysis highlights the versatility of our framework: it generalizes seamlessly to diverse readers (e.g., GPT-4o), integrates orthogonally with query expansion modules like Query2Doc, and remains robust even when trained with noisy supervisors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
