CTRL-RAG：基于对比似然奖励的强化学习用于上下文忠实的RAG模型
CTRL-RAG: Contrastive Likelihood Reward Based Reinforcement Learning for Context-Faithful RAG Models
AI总结：
CTRL-RAG通过对比似然奖励机制提升RAG模型的上下文忠实性，结合内部与外部奖励框架，在多个基准测试中表现优异。
摘要：
随着检索增强生成（RAG）的广泛应用，训练大型语言模型（LLMs）以进行上下文敏感推理和忠实性变得越来越重要。现有的面向RAG的强化学习（RL）方法依赖于外部奖励，这些奖励往往无法评估文档的忠实性，并且在开放领域设置中可能误判相似答案。此外，目前没有基于RAG的自我奖励机制。此外，尽管这种机制原则上可以基于文档估计答案的置信度，但在自我判断中缺乏客观反馈可能导致幻觉积累和最终模型崩溃。为了解决这些问题，我们提出了一种新的“内部-外部”混合奖励框架，其核心是基于对比似然奖励（CLR）。CLR直接优化在提示下和无支持证据条件下响应的对数似然差距。这鼓励模型提取相关证据，并在特定上下文中增强其置信度。实验表明，我们的方法（单独使用或与外部正确性奖励结合）在单跳、多跳、垂直领域和忠实性基准测试中表现强劲。我们的训练代码和模型即将发布。
英文摘要：
With the growing use of Retrieval-Augmented Generation (RAG), training large language models (LLMs) for context-sensitive reasoning and faithfulness is increasingly important. Existing RAG-oriented reinforcement learning (RL) methods rely on external rewards that often fail to evaluate document faithfulness, and may misjudge similar answers in open-domain settings. In addition, there is no RAG-based selfreward mechanism. Moreover, although such a mechanism could in principle estimate answer confidence given documents, the absence of objective feedback in a self-judgment can cause hallucination accumulation and eventual model collapse. To tackle these issues, we propose a novel "internal-external" hybrid reward framework centered on a Contrastive Likelihood Reward (CLR). CLR directly optimizes the log-likelihood gap between responses conditioned on prompts with and without supporting evidence. This encourages the model to extract relevant evidence and increases its confidence when grounded in a specific context. Experiments show that our method (used alone or combined with external correctness rewards) achieves strong performance on singlehop, multi-hop, vertical-domain, and faithfulness benchmarks. Our training code and models are coming soon.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
