MetaRAG：面向智能体检索增强生成的信念-行动对齐策略优化
MetaRAG: Belief-Action Aligned Policy Optimization for Agentic RAG
浏览论文内容
中文总结 AI 辅助
MetaRAG是面向智能体RAG的信念-行动对齐策略优化框架，通过优先验证的行动生成与内部信念探测，提升了智能体RAG的准确率-效率权衡，收益可迁移至多种场景与模型。
中文摘要 AI 辅助
智能体检索增强生成（RAG）要求语言模型决定何时继续搜索、何时回答。现有基于强化学习（RL）的方法依赖外部监督，忽略了智能体对当前证据是否充足的内部信念。为解决该问题，我们将搜索决策质量重新表述为信念-行动对齐，提出面向智能体RAG的信念-行动对齐策略优化框架MetaRAG。MetaRAG采用优先验证的行动生成，在每次实际行动前引出明确的验证过程；同时采用内部信念探测，从相同的问题-历史上下文中估计策略模型自身的可回答性信念。基于此，MetaRAG推导一致性奖励，并由答案正确性进一步门控，避免强化内部一致但错误的轨迹。信念探测器仅在训练期间使用，无推理时开销。在7个公开问答基准上的实验表明，MetaRAG在准确率-效率权衡上始终优于强大的基于RL的智能体RAG基线，其收益可迁移至深度研究设置、不同优化器及多个模型主干。
英文摘要
Agentic retrieval-augmented generation (RAG) requires language models to decide when to continue searching and when to answer. Existing RL-based methods rely on external supervision and overlook the agent's internal belief about whether the current evidence is sufficient. To address this problem, we reformulate the search decision quality as belief-action alignment and propose MetaRAG, a belief-action aligned policy optimization framework for agentic RAG. MetaRAG uses Verify-first Action Generation to elicit an explicit verification process before each actual action, and Internal Belief Probing to estimate the policy model's own answerability belief from the same question-history context. Based on these, MetaRAG derives a consistency reward that is further gated by answer correctness, avoiding reinforcement of internally consistent but incorrect trajectories. The belief probe is used only during training and introduces no inference-time overhead. Experiments on seven public QA benchmarks show that MetaRAG consistently improves the accuracy-efficiency trade-off over strong RL-based agentic RAG baselines, with gains that transfer to deep research settings, different optimizers, and multiple model backbones.
发表机构
- Zhejiang University（浙江大学）
- Ant Group（蚂蚁集团）
- City University of Hong Kong（香港城市大学）
机构由 AI 辅助整理，请以论文原文为准。
