紧凑语言模型能否像智能体一样搜索？基于知识增强的策略优化以保持智能体RAG能力
Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities
AI总结：
本文提出Distillation-Guided Policy Optimization方法，通过教师示范初始化和持续教师指导，使紧凑模型实现复杂的智能体搜索行为，甚至在某些情况下超越大模型。
摘要：
强化学习已成为激发语言模型代理RAG行为（如搜索和规划）的主导后训练方法。尽管在大模型上表现优异，但将其应用于紧凑模型（如0.5-1B参数）存在独特挑战。紧凑模型初始表现差，导致奖励稀疏和训练不稳定。为此，我们提出Distillation-Guided Policy Optimization（DGPO），采用冷启动初始化和持续教师指导。为理解紧凑模型如何保持智能体行为，我们引入Agentic RAG Capabilities（ARC），细粒度分析推理、搜索协调和响应合成。全面实验表明，DGPO使紧凑模型实现复杂的智能体搜索行为，甚至在某些情况下超越大教师模型。DGPO使在计算资源受限环境下实现智能体RAG成为可能。
英文摘要：
Reinforcement Learning has emerged as a dominant post-training approach to elicit agentic RAG behaviors such as search and planning from language models. Despite its success with larger models, applying RL to compact models (e.g., 0.5--1B parameters) presents unique challenges. The compact models exhibit poor initial performance, resulting in sparse rewards and unstable training. To overcome these difficulties, we propose Distillation-Guided Policy Optimization (DGPO), which employs cold-start initialization from teacher demonstrations and continuous teacher guidance during policy optimization. To understand how compact models preserve agentic behavior, we introduce Agentic RAG Capabilities (ARC), a fine-grained metric analyzing reasoning, search coordination, and response synthesis. Comprehensive experiments demonstrate that DGPO enables compact models to achieve sophisticated agentic search behaviors, even outperforming the larger teacher model in some cases. DGPO makes agentic RAG feasible in computing resource-constrained environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
