arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.16604cs.LG
收藏

R2V Agent:教SLMs何时请求帮助

R2V Agent: Teaching SLMs When to Ask for Help

New York University Tandon School of Engineering(纽约大学Tandon工程学院);University of California, San Diego(加州大学圣地亚哥分校);Stanford University(斯坦福大学);ServiceNow Research(ServiceNow研究);Mila - Quebec AI Institute(魁北克AI研究院);Université de Montréal(蒙特利尔大学)

Raghu Vamshi Hemadri, Humaira Firdowse Mohammed, Rishabh Maheshwary, Srivatsava Daruru, Sagar Davasam, Vikas Yadav, Srinivas Sunkara, Sai Rajeswar

AI总结:

R2V-Agent通过风险校准的SLM-LLM路由框架提升交互代理的可靠性,结合小型语言模型策略、更强的教师LLM、轻量级过程验证器和校准的步骤路由器,在多个基准测试中显著提升性能与成本效率。

摘要:

高效的智能体系统应仅在本地模型可能失败的决策中承担昂贵的前沿模型成本。现有LLM级联通常在执行前路由整个查询,但任务难度在轨迹中途变化——在不稳定的工具调用、截断观察或叠加的本地错误后发生,使预执行路由变得脆弱。我们引入R2V-Agent,一种用于交互智能体的风险校准SLM-LLM路由框架。R2V结合四个组件:一个蒸馏的小语言模型(SLM)策略、一个更强的教师LLM、一个轻量级的过程验证器,该验证器在每一步评分候选动作,以及一个校准的步骤级路由器。路由器是我们的核心贡献:在SLM训练后,它在每个步骤估计残余失败风险,并仅在教师干预必要时升级。为了使路由问题明确界定,我们首先使用标准离线流水线训练一个稳定的本地SLM:通过行为克隆(BC)在教师轨迹上进行训练,随后通过验证器引导的直接偏好优化(DPO)进行优化,结合一致性正则化。然后,路由器在该固定策略的残余失败上进行训练,使用Brier校准的概率估计和一个条件价值-at-风险(CVaR)约束的目标,该目标惩罚所有扰动种子下的最坏情况失败。在HumanEval+、TextWorld和TerminalBench四个SLM骨干网络上,R2V改进了可靠性-成本前沿:它在HumanEval+上达到94.3%的成功率,仅需0.60%的LLM升级,将TextWorld从64.6%的SLM-only成功提升到98.2%在41.7%的升级率下,最终在TerminalBench上达到93.3%的成功率,在33.9%的LLM调用中,大致是启发式路由器成本的一半。

英文摘要:

Efficient agentic systems should incur expensive frontier-model costs only on decisions where a cheaper local model is likely to fail. Existing LLM cascades usually route whole queries before execution, but task difficulty shifts mid-trajectory - after flaky tool calls, truncated observations, or compounding local errors - making pre-execution routing brittle. We introduce \textbf{R2V-Agent}, a risk-calibrated SLM-LLM routing framework for interactive agents. R2V combines four components: a distilled small language model (SLM) policy, a stronger teacher LLM, a lightweight process verifier that scores candidate actions at each step, and a calibrated step-level router. The router is our central contribution: after the SLM is trained, it estimates residual failure risk at each step and escalates only when teacher intervention is warranted. To make the routing problem well-defined, we first train a stable local SLM using a standard offline pipeline: behavioral cloning (BC) on teacher trajectories, followed by verifier-guided Direct Preference Optimization (DPO) with consistency regularization. The router is then trained on this fixed policy's residual failures using Brier-calibrated probability estimation and a Conditional Value-at-Risk (CVaR)-constrained objective that penalizes worst-case failures across perturbation seeds. Across HumanEval+, TextWorld, and TerminalBench with four SLM backbones, R2V improves the reliability-cost frontier: it achieves $94.3\%$ HumanEval+ success with $0.60\%$ LLM escalation, recovers TextWorld from $64.6\%$ SLM-only success to $98.2\%$ at $41.7\%$ escalation, and reaches $93.3\%$ TerminalBench success at $33.9\%$ LLM calls, roughly half the heuristic-router cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑