SCRIBE:结构化中层监督用于工具使用语言模型
SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models
AI总结:
SCRIBE通过引入中层抽象监督框架,减少多步推理中的信用分配难度,提升工具使用代理的可靠性与准确性。
摘要:
训练可靠的工具增强代理仍然是重大挑战,主要由于多步推理中的信用分配困难。虽然过程级奖励模型提供了有前景的方向,但现有基于LLM的评判者往往产生噪声和不一致的信号,因为缺乏细粒度、任务特定的评分标准来区分高层规划与低层执行。本文介绍SCRIBE(技能条件奖励与中间行为评估),一种强化学习框架,干预于新的中层抽象。SCRIBE将奖励建模扎根于精心编纂的技能原型库,将开放式的LLM评估转化为约束验证问题。通过将每个子目标路由到对应的原型,奖励模型配备了精确的结构化评分标准,显著减少了奖励方差。实验结果表明,SCRIBE在一系列推理和工具使用基准上实现了最先进的性能。特别是,它将Qwen3-4B模型的AIME25准确性从43.3%提升到63.3%,并在复杂的多轮工具交互中显著提高了成功率。进一步分析训练动态揭示了抽象层次间的共进化,其中中层技能的掌握始终先于有效高层规划行为的出现。最后,我们证明SCRIBE可以与低层工具优化相加,提供了一条可扩展且互补的路径,朝着更自主和可靠的工具使用代理迈进。
英文摘要:
Training reliable tool-augmented agents remains a significant challenge, largely due to the difficulty of credit assignment in multi-step reasoning. While process-level reward models offer a promising direction, existing LLM-based judges often produce noisy and inconsistent signals because they lack fine-grained, task-specific rubrics to distinguish high-level planning from low-level execution. In this work, we introduce SCRIBE (Skill-Conditioned Reward with Intermediate Behavioral Evaluation), a reinforcement learning framework that intervenes at a novel mid-level abstraction. SCRIBE grounds reward modeling in a curated library of skill prototypes, transforming open-ended LLM evaluation into a constrained verification problem. By routing each subgoal to a corresponding prototype, the reward model is equipped with precise, structured rubrics that substantially reduce reward variance. Experimental results show that SCRIBE achieves state-of-the-art performance across a range of reasoning and tool-use benchmarks. In particular, it improves the AIME25 accuracy of a Qwen3-4B model from 43.3% to 63.3%, and significantly increases success rates in complex multi-turn tool interactions. Further analysis of training dynamics reveals a co-evolution across abstraction levels, where mastery of mid-level skills consistently precedes the emergence of effective high-level planning behaviors. Finally, we demonstrate that SCRIBE is additive to low-level tool optimizations, providing a scalable and complementary pathway toward more autonomous and reliable tool-using agents.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。