RubricRefine: 通过无训练预执行细化提升工具使用代理的可靠性
RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement
AI总结：
RubricRefine通过预执行语义合同验证，在无执行尝试的情况下提升工具使用代理的可靠性，平均在M3ToolEval上达到0.86，比现有推理时间基线低2.6倍。
摘要：
迭代自我细化是一种流行的推理时间可靠性技术，但其在代码模式工具使用中的有效性严重依赖反馈信号的结构：无结构的批评在不同模型间不一致，即使使用真实执行反馈进行修订也只能小幅提升（0.75 vs. 0.65基线）。主导的失败是跨工具合同违规（错误输出形状、错误工具路由、断裂的参数来源），这些失败在完成运行时不会引发错误，使运行时反馈不足。我们引入RubricRefine，一种无训练的预执行语义合同验证方法，生成任务和注册表特定的评分标准，对候选代码进行显式合同检查评分，并在任何执行发生前迭代修复失败。RubricRefine在M3ToolEval上达到0.86，平均跨七个模型，无执行尝试，比现有推理时间基线提升最高2.6倍。性能在主要单步API-Bank上保持稳定，与方法对跨工具合同结构的依赖一致。评分类别消融和校准分析进一步阐明了该方法何时及为何有效。
英文摘要：
Iterative self-refinement is a popular inference-time reliability technique, but its effectiveness in code-mode tool use depends heavily on the structure of the feedback signal: unstructured critique helps inconsistently across models, and even revision with real execution feedback improves only modestly. The dominant failures are inter-tool contract violations (wrong output shape, incorrect tool routing, broken argument provenance) that run to completion without raising errors, making runtime feedback insufficient. We introduce RubricRefine, a training-free method for pre-execution contract checking that generates task- and registry-specific rubrics, scores candidate code against explicit contract checks, and iteratively repairs failures before any execution occurs. RubricRefine reaches $0.86$, averaged across seven models, on M3ToolEval with zero execution attempts, improving over prior inference-time baselines at lower latency than rubric-guided reranking. Performance remains flat on the predominantly single-step API-Bank, consistent with the method's reliance on inter-tool contract structure. Results on AppWorld demonstrate that our method maintains an advantage in the multi-turn setting. Because the rubric is derived from the supplied tool documentation, the method's advantage survives incomplete documentation but reverses under incorrect documentation. A rubric-category ablation identifies which rules are load-bearing, and top-bin calibration enables early stopping even where aggregate calibration is poor.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
