自适应视觉-语言模型路由用于计算机使用代理
Adaptive Vision-Language Model Routing for Computer Use Agents
AI总结：
本文提出自适应VLM路由框架，通过动态选择模型降低推理成本，同时保持可靠性。在ScreenSpot-Pro数据集和OpenClaw基准测试中，AVR实现了高达78%的推理成本降低，并结合视觉困惑代理机制提升安全性。
摘要：
计算机使用代理（CUAs）通过依赖视觉-语言模型（VLM）来解读截图并预测基础工具调用，将自然语言指令转换为图形用户界面（GUI）操作，如点击、键盘输入和滚动。然而，VLM的接地准确性在不同模型间差异显著，而当前CUA系统通常将每个操作路由到单一固定模型，不考虑难度。本文提出自适应VLM路由（AVR）框架，在CUA调度器和VLM池之间插入轻量级语义路由层。对于每个工具调用，AVR通过多模态嵌入估计操作难度，探测小型VLM以测量置信度，并将操作路由到最便宜的模型，其预测准确性满足目标可靠性阈值。对于具有记忆能力的温暖代理，检索的上下文进一步缩小小型和大型模型之间的能力差距，允许许多操作无需升级即可处理。本文将路由形式化为成本-准确性权衡，推导出基于阈值的模型选择策略，并通过ScreenSpot-Pro接地数据和OpenClaw代理路由基准评估AVR。在这些设置中，AVR在所有大型模型基线内保持2个百分点以内，同时实现高达78%的推理成本降低。当与视觉困惑代理守护栏结合时，AVR还直接将高风险操作升级到最强可用模型，统一了效率和安全性。材料包括模型、基准和代码：https://github.com/vllm-project/semantic-router。
英文摘要：
Computer Use Agents (CUAs) translate natural-language instructions into Graphical User Interface (GUI) actions such as clicks, keystrokes, and scrolls by relying on a Vision-Language Model (VLM) to interpret screenshots and predict grounded tool calls. However, grounding accuracy varies dramatically across VLMs, while current CUA systems typically route every action to a single fixed model regardless of difficulty. We propose \textbf{Adaptive VLM Routing} (AVR), a framework that inserts a lightweight semantic routing layer between the CUA orchestrator and a pool of VLMs. For each tool call, AVR estimates action difficulty from multimodal embeddings, probes a small VLM to measure confidence, and routes the action to the cheapest model whose predicted accuracy satisfies a target reliability threshold. For \textit{warm} agents with memory of prior UI interactions, retrieved context further narrows the capability gap between small and large models, allowing many actions to be handled without escalation. We formalize routing as a cost--accuracy trade-off, derive a threshold-based policy for model selection, and evaluate AVR using ScreenSpot-Pro grounding data together with the OpenClaw agent routing benchmark. Across these settings, AVR projects inference cost reductions of up to 78\% while staying within 2 percentage points of an all-large-model baseline. When combined with the Visual Confused Deputy guardrail, AVR also escalates high-risk actions directly to the strongest available model, unifying efficiency and safety within a single routing framework. Materials are also provided Model, benchmark, and code: https://github.com/vllm-project/semantic-router.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
