发表机构
Stanford University（斯坦福大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究探索能否用基础模型通过视觉接口控制机器人，提出 VIA 框架，将机器人控制转为代理任务，该框架无需特定微调及特权信息，能零样本解决多种桌面操作任务，凭借基础模型能力提升取得高成功率，证明前沿代理技能可借合适接口用于机器人控制。
AI 中文摘要
机器人操作是一项复杂任务，需要视觉理解、物理推理、规划和闭环控制。通用基础模型在某些方面能力显著，当前方法常通过在机器人数据上微调将其转换为视觉-语言-动作模型来输出低级动作，但因微调数据和计算有限，VLA 能力受限。受基础模型通过视觉接口操作软件能力增长的启发，提出 VIA 框架，将机器人控制重塑为代理任务，由现成的基础模型驱动代理通过基于浏览器的 3D 接口控制机器人，无需特定微调及特权状态信息。VIA 继承代理的一般推理等能力，能零样本解决多种桌面操作任务，在不同任务上取得了高成功率，结果表明合适接口下前沿代理技能可直接用于机器人控制。
英文摘要
Robot manipulation is a complex task that requires visual perception, physical reasoning, planning, and closed-loop control. General-purpose foundation models (FMs) have grown remarkably capable of some of these, especially perception and reasoning. Inspired by the growing ability of FM-powered agents to operate software through visual interfaces, we ask whether that same competence suffices to control a robot directly given the right interface. We present VIA (Visual Interface Agent), a framework that recasts robot control as an agentic task where an off-the-shelf agent drives a manipulator directly through an interface. The interface is a virtual workspace of interactable visual components where the agent can probe pixels of interest with mouse-like tools and command a virtual end-effector to set new target poses with a small set of general tools. We show that VIA enables agents of varying capabilities to perform diverse tabletop manipulation tasks in simulation. We also use VIA to drive an off-the-shelf mobile manipulation robot in the real world to complete tasks that require both navigation and manipulation. Both settings are zero-shot, requiring no robot training or additional action primitives. Performance scales well with the size and strength of the underlying FMs. These results suggest that frontier agents already possess skills that transfer directly to robot control given the right interface: your coding or computer-use agent is, in a sense, secretly a robot-control agent.
