arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.01414cs.CV
收藏

Agent技能应超越文本:视觉技能的必要性

Agent Skills Should Go Beyond Text: The Case for Visual Skills

Peking University(北京大学);University of Wisconsin(威斯康星大学);MIT-IBM Watson AI Lab(麻省理工-IBM沃森人工智能实验室)

Binxiao Xu, Ruichuan An, Bocheng Zou, Hang Hua

AI总结:

针对现有技能学习方法仅存储文本经验导致视觉任务瓶颈的问题,提出多模态技能范式,结合文本逻辑与视觉支持,通过自动系统将经验转化为可复用的视觉技能,在GUI等视觉任务中显著优于纯文本技能。

摘要:

可复用技能是扩展智能体能力的关键机制,使智能体能够积累经验并解决日益复杂的任务。然而,现有大多数技能学习方法仅将可复用经验存储为文本资产,如指令、推理轨迹或总结的轨迹。我们认为,这种纯文本范式为视觉中心任务造成了根本性瓶颈,因为可复用知识通常依赖于空间布局、视觉定位、细粒度外观和局部状态变化。为解决这一局限,我们提出\NAME,一种结合声明式文本逻辑与显式视觉支持的多模态技能范式。我们区分三种可复用形式:静态先验(用于稳定的空间惯例)、动态先验(用于现场视觉工作记忆)以及交错视觉技能(将有序文本步骤绑定到源帧、截图或页面区域,以证明其合理性)。视觉技能不仅描述要做什么,还编码了在哪里看、如何检查以及如何验证视觉结果。为了规模化构建视觉技能,我们引入\SYSTEM,一种自动系统,通过保留任务轨迹中的文本推理、空间引用、视觉边界和交互模式,将智能体经验转化为可复用的多模态技能。在GUI和其他视觉中心任务上的实验表明,视觉技能始终优于纯文本技能,尤其是在成功需要空间对应、视觉证据和状态感知交互时。这些结果支持我们的核心立场:可复用智能体技能应超越文本,成为未来多模态智能体的多模态资产。

英文摘要:

Reusable skills are a key mechanism for extending agent capabilities, allowing agents to accumulate experience and solve increasingly complex tasks. Yet most existing skill-learning methods store reusable experience as text-only assets, such as instructions, reasoning traces, or summarized trajectories. We argue that this text-only paradigm creates a fundamental bottleneck for visual-centric tasks, where reusable knowledge often depends on spatial layout, visual grounding, fine-grained appearance, and localized state changes. To address this limitation, we propose \textbf{\NAME}, a multimodal skill paradigm that combines declarative textual logic with explicit visual support. We distinguish three reusable forms: static priors for stable spatial conventions, dynamic priors for in-situ visual working memory, and interleaved visual skills that bind ordered text steps to the source frames, screenshots, or page regions that justify them. Rather than only describing what to do, visual skills also encode where to look, how to inspect, and how to verify visual outcomes. To scale visual-skill construction, we introduce \textbf{\SYSTEM}, an automatic system that converts agent experience into reusable multimodal skills by preserving textual reasoning, spatial references, visual boundaries, and interaction patterns from task trajectories. Experiments on GUI and other visual-centric tasks show that visual skills consistently outperform text-only skills, particularly when success requires spatial correspondence, visual evidence, and state-aware interaction. These results support our central position: reusable agent skills should go beyond text and become multimodal assets for future multimodal agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑