arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.30185cs.AI
收藏

Dynamo: 视觉-语言代理的动态技能-工具演化

Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agents

Qwen Large Model Application Team, Alibaba(阿里云大模型应用团队);Alibaba Zhejiang University(阿里巴巴浙江大学);University of Waterloo(多伦多大学);Vector Institute(向量研究所);Nanjing University(南京大学);Binjiang Institute of Zhejiang University(浙江大学滨江学院)

Yutao Sun, Yanting Miao, Hao-Xuan Ma, Mengyu Zhou, Mingshuai Chen, Tiancheng Zhao, Dexin Wang, Lei Lv, Li Xu, Xiaoxi Jiang, Guanjun Jiang

AI总结:

提出Dynamo,一种无需训练、无需权重更新的框架,通过让冻结的VLM在少量标注数据上自我检查并演化可复用的推理技能和可执行的视觉工具,提升视觉推理性能。

摘要:

提升视觉-语言模型(VLM)在视觉推理上的性能通常需要重新训练或手工设计的提示和工具。我们提出Dynamo,一种无需训练的框架,它适应冻结的VLM而无需任何权重更新。在一个小的标注训练子集上,代理检查自身正确和错误的尝试,并演化两种互补能力:用于认知瓶颈的可复用推理技能,以及用于感知瓶颈的可执行视觉工具。每个生成的工具都与一个指定何时调用它的技能配对,两种能力类型都累积在一个持久库中。在四个视觉推理基准和五个VLM骨干网络上,Dynamo在所有20个模型-基准设置上提升了直接推理性能(平均+5.6准确率)。当工具集预先给出时,该框架学习何时调用每个工具,并且每步工具选择在所有测试的骨干网络上都有提升。与任务特定的强化学习(VTool-R1, DeepEyes)相比,Dynamo以极少的计算量缩小了65-99%的RL差距,并且当RL可用时,可以与之相加组合。

英文摘要:

Improving vision-language models (VLMs) on visual reasoning typically requires retraining or hand-designed prompts and tools. We present Dynamo, a training-free framework that adapts a frozen VLM without any weight updates. On a small labeled training subset, the agent inspects its own correct and incorrect attempts and evolves two complementary capabilities: reusable reasoning skills for cognitive bottlenecks, and executable visual tools for perceptual ones. Each generated tool is paired with a skill that specifies when to invoke it, and both capability types accumulate in a persistent library. Across four visual reasoning benchmarks and five VLM backbones, Dynamo improves direct inference on all 20 model--benchmark settings (avg. +5.6 acc). When the tool set is given in advance, the framework learns when to call each tool, and per-step tool choice improves on every tested backbone. Against task-specific RL (VTool-R1, DeepEyes), Dynamo closes 65--99% of the RL gap at a fraction of the compute, and combines additively with RL when available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑