将视觉-语言-动作模型进化为具备即时工具使用能力的智能体
Evolve Vision-Language-Action Model into an Agent with On-the-fly Tool-use
浏览论文内容
中文总结 AI 辅助
该研究提出带工具使用的智能体机器人（ART）框架，调优VLA模型以利用工具模块，在模拟及真实任务中成功率较基线高20%，为VLA系统实际部署奠定基础。
中文摘要 AI 辅助
本文将端到端视觉-语言-动作（VLA）模型与智能体工具使用相结合，提出了带工具使用的智能体机器人（ART）。ART是一种工具注入框架，可对任意VLA模型进行调优，以利用现成工具模块实现低级视觉、高级 affordance（可供性）及具身性增强。与采用完整连续动作解空间的普通VLA模型相比，ART通过工具使用降低了动作解空间的复杂度，不仅提升了跨任务的泛化能力，还减少了数据依赖。为验证该框架高泛化性、低数据依赖的优势，我们首先构建了包含3万条工具使用轨迹及动作演示的数据集，其规模远小于基线方法所用数据集；随后设计了针对具挑战性环境中长轨迹工具使用推理的训练方案。实验表明，ART在模拟及真实世界任务（如陌生视角下的黑暗环境抓取）中，成功率较主流基线高出20%。实证结果凸显了基于智能体方法的优势：模块化工具利用实现了更高效的训练、轻量化部署及新工具的可扩展集成，该设计提升了鲁棒性、适应性与可扩展性，为VLA系统在复杂真实场景的实际部署铺平了道路。
英文摘要
This paper integrates end-to-end Visual-Language-Action (VLA) models with agentic tool-use to propose Agentic Robot with Tool-use (ART). ART is a tool-injection framework that tunes any VLA model to leverage off-the-shelf tool modules for low-level vision, high-level affordance, and embodiment enhancement. Compared to vanilla VLA models with a whole continuous action solution space, ART reduces the complexity of the action solution space through tool-use, which not only improves generalizability across different tasks but also reduces data dependency. To demonstrate the advantages (high generalizability and low data dependency) of this framework, we first built a dataset of 30K tool-use trajectories and action demonstrations, which is much smaller than those used by baseline methods. We then designed a training regimen for long-trajectory tool-use reasoning in challenging environments. Experiments show that ART achieves a 20% higher success rate than mainstream baselines on simulation and real-world tasks, such as pick-and-place in the dark at novel viewpoints. Empirical results highlight the benefits of an agent-based approach: modular tool utilization enables more efficient training, lightweight deployment, and scalable integration of new tools. This design fosters robustness, adaptability, and extensibility, paving the way for the practical deployment of VLA systems in complex real-world scenarios.
发表机构
- Astribot（星舟机器人）
- Tsinghua University（清华大学）
- Juxi Tech（矩芯科技）
- IntelliFusion Inc.（智融公司）
- The Chinese University of Hong Kong（香港中文大学）
机构由 AI 辅助整理，请以论文原文为准。
