arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.08481cs.ROcs.CV
收藏

PhysVLM:使视觉语言模型能够理解机器人的物理可达性

PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachability

School of Traffic and Transportation, Beijing Jiaotong University(北京交通大学交通运输学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);ObjectEye Inc.(奥森科技(ObjectEye公司));Guangdong Polytechnic Normal University(广东技术师范大学)

Weijie Zhou, Manli Tao, Chaoyang Zhao, Haiyun Guo, Honghui Dong, Ming Tang, Jinqiao Wang

AI总结:

本文提出统一的 S-P Map 可达性表示与 PhysVLM,通过额外编码器融合可达性特征,并构建 Phys100K 和 EQA-phys,显著提升多机器人物理可达性推理性能。

摘要:

理解环境以及机器人的物理可达性对于任务执行至关重要。尽管最先进的视觉语言模型(VLM)在环境感知方面表现出色,但由于缺乏对机器人物理可达性的理解,它们在具身视觉推理任务中往往会生成不准确或不切实际的回答。为解决这一问题,我们提出了一种适用于多种机器人的物理可达性统一表示,即空间-物理可达性地图(Space-Physical Reachability Map,S-P Map),并提出 PhysVLM,这是一种将该可达性信息整合到视觉推理中的视觉语言模型。具体而言,S-P Map 将机器人的物理可达性抽象为一种通用空间表示,独立于特定机器人配置,使模型能够关注可达性特征,而不是机器人特有的参数。随后,PhysVLM 扩展了传统 VLM 架构,通过加入一个额外的特征编码器来处理 S-P Map,使模型能够在不损害其通用视觉语言能力的情况下对物理可达性进行推理。为训练和评估 PhysVLM,我们构建了一个大规模多机器人数据集 Phys100K,以及一个具有挑战性的基准 EQA-phys,其中包含六种不同机器人在仿真和真实环境中的任务。实验结果表明,PhysVLM 优于现有模型,在 EQA-phys 上相比 GPT-4o 实现了 14% 的提升,并在 RoboVQA-val 和 OpenEQA 基准上超过 RoboMamba 和 SpatialVLM 等先进具身 VLM。此外,S-P Map 表现出与各种 VLM 的强兼容性,将其整合到 GPT-4o-mini 中可带来 7.1% 的性能提升。

英文摘要:

Understanding the environment and a robot's physical reachability is crucial for task execution. While state-of-the-art vision-language models (VLMs) excel in environmental perception, they often generate inaccurate or impractical responses in embodied visual reasoning tasks due to a lack of understanding of robotic physical reachability. To address this issue, we propose a unified representation of physical reachability across diverse robots, i.e., Space-Physical Reachability Map (S-P Map), and PhysVLM, a vision-language model that integrates this reachability information into visual reasoning. Specifically, the S-P Map abstracts a robot's physical reachability into a generalized spatial representation, independent of specific robot configurations, allowing the model to focus on reachability features rather than robot-specific parameters. Subsequently, PhysVLM extends traditional VLM architectures by incorporating an additional feature encoder to process the S-P Map, enabling the model to reason about physical reachability without compromising its general vision-language capabilities. To train and evaluate PhysVLM, we constructed a large-scale multi-robot dataset, Phys100K, and a challenging benchmark, EQA-phys, which includes tasks for six different robots in both simulated and real-world environments. Experimental results demonstrate that PhysVLM outperforms existing models, achieving a 14\% improvement over GPT-4o on EQA-phys and surpassing advanced embodied VLMs such as RoboMamba and SpatialVLM on the RoboVQA-val and OpenEQA benchmarks. Additionally, the S-P Map shows strong compatibility with various VLMs, and its integration into GPT-4o-mini yields a 7.1\% performance improvement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑