arXiv 2606.061552026-06-04cs.ROcs.CVcs.MM

AffordanceVLA：一种通过可供性感知理解赋能动作生成的视觉-语言-动作模型

AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding

Peking University（北京大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Chinese University of Hong Kong（香港中文大学）；Knowin AI

Qize Yu, Jiadi You, Yuran Wang, Jiaqi Liang, Bowen Ping, Yang Tian, Yue Chen, Minghong Cai, Zeying Gong, Ruihai Wu, Yinchuan Li, Junwei Liang, Yingcong Chen

备注：: Preprint. Code and project page are available. Code: https://github.com/Skywalker-yqz/AffordanceVLA Project page: https://skywalker-yqz.github.io/AffordanceVLA/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出AffordanceVLA框架，通过引入结构化可供性预测作为任务导向的中间表示，解决VLA模型中语义空间与具身控制策略的结构不匹配问题，实现精确的感知-动作映射。

摘要：

视觉-语言-动作（VLA）模型利用预训练视觉-语言模型（VLM）的丰富世界知识来实现指令跟随的机器人操作。然而，VLM语义空间与具身控制策略之间的结构不匹配常常阻碍精确感知-动作映射的学习。为解决这一挑战，我们提出 extbf{AffordanceVLA}，一个统一框架，引入结构化可供性预测作为任务导向的中间表示，以建立更精确和鲁棒的感知-动作映射。具体而言，我们通过三个互补组件逐步建模操作先验：1） extbf{Which2Act}，通过视觉潜在预测进行以物体为中心的定位以抑制干扰；2） extbf{Where2Act}，通过可供性图估计进行2D交互定位；3） extbf{How2Act}，用于引导操作策略的3D几何推理。这些可供性线索提供了空间定位、语义条件化和动作耦合的中间表示，从而自然地桥接视觉、语言和动作。我们将这些模块集成到具有专门专家的混合Transformer（MoT）架构中，并使用三阶段训练策略和渐进式数据课程训练模型。为克服机器人数据集中密集可供性标签的稀缺性，我们还开发了一个鲁棒的自动化数据增强流水线。在仿真和真实世界中的大量实验表明，AffordanceVLA在多种操作场景中实现了强大的性能。

英文摘要：

Vision-Language-Action (VLA) models leverage the rich world knowledge of pretrained vision-language models (VLMs) to enable instruction-following robotic manipulation. However, the structural mismatch between VLM semantic spaces and embodied control policies often hinders the learning of precise perception--action mappings. To address this challenge, we propose \textbf{AffordanceVLA}, a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation to establish a more precise and robust perception--action mapping. Specifically, we progressively model manipulation priors through three complementary components: 1) \textbf{Which2Act} for object-centric grounding via visual latent prediction to suppress distractions; 2) \textbf{Where2Act} for 2D interaction localization via affordance map estimation; and 3) \textbf{How2Act} for 3D geometric reasoning to guide manipulation policies. These affordance cues provide spatially grounded, semantically conditioned, and action-coupled intermediate representations, thereby naturally bridging vision, language and action. We integrate these modules into a Mixture-of-Transformer (MoT) architecture with specialized experts and train the model using a three-stage training strategy with a progressive data curriculum. To overcome the scarcity of dense affordance labels in robotic datasets, we also develop a robust automated data augmentation pipeline. Extensive experiments on simulation and real-world demonstrate that AffordanceVLA achieves strong performance across diverse manipulation scenarios.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。