arXiv 2511.124362025-11-16cs.RO

RoboAfford++：面向机器人操作与导航中多模态可供性学习的生成式AI增强数据集

RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation

Xiaomi EV（小米电动车）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学 Gallagher人工智能学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）

Xiaoshuai Hao, Yingbo Tang, Lingfeng Zhang, Yanbiao Ma, Yunfeng Diao, Ziyu Jia, Wenbo Ding, Hangjun Ye, Long Chen

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有视觉语言模型难以推断物理交互可执行位置的问题，本文构建生成式AI增强的多模态可供性数据集RoboAfford++及配套评估基准，实验证明该数据集可有效提升模型的可供性推理能力。

摘要：

机器人操作与导航是具身智能的基础能力，可支撑机器人与物理世界实现有效交互。实现这些能力需要对环境形成连贯理解，包括用于定位目标物体的目标识别、用于识别潜在交互区域的物体可供性，以及用于辨别物体放置和机器人移动最优区域的空间可供性。尽管视觉语言模型（Vision-Language Models, VLMs）在高层任务规划和场景理解方面表现优异，但它们往往难以推断物理交互的可执行位置，例如功能性抓取点和可允许放置区域。这一局限源于其训练数据集中缺乏物体与空间可供性的细粒度标注。为解决这一挑战，我们推出了RoboAfford++，这是一个面向机器人操作与导航多模态可供性学习的生成式AI增强数据集。我们的数据集包含869987张图像，搭配200万条问答（QA）标注，覆盖三项核心任务：基于属性与空间关系识别目标物体的物体可供性识别、精准定位操作所需功能部件的物体可供性预测，以及识别物体放置与机器人导航可用自由空间的空间可供性定位。作为该数据集的补充，我们提出了RoboAfford-Eval，这是一个用于评估真实场景下可供性感知预测能力的综合基准，包含针对上述三项任务精心标注的338个样本。大量实验结果表明，现有VLMs在可供性学习方面存在不足，而在RoboAfford++数据集上进行微调可显著提升其对物体与空间可供性的推理能力，验证了该数据集的有效性。

英文摘要：

Robotic manipulation and navigation are fundamental capabilities of embodied intelligence, enabling effective robot interactions with the physical world. Achieving these capabilities requires a cohesive understanding of the environment, including object recognition to localize target objects, object affordances to identify potential interaction areas and spatial affordances to discern optimal areas for both object placement and robot movement. While Vision-Language Models (VLMs) excel at high-level task planning and scene understanding, they often struggle to infer actionable positions for physical interaction, such as functional grasping points and permissible placement regions. This limitation stems from the lack of fine-grained annotations for object and spatial affordances in their training datasets. To tackle this challenge, we introduce RoboAfford++, a generative AI-enhanced dataset for multimodal affordance learning for both robotic manipulation and navigation. Our dataset comprises 869,987 images paired with 2.0 million question answering (QA) annotations, covering three critical tasks: object affordance recognition to identify target objects based on attributes and spatial relationships, object affordance prediction to pinpoint functional parts for manipulation, and spatial affordance localization to identify free space for object placement and robot navigation. Complementing this dataset, we propose RoboAfford-Eval, a comprehensive benchmark for assessing affordance-aware prediction in real-world scenarios, featuring 338 meticulously annotated samples across the same three tasks. Extensive experimental results reveal the deficiencies of existing VLMs in affordance learning, while fine-tuning on the RoboAfford++ dataset significantly enhances their ability to reason about object and spatial affordances, validating the dataset's effectiveness.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。