Translating high-level linguistic instructions into precise robotic actions in the physical world remains challenging, particularly when considering the feasibility of interacting with 3D objects. In this paper, we introduce 3D-TAFS, a novel training-free multimodal framework for 3D affordance segmentation. To facilitate a comprehensive evaluation of such frameworks, we present IndoorAfford-Bench, a large-scale benchmark containing 9,248 images spanning 20 diverse indoor scenes across 6 areas, supporting standardized interaction queries. In particular, our framework integrates a large multimodal model with a specialized 3D vision network, enabling a seamless fusion of 2D and 3D visual understanding with language comprehension. Extensive experiments on IndoorAfford-Bench validate the proposed 3D-TAFS's capability in handling interactive 3D affordance segmentation tasks across diverse settings, showcasing competitive performance across various metrics. Our results highlight 3D-TAFS's potential for enhancing human-robot interaction based on affordance understanding in complex indoor environments, advancing the development of more intuitive and efficient robotic frameworks for real-world applications.
RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping
The Chinese University of Hong Kong(香港中文大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Dexmal;Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);SKL-IOTSC, CIS, University of Macau(澳门科学馆-物联网与智能系统研究中心,澳门大学)
3D Affordance Keypoint Detection for Robotic Manipulation
Advanced Robotics Centre, National University of Singapore(新加坡国立大学先进机器人中心);Department of Mechanical Engineering, National University of Singapore(新加坡国立大学机械工程系)
Bayesian Deep Learning for Affordance Segmentation in images
Instituto de Ingenieria de Aragon (I3A)(阿拉贡工程研究所);Universidad de Zaragoza(萨拉戈萨大学)
示能表示是机器人技术中的一个基本概念,因为它们根据智能体的感觉运动能力和环境将智能体的可用动作联系起来。我们提出了一种新的贝叶斯深度网络来检测图像中的启示,同时我们量化了空间水平上的随机和认知方差的分布。我们采用Mask-RCNN架构来学习使用Monte Carlo dropout的概率表示。我们的结果优于确定性网络的最先进的。我们将这种改进归因于编码器上更好的概率特征空间表示和在掩模生成时引起的贝叶斯可变性,其更好地适应对象轮廓。我们还介绍了新的基于概率的掩模质量的措施,揭示了语义和空间上的概率实例分割模型的差异。我们通过比较二进制掩码而不是预测的边界框来修改现有的概率检测质量度量,从而实现概率分割的更细粒度评估。我们发现任意方差的轮廓的对象,由于相机噪声,而认知方差出现在视觉挑战像素。
SurgAM: Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy
Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程系);Department of Thoracic Surgery, Peking University People’s Hospital(北京大学人民医院胸外科);Thoracic Oncology Institute, Peking University People’s Hospital(北京大学人民医院胸部肿瘤研究所);Research Unit of Intelligence Diagnosis and Treatment in Early Non-small Cell Lung Cancer, Chinese Academy of Medical Sciences(中国医学科学院早期非小细胞肺癌智能诊断与治疗研究组);Institute of Advanced Clinical Medicine, Peking University(北京大学先进临床医学院);Beijing Key Laboratory of Innovative Application of Big Data in Lung Cancer, Peking University People’s Hospital(北京大学人民医院肺癌大数据创新应用北京市重点实验室)
AnchorDP3: 3D Affordance Guided Sparse Diffusion Policy for Robotic Manipulation
Jingdong Technology Information Technology Co., Ltd(京东科技信息技术有限公司);Tsinghua University(清华大学);Fudan University(复旦大学);Southeast University(东南大学);University of Science and Technology of China(中国科学技术大学)