arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.10078cs.RO
收藏

3D-TAFS:一种用于3D可供性分割的免训练框架

3D-TAFS: A Training-free Framework for 3D Affordance Segmentation

HKUST(香港科技大学);NUS(新加坡国立大学);University of Macau(澳门大学)

Meng Chu, Xuan Zhang, Zhedong Zheng, Tat-Seng Chua

AI总结:

针对3D物体交互可行性难题,提出免训练多模态框架3D-TAFS,融合2D与3D视觉及语言理解进行3D可供性分割,并构建大规模基准IndoorAfford-Bench验证了其有效性。

摘要:

将高级语言指令转化为物理世界中精确的机器人动作仍具挑战性,特别是在考虑与3D物体交互的可行性时。本文提出了3D-TAFS,一种用于3D可供性分割的新型免训练多模态框架。为全面评估此类框架,我们提出了IndoorAfford-Bench,这是一个包含9,248张图像的大规模基准,涵盖6个区域的20个不同室内场景,支持标准化交互查询。具体而言,该框架将大型多模态模型与专门的3D视觉网络相结合,实现了2D与3D视觉理解及语言理解的无缝融合。在IndoorAfford-Bench上的大量实验验证了3D-TAFS在处理跨多种设置的交互式3D可供性分割任务方面的能力,在各项指标上展现出有竞争力的性能。结果突显了3D-TAFS在复杂室内环境中基于可供性理解增强人机交互的潜力,推动了面向真实世界应用的更直观高效的机器人框架的发展。

英文摘要:

Translating high-level linguistic instructions into precise robotic actions in the physical world remains challenging, particularly when considering the feasibility of interacting with 3D objects. In this paper, we introduce 3D-TAFS, a novel training-free multimodal framework for 3D affordance segmentation. To facilitate a comprehensive evaluation of such frameworks, we present IndoorAfford-Bench, a large-scale benchmark containing 9,248 images spanning 20 diverse indoor scenes across 6 areas, supporting standardized interaction queries. In particular, our framework integrates a large multimodal model with a specialized 3D vision network, enabling a seamless fusion of 2D and 3D visual understanding with language comprehension. Extensive experiments on IndoorAfford-Bench validate the proposed 3D-TAFS's capability in handling interactive 3D affordance segmentation tasks across diverse settings, showcasing competitive performance across various metrics. Our results highlight 3D-TAFS's potential for enhancing human-robot interaction based on affordance understanding in complex indoor environments, advancing the development of more intuitive and efficient robotic frameworks for real-world applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑