arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.04404cs.CVcs.AI
收藏

3D-Agent:三模态多智能体协作用于可扩展的3D物体标注

3D-Agent:Tri-Modal Multi-Agent Collaboration for Scalable 3D Object Annotation

Sun Yat-sen University(中山大学);Shanghai Jiao Tong University(上海交通大学);Snap Inc.(Snap公司)

Jusheng Zhang, Yijia Fan, Zimo Wen, Jian Wang, Keze Wang

AI总结:

Tri MARF通过三模态多智能体协作提升大规模3D物体标注效率与精度

摘要:

受自动驾驶机器人和增强现实中3D物体标注应用的推动,3D标注面临超越2D标注的挑战,包括空间复杂性、遮挡和视角不一致。现有基于单模型的方法往往难以有效解决这些问题。我们提出Tri MARF,一种新的框架,整合三模态输入,包括2D多视角图像、文本描述和3D点云,在多智能体协作架构中以增强大规模3D标注。Tri MARF由三个专门的智能体组成:一个视觉语言模型智能体用于生成多视角描述,一个信息聚合智能体用于选择最优描述,以及一个门控智能体,用于将文本语义对齐3D几何以实现精细化标注。在Objaverse、LVIS、Objaverse XL和ABO上进行的大量实验表明,Tri MARF显著优于现有方法,达到CLIPScore为88.7,相比现有最先进方法的检索准确率为45.2和43.8,在ViLT R@5上,以及在单个NVIDIA A100 GPU上每小时可处理多达12000个对象。

英文摘要:

Driven by applications in autonomous driving robotics and augmented reality 3D object annotation presents challenges beyond 2D annotation including spatial complexity occlusion and viewpoint inconsistency Existing approaches based on single models often struggle to address these issues effectively We propose Tri MARF a novel framework that integrates tri modal inputs including 2D multi view images textual descriptions and 3D point clouds within a multi agent collaborative architecture to enhance large scale 3D annotation Tri MARF consists of three specialized agents a vision language model agent for generating multi view descriptions an information aggregation agent for selecting optimal descriptions and a gating agent that aligns textual semantics with 3D geometry for refined captioning Extensive experiments on Objaverse LVIS Objaverse XL and ABO demonstrate that Tri MARF substantially outperforms existing methods achieving a CLIPScore of 88 point 7 compared to prior state of the art methods retrieval accuracy of 45 point 2 and 43 point 8 on ViLT R at 5 and a throughput of up to 12000 objects per hour on a single NVIDIA A100 GPU

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑