arXiv 2207.064032022-07-13cs.CVcs.AIcs.CLcs.GRcs.LG

基于神经场的3D概念

3D Concept Grounding on Neural Fields

University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Massachusetts Institute of Technology（麻省理工学院）；Shanghai Jiao Tong University（上海交通大学）；MIT（麻省理工学院）；UMass Amherst（马萨诸塞大学阿默斯特分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）

Yining Hong, Yilun Du, Chunru Lin, Joshua B. Tenenbaum, Chuang Gan

备注：: Project page: http://3d-cg.csail.mit.edu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

摘要：

在本文中，我们解决了具有挑战性的问题，三维概念接地（即分割和学习视觉概念）通过查看RGBD图像和推理配对的问题和答案。现有的视觉推理方法通常利用监督方法来提取概念所基于的2D分割掩模。相比之下，人类能够将概念建立在图像的底层3D表示上。然而，传统上推断的3D表示（例如，点云、体素网格和网格）不能灵活地捕获连续的3D特征，因此使得基于所引用对象的语言描述将概念接地到3D区域具有挑战性。为了解决这两个问题，我们建议利用神经场的连续性和可微性来分割和学习概念。具体地，场景中的每个3D坐标被表示为高维描述符。然后可以通过计算3D坐标的描述符向量和语言概念的向量嵌入之间的相似性来执行概念基础，这使得分割和概念学习能够以可区分的方式在神经场上联合学习。因此，3D语义和实例分割都可以使用神经域之上的一组定义的神经运算符（例如，过滤和计数）。实验结果表明，我们提出的框架优于无监督/语言介导的分割模型的语义和实例分割任务，以及优于现有的模型上具有挑战性的3D感知视觉推理任务。此外，我们的框架可以很好地推广到看不见的形状类别和真实扫描。

英文摘要：

In this paper, we address the challenging problem of 3D concept grounding (i.e. segmenting and learning visual concepts) by looking at RGBD images and reasoning about paired questions and answers. Existing visual reasoning approaches typically utilize supervised methods to extract 2D segmentation masks on which concepts are grounded. In contrast, humans are capable of grounding concepts on the underlying 3D representation of images. However, traditionally inferred 3D representations (e.g., point clouds, voxelgrids, and meshes) cannot capture continuous 3D features flexibly, thus making it challenging to ground concepts to 3D regions based on the language description of the object being referred to. To address both issues, we propose to leverage the continuous, differentiable nature of neural fields to segment and learn concepts. Specifically, each 3D coordinate in a scene is represented as a high-dimensional descriptor. Concept grounding can then be performed by computing the similarity between the descriptor vector of a 3D coordinate and the vector embedding of a language concept, which enables segmentations and concept learning to be jointly learned on neural fields in a differentiable fashion. As a result, both 3D semantic and instance segmentations can emerge directly from question answering supervision using a set of defined neural operators on top of neural fields (e.g., filtering and counting). Experimental results show that our proposed framework outperforms unsupervised/language-mediated segmentation models on semantic and instance segmentation tasks, as well as outperforms existing models on the challenging 3D aware visual reasoning tasks. Furthermore, our framework can generalize well to unseen shape categories and real scans.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文