统一语义Transformer用于3D场景理解
Unified Semantic Transformer for 3D Scene Understanding
AI总结：
提出UNITE，一个统一的语义Transformer，通过端到端训练从RGB图像直接预测多种密集语义属性，实现3D场景理解，并在多个任务上达到最先进性能。
摘要：
整体3D场景理解涉及捕捉和解析非结构化3D环境。由于现实世界的固有复杂性，现有模型主要被开发并局限于特定任务。我们引入UNITE，一个用于3D场景理解的统一语义Transformer，这是一种新颖的前馈神经网络，将多种3D密集语义室内任务统一在单个模型中。我们的模型以完全端到端的方式在未见过的场景上训练，仅需几秒钟即可推断完整的3D语义几何。我们的方法能够直接从RGB图像预测多个密集语义属性，包括3D场景分割、实例嵌入、开放词汇特征和关节。该方法使用2D蒸馏和自监督相结合的训练方式，并利用新颖的多视图损失确保3D视图一致性。我们证明UNITE在多个不同的密集室内语义任务上达到了最先进的性能，甚至在许多情况下超越了任务特定模型，超过了使用真实3D几何的方法。参见项目网站 unite-page.github.io。
英文摘要：
Holistic 3D scene understanding involves capturing and parsing unstructured 3D environments. Due to the inherent complexity of the real world, existing models have predominantly been developed and limited to be task-specific. We introduce UNITE, a Unified Semantic Transformer for 3D scene understanding, a novel feed-forward neural network that unifies a diverse set of 3D dense semantic indoor tasks within a single model. Our model operates on unseen scenes trained in a fully end-to-end manner and only takes a couple seconds to infer the full 3D semantic geometry. Our approach is capable of directly predicting multiple dense semantic attributes, including 3D scene segmentation, instance embeddings, open-vocabulary features, and articulations, solely from RGB images. The method is trained using a combination of 2D distillation, heavily relying on self-supervision and leverages novel multi-view losses designed to ensure 3D view consistency. We demonstrate that UNITE achieves state-of-the-art performance on several different dense indoor semantic tasks and even outperforms task-specific models, in many cases, surpassing methods that operate on ground truth 3D geometry. See the project website at unite-page.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
