用于3D视觉定位的细粒度空间与语言损失
Fine-Grained Spatial and Verbal Losses for 3D Visual Grounding
AI总结：
针对3D视觉定位中现有损失函数忽视空间与词级结构的问题，本文提出视觉偏移损失和语言跨度损失，并设计AsphaltNet架构，在ReferIt3D基准上取得竞争性结果。
摘要：
3D视觉定位旨在识别3D场景中由伴随语言描述所指代的实例。尽管在常用的基于选择的定位框架内已提出了多种架构，但所使用的损失函数相对研究不足。特别是，大多数方法依赖于对候选实例预测分布的基本监督交叉熵损失，这未能建模实例间的空间关系以及语言指代内部的细粒度词级结构。少数尝试通过从描述中学习所指实例的类别来全局监督语言嵌入，或采用视觉-语言对比以更好分离实例嵌入，但并未从根本上消除上述局限性。针对这些不足，我们引入了两种用于3D视觉定位的新损失：一种是对从每个实例到真实所指实例的回归向量偏移进行监督的视觉级偏移损失，另一种是对描述中所指实例的词级跨度预测进行监督的语言相关跨度损失。此外，我们为新的3D视觉定位架构AsphaltNet的视觉-语言融合模块配备了自上而下的双向注意力融合块，这使得我们两种损失的监督信号能够传播到网络的相应反向分支，从而帮助后者学习上下文感知的实例嵌入和定位感知的语言嵌入。AsphaltNet提出了新颖的辅助损失以辅助3D视觉定位，在ReferIt3D基准上取得了与最先进方法相竞争的结果。
英文摘要：
3D visual grounding consists of identifying the instance in a 3D scene which is referred by an accompanying language description. While several architectures have been proposed within the commonly employed grounding-by-selection framework, the utilized losses are comparatively under-explored. In particular, most methods rely on a basic supervised cross-entropy loss on the predicted distribution over candidate instances, which fails to model both spatial relations between instances and the internal fine-grained word-level structure of the verbal referral. Sparse attempts to additionally supervise verbal embeddings globally by learning the class of the referred instance from the description or employing verbo-visual contrast to better separate instance embeddings do not fundamentally lift the aforementioned limitations. Responding to these shortcomings, we introduce two novel losses for 3D visual grounding: a visual-level offset loss on regressed vector offsets from each instance to the ground-truth referred instance and a language-related span loss on predictions for the word-level span of the referred instance in the description. In addition, we equip the verbo-visual fusion module of our new 3D visual grounding architecture AsphaltNet with a top-down bidirectional attentive fusion block, which enables the supervisory signals from our two losses to propagate to the respective converse branches of the network and thus aid the latter to learn context-aware instance embeddings and grounding-aware verbal embeddings. AsphaltNet proposes novel auxiliary losses to aid 3D visual grounding with competitive results compared to the state-of-the-art on the ReferIt3D benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
