面向弱监督3D视觉定位的由粗到细语义匹配知识蒸馏
Distilling Coarse-to-Fine Semantic Matching Knowledge for Weakly Supervised 3D Visual Grounding
AI总结：
针对3D视觉定位需密集标注成本高的问题，提出由粗到细语义匹配的弱监督方法，结合掩码关键词重构衡量相似度，并将知识蒸馏到两阶段模型，在多个基准上验证了有效性。
摘要：
3D视觉定位任务是在3D场景中找到与给定句子查询对应的目标对象。尽管已有诸多方法被提出并取得了出色的性能，但它们都需要3D点云中的密集对象-句子对标注，这类标注既耗时又昂贵。为解决细粒度标注数据难以获取的问题，我们提出利用弱监督标注来学习3D视觉定位模型，即仅使用粗粒度的场景-句子对应关系来学习对象-句子关联。为此，我们设计了一种新颖的语义匹配模型，以由粗到细的方式分析对象候选框与句子之间的语义相似度。具体而言，我们首先提取对象候选框，并基于特征和类别相似度矩阵粗选出前K个候选对象。接下来，我们依次使用每个候选对象重构句子中被掩码的关键词，重构精度可精细反映每个候选对象与查询的语义相似度。此外，我们将这种由粗到细的语义匹配知识蒸馏到典型的两阶段3D视觉定位模型中，该模型通过充分利用现有架构中已被深入研究的结构，降低了推理成本并提升了性能。我们在ScanRefer、Nr3D和Sr3D上开展了大量实验，验证了所提方法的有效性。
英文摘要：
3D visual grounding involves finding a target object in a 3D scene that corresponds to a given sentence query. Although many approaches have been proposed and achieved impressive performance, they all require dense object-sentence pair annotations in 3D point clouds, which are both time-consuming and expensive. To address the problem that fine-grained annotated data is difficult to obtain, we propose to leverage weakly supervised annotations to learn the 3D visual grounding model, i.e., only coarse scene-sentence correspondences are used to learn object-sentence links. To accomplish this, we design a novel semantic matching model that analyzes the semantic similarity between object proposals and sentences in a coarse-to-fine manner. Specifically, we first extract object proposals and coarsely select the top-K candidates based on feature and class similarity matrices. Next, we reconstruct the masked keywords of the sentence using each candidate one by one, and the reconstructed accuracy finely reflects the semantic similarity of each candidate to the query. Additionally, we distill the coarse-to-fine semantic matching knowledge into a typical two-stage 3D visual grounding model, which reduces inference costs and improves performance by taking full advantage of the well-studied structure of the existing architectures. We conduct extensive experiments on ScanRefer, Nr3D, and Sr3D, which demonstrate the effectiveness of our proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
