ViewRefer：利用GPT与原型引导掌握多视图知识实现3D视觉定位
ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidance
AI总结：
ViewRefer提出多视图3D视觉定位框架，利用GPT扩展文本描述并结合跨视图注意力与可学习多视图原型，在三个基准上显著超越现有方法。
摘要：
从多视图输入理解3D场景已被证明可以缓解3D视觉定位中的视图差异问题。然而，现有方法通常忽略文本模态中嵌入的视图线索，并且未能权衡不同视图的相对重要性。本文提出ViewRefer，这是一个用于3D视觉定位的多视图框架，探索如何从文本和3D两种模态中掌握视图知识。在文本分支中，ViewRefer利用大规模语言模型（如GPT）的多样化语言知识，将单个定位文本扩展为多个几何一致的描述。同时，在3D模态中，引入了一个具有跨视图注意力的Transformer融合模块，以增强不同视图间对象的交互。在此基础上，我们进一步提出一组可学习的多视图原型，这些原型记忆不同视图的场景无关知识，并从两个角度增强框架：一个用于获得更鲁棒文本特征的视图引导注意力模块，以及在最终预测阶段的视图引导评分策略。通过我们设计的范式，ViewRefer在三个基准上取得了优越性能，在Sr3D、Nr3D和ScanRefer上分别超过第二名+2.8%、+1.5%和+1.35%。代码已发布在https://github.com/Ivan-Tang-3D/ViewRefer3D。
英文摘要：
Understanding 3D scenes from multi-view inputs has been proven to alleviate the view discrepancy issue in 3D visual grounding. However, existing methods normally neglect the view cues embedded in the text modality and fail to weigh the relative importance of different views. In this paper, we propose ViewRefer, a multi-view framework for 3D visual grounding exploring how to grasp the view knowledge from both text and 3D modalities. For the text branch, ViewRefer leverages the diverse linguistic knowledge of large-scale language models, e.g., GPT, to expand a single grounding text to multiple geometry-consistent descriptions. Meanwhile, in the 3D modality, a transformer fusion module with inter-view attention is introduced to boost the interaction of objects across views. On top of that, we further present a set of learnable multi-view prototypes, which memorize scene-agnostic knowledge for different views, and enhance the framework from two perspectives: a view-guided attention module for more robust text features, and a view-guided scoring strategy during the final prediction. With our designed paradigm, ViewRefer achieves superior performance on three benchmarks and surpasses the second-best by +2.8%, +1.5%, and +1.35% on Sr3D, Nr3D, and ScanRefer. Code is released at https://github.com/Ivan-Tang-3D/ViewRefer3D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
