从数千到数十亿：通过来自2D VLM的渲染监督蒸馏实现3D视觉语言基础定位
From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs
AI总结：
提出LIFT-GS渲染监督蒸馏框架，借助可微渲染将SAM、CLIP和LLaMA等2D VLM监督迁移到3D表示，无需3D标注即显著提升开放词汇分割与指代表达定位性能。
摘要：
3D视觉语言基础定位面临一个根本性的数据瓶颈：尽管2D模型可在数十亿张图像上训练，但3D模型只能使用数千个带标注场景——这一相差六个数量级的差距严重限制了性能。我们提出LIFT-GS，这是一种实用的蒸馏技术，通过可微渲染连接3D与2D监督来克服该限制。LIFT-GS从点云预测3D Gaussian表示，并利用这些表示将预测的语言条件化3D掩码渲染到2D视图中，从而无需任何3D标注即可获得来自2D基础模型（SAM、CLIP、LLaMA）的监督。这种渲染监督形式支持完整编码器-解码器架构的端到端训练，并且本质上与模型无关。LIFT-GS在开放词汇实例分割上取得25.7% mAP的最先进结果（此前SOTA为20.2%），并在指代表达定位任务上持续取得10–30%的提升。值得注意的是，预训练实际上使微调数据集扩大了2倍，展现出强劲的扩展特性，表明3D VLG目前运行在严重数据稀缺的机制下。项目页面：https://liftgs.github.io
英文摘要：
3D vision-language grounding faces a fundamental data bottleneck: while 2D models train on billions of images, 3D models have access to only thousands of labeled scenes--a six-order-of-magnitude gap that severely limits performance. We introduce $\textbf{LIFT-GS}$, a practical distillation technique that overcomes this limitation by using differentiable rendering to bridge 3D and 2D supervision. LIFT-GS predicts 3D Gaussian representations from point clouds and uses them to render predicted language-conditioned 3D masks into 2D views, enabling supervision from 2D foundation models (SAM, CLIP, LLaMA) without requiring any 3D annotations. This render-supervised formulation enables end-to-end training of complete encoder-decoder architectures and is inherently model-agnostic. LIFT-GS achieves state-of-the-art results with $25.7\%$ mAP on open-vocabulary instance segmentation (vs. $20.2\%$ prior SOTA) and consistent $10-30\%$ improvements on referential grounding tasks. Remarkably, pretraining effectively multiplies fine-tuning datasets by 2X, demonstrating strong scaling properties that suggest 3D VLG currently operates in a severely data-scarce regime. Project page: https://liftgs.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
