扩展用于多视图全景分割的大视图合成模型
Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation
浏览论文内容
中文总结 AI 辅助
研究将大视图合成模型从外观渲染扩展到3D场景理解，提出复用冻结模型传播全景标签的分割管道，无需3D重建和特定训练，在ScanNet和Replica数据集上实验，分割质量佳且新视图合成表现优。
中文摘要 AI 辅助
大视图合成模型通过交叉视图注意力合成新视图，无需显式3D表示，近期研究表明其仅从RGB监督就能学习到准确的空间对应关系。我们发现这种对应关系不仅适用于外观。当非真实感信号如二进制编码的全景标签通过模型时，能以一致空间结构传播到新视图。基于此，我们首次将大视图合成模型从外观渲染扩展到3D场景理解。提出一种全景分割管道，复用冻结的视图合成模型将全景标签从输入视图传播到新视图，无需3D重建或对视图合成模型进行任何分割特定训练。在ScanNet上，我们的方法在分割质量上与基于高斯且需显式3D重建的方法相当，在新视图合成上比它们高出超7dB，标签传播还能跨数据集，在Replica上无需微调就超越这些方法。
英文摘要
Large view synthesis models synthesize novel views through cross-view attention without explicit 3D representations, and recent studies have shown that they learn accurate spatial correspondence from RGB supervision alone. We observe that this correspondence generalizes beyond appearance. When non-photorealistic signals such as binary encoded panoptic labels are passed through the model, they are propagated to novel views with consistent spatial structure. These results indicate that the correspondence learned for RGB view synthesis can also propagate view-independent per-pixel labels. From this observation, we present the first work to extend large view synthesis models beyond appearance rendering to 3D scene understanding. We propose a panoptic segmentation pipeline that reuses a frozen view synthesis model to propagate panoptic labels from input views to novel views, without 3D reconstruction or any segmentation-specific training of the view synthesis model. Given panoptic labels on the input views, we encode them into binary channel representations and pass them through the same model to render target-view segmentation. On ScanNet, our method achieves segmentation quality on par with Gaussian based approaches requiring explicit 3D reconstruction, while outperforming them in novel view synthesis by more than 7 dB. The label propagation also transfers across datasets, surpassing these approaches on Replica without any fine-tuning.
发表机构
- POSTECH（浦项科技大学）
- POSCO DX（浦项制铁DX公司）
- Chung-Ang University（中央大学）
- Seoul National University（首尔国立大学）
机构由 AI 辅助整理，请以论文原文为准。
