无需训练的实例感知3D场景重建与基于扩散的视图合成从稀疏图像
Training-Free Instance-Aware 3D Scene Reconstruction and Diffusion-Based View Synthesis from Sparse Images
AI总结：
本文提出一种无需训练的系统，从稀疏未定位RGB图像重建、理解并渲染3D室内场景，通过点云重建、2D到3D实例提升和扩散模型渲染实现高保真结果，支持实例编辑。
摘要：
我们介绍了一种新颖的、无需训练的系统，用于从稀疏的未定位RGB图像中重建、理解和渲染3D室内场景。与传统辐射场方法不同，该系统无需密集视图和每场景优化即可实现高保真结果。该系统集成了三个关键创新：（1）一个稳健的点云重建模块，通过基于变形的异常移除策略过滤不可靠的几何结构；（2）一个基于变形的2D到3D实例提升机制，将2D分割掩码传播到一致的实例感知3D表示；（3）一种新的渲染方法，将点云投影到新视图并利用3D-aware扩散模型细化渲染结果。我们的方法利用扩散模型的生成能力来补偿缺失的几何结构，并在稀疏输入条件下增强真实感。我们进一步证明，通过仅修改点云即可在系统中自然支持实例级别的场景编辑，如实例移除，从而合成一致的编辑视图而无需重新训练。我们的结果为无需依赖场景特定优化的高效、可编辑的3D内容生成开辟了新方向。项目页面：https://jiatongxia.github.io/TID3R/
英文摘要：
We introduce a novel, training-free system for reconstructing, understanding, and rendering 3D indoor scenes from a sparse set of unposed RGB images. Unlike traditional radiance field approaches that require dense views and per-scene optimization, our pipeline achieves high-fidelity results without any training or pose preprocessing. The system integrates three key innovations: (1) A robust point cloud reconstruction module that filters unreliable geometry using a warping-based anomaly removal strategy; (2) A warping-guided 2D-to-3D instance lifting mechanism that propagates 2D segmentation masks into a consistent, instance-aware 3D representation; and (3) A novel rendering approach that projects the point cloud into new views and refines the renderings with a 3D-aware diffusion model. Our method leverages the generative power of diffusion to compensate for missing geometry and enhances realism, especially under sparse input conditions. We further demonstrate that object-level scene editing such as instance removal can be naturally supported in our pipeline by modifying only the point cloud, enabling the synthesis of consistent, edited views without retraining. Our results establish a new direction for efficient, editable 3D content generation without relying on scene-specific optimization. Project page: https://jiatongxia.github.io/TID3R/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
