ZeroScene: 一种基于单张图像的3D场景生成零样本框架及可控纹理编辑
ZeroScene: A Zero-Shot Framework for 3D Scene Generation from a Single Image and Controllable Texture Editing
AI总结：
ZeroScene通过零样本方法实现单图像3D场景生成与可控纹理编辑，结合大视觉模型先验知识，提升场景连贯性和渲染真实感。
摘要：
在3D内容生成领域，单图像场景重建方法仍难以在复杂环境中同时保证个体资产的质量和整体场景的连贯性，而纹理编辑技术往往难以维持局部连续性和多视角一致性。本文提出了一种新颖的系统ZeroScene，利用大视觉模型的先验知识，在零样本方式下完成单图像到3D场景重建和纹理编辑。ZeroScene从输入图像中提取物体级的2D分割和深度信息，推断场景内的空间关系。然后联合优化点云的3D和2D投影损失，更新物体姿态以实现精确的场景对齐，最终构建出一个连贯且完整的3D场景，涵盖前景和背景。此外，ZeroScene支持场景中物体的纹理编辑。通过约束扩散模型并引入基于掩码的逐步图像生成策略，我们有效地在多个视角下保持纹理一致性，并通过物理基础渲染（PBR）材质估计进一步增强渲染结果的真实感。实验结果表明，我们的框架不仅确保了生成资产的几何和外观准确性，还忠实重建了场景布局，并生成了高度细节化的纹理，与文本提示紧密匹配。
英文摘要：
In the field of 3D content generation, single image scene reconstruction methods still struggle to simultaneously ensure the quality of individual assets and the coherence of the overall scene in complex environments, while texture editing techniques often fail to maintain both local continuity and multi-view consistency. In this paper, we propose a novel system ZeroScene, which leverages the prior knowledge of large vision models to accomplish both single image-to-3D scene reconstruction and texture editing in a zero-shot manner. ZeroScene extracts object-level 2D segmentation and depth information from input images to infer spatial relationships within the scene. It then jointly optimizes 3D and 2D projection losses of the point cloud to update object poses for precise scene alignment, ultimately constructing a coherent and complete 3D scene that encompasses both foreground and background. Moreover, ZeroScene supports texture editing of objects in the scene. By imposing constraints on the diffusion model and introducing a mask-guided progressive image generation strategy, we effectively maintain texture consistency across multiple viewpoints and further enhance the realism of rendered results through Physically Based Rendering (PBR) material estimation. Experimental results demonstrate that our framework not only ensures the geometric and appearance accuracy of generated assets, but also faithfully reconstructs scene layouts and produces highly detailed textures that closely align with text prompts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
