通过文本引导的生成图像编辑与基于区域的控制改进3D场景风格化
Improved 3D Scene Stylization via Text-Guided Generative Image Editing with Region-Based Control
AI总结：
本文提出了一种改进的3D场景风格化方法，通过文本引导和基于区域的控制，提升风格一致性和视图一致性。
摘要：
近期在文本驱动的3D场景编辑和风格化方面的进展，利用了2D生成模型的强大能力，已经展示了有希望的结果。然而，在确保高质量的风格化和视图一致性的同时仍存在挑战。此外，将风格一致地应用到场景中的不同区域或物体上，尤其是在语义对应的情况下，是一项具有挑战性的任务。为了解决这些限制，我们引入了技术，以提高3D风格化质量，同时保持视图一致性和提供可选的区域控制风格转移。我们的方法通过重新训练初始的3D表示，使用源视图的风格化多视角2D图像来实现风格化。因此，确保风格化多视角图像的风格一致性和视图一致性至关重要。我们通过扩展风格对齐的深度条件视图生成框架，用单个参考基于的注意力共享机制取代完全共享的注意力机制，从而有效对齐不同视角的风格。此外，受最近的3D修补方法启发，我们利用多个深度图的网格作为单图像参考，进一步加强风格化图像之间的视图一致性。最后，我们提出了多区域重要性加权切片Wasserstein距离损失，允许使用现成模型的分割掩码将风格应用于不同的图像区域。我们证明了这个可选功能增强了风格转移的忠实性，并允许在场景的不同区域混合不同的风格。实验评估，包括定性和定量，证明了我们的流程有效提高了文本驱动的3D风格化结果。项目页面：https://haruolabs.github.io/improved-gs-style-page/
英文摘要：
Recent advances in text-driven 3D scene editing and stylization, which leverage the powerful capabilities of 2D generative models, have demonstrated promising outcomes. However, challenges remain in ensuring high-quality stylization and view consistency simultaneously. Moreover, applying style consistently to different regions or objects in the scene with semantic correspondence is a challenging task. To address these limitations, we introduce techniques that enhance the quality of 3D stylization while maintaining view consistency and providing optional region-controlled style transfer. Our method achieves stylization by re-training an initial 3D representation using stylized multi-view 2D images of the source views. Therefore, ensuring both style consistency and view consistency of stylized multi-view images is crucial. We achieve this by extending the style-aligned depth-conditioned view generation framework, replacing the fully shared attention mechanism with a single reference-based attention-sharing mechanism, which effectively aligns style across different viewpoints. Additionally, inspired by recent 3D inpainting methods, we utilize a grid of multiple depth maps as a single-image reference to further strengthen view consistency among stylized images. Finally, we propose Multi-Region Importance-Weighted Sliced Wasserstein Distance Loss, allowing styles to be applied to distinct image regions using segmentation masks from off-the-shelf models. We demonstrate that this optional feature enhances the faithfulness of style transfer and enables the mixing of different styles across distinct regions of the scene. Experimental evaluations, both qualitative and quantitative, demonstrate that our pipeline effectively improves the results of text-driven 3D stylization. Project Page: https://haruolabs.github.io/improved-gs-style-page/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
