arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.13663cs.CV

Free-Editor:零样本文本驱动的3D场景编辑

Free-Editor: Zero-shot Text-driven 3D Scene Editing

University of Central Florida(中佛罗里达大学);Wayne State University(韦恩州立大学)

Nazmul Karim, Hasan Iqbal, Umar Khalid, Jing Hua, Chen Chen

AI总结:

针对现有3D场景编辑需重训练模型、存在多视图不一致的问题,提出免训练的Free-Editor技术,通过单视图编辑和Edit Transformer实现高效一致的文本驱动3D场景编辑,速度达SOTA方法的20倍。

摘要:

文本生成图像(Text-to-Image, T2I)扩散模型近期凭借其在2D内容生成与编辑中的通用性和易用性受到广泛关注。然而,由于大规模数据集的匮乏,专门训练用于3D场景编辑的扩散模型颇具挑战。当前,编辑3D场景要么需要重新训练模型以适配各类3D编辑需求,要么需要针对每种独特的编辑类型开发专门方法。此外,现有最优(state-of-the-art, SOTA)技术需要来自同一场景的多张同步编辑图像才能实现有效的场景编辑。鉴于T2I模型目前的局限性,在多张图像间实现一致的编辑效果仍然困难,这会导致编辑中的多视图不一致问题。当使用这些图像时,这种不一致性会削弱3D场景编辑的性能。在本研究中,我们提出了一种全新的免训练3D场景编辑技术,名为Free-Editor,它使用户在测试阶段无需重新训练模型即可编辑3D场景。我们的方法通过采用单视图编辑方案,有效解决了现有SOTA方法中存在的多视图风格不一致问题。具体而言,我们证明仅修改单个视图即可实现对特定3D场景的编辑。为实现这一点,我们提出了Edit Transformer(编辑Transformer),它分别通过自视图注意力机制和跨视图注意力机制来确保视图内一致性和视图间风格迁移。通过消除模型重训练和多视图编辑的需求,我们的方法大幅减少了编辑时间和内存资源消耗,运行速度约为SOTA方法的20倍。我们在多个基准数据集上开展了大量实验,展示了所提技术多样化的编辑能力。

英文摘要:

Text-to-Image (T2I) diffusion models have recently gained traction for their versatility and user-friendliness in 2D content generation and editing. However, training a diffusion model specifically for 3D scene editing is challenging due to the scarcity of large-scale datasets. Currently, editing 3D scenes necessitates either retraining the model to accommodate various 3D edits or developing specific methods tailored to each unique editing type. Moreover, state-of-the-art (SOTA) techniques require multiple synchronized edited images from the same scene to enable effective scene editing. Given the current limitations of T2I models, achieving consistent editing effects across multiple images remains difficult, leading to multi-view inconsistency in editing. This inconsistency undermines the performance of 3D scene editing when these images are utilized. In this study, we introduce a novel, training-free 3D scene editing technique called \textsc{Free-Editor}, which enables users to edit 3D scenes without the need for model retraining during the testing phase. Our method effectively addresses the issue of multi-view style inconsistency found in state-of-the-art (SOTA) methods through the implementation of a single-view editing scheme. Specifically, we demonstrate that editing a particular 3D scene can be achieved by modifying only a single view. To facilitate this, we present an Edit Transformer that ensures intra-view consistency and inter-view style transfer using self-view and cross-view attention mechanisms, respectively. By eliminating the need for model retraining and multi-view editing, our approach significantly reduces editing time and memory resource requirements, achieving runtimes approximately 20 times faster than SOTA methods. We have performed extensive experiments on various benchmark datasets, showcasing the diverse editing capabilities of our proposed technique.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑