arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.13552cs.CVcs.GR
收藏

DragScene:基于单视图拖拽指令的交互式3D场景编辑

DragScene: Interactive 3D Scene Editing with Single-view Drag Instructions

Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);College of Artificial Intelligence, Xi’an Jiaotong University(西安交通大学人工智能学院);School of Software, Beihang University(北京航空航天大学软件学院);The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Chenghao Gu, Zhenzhe Li, Zhengqi Zhang, Yunpeng Bai, Shuzhao Xie, Zhi Wang

AI总结:

提出DragScene框架,通过在参考视图进行2D拖拽编辑并利用点表示提取3D线索引导多视图优化,实现精确灵活且保持多视图一致性的3D场景编辑。

摘要:

3D编辑在基于各种指令编辑场景方面展现了卓越的能力。然而,现有方法难以实现直观的局部编辑,例如选择性地让花朵绽放。拖拽式编辑在通过直接操作而非模糊文本指令编辑图像方面展现了出色的能力。尽管如此,由于多视图不一致性,将基于拖拽的编辑扩展到3D场景面临巨大挑战。为此,我们提出DragScene,一个将拖拽式编辑与多种3D表示相集成的框架。首先,在参考视图上进行潜变量优化,以基于用户指令生成2D编辑。随后,使用基于点的表示从参考视图重建粗糙的3D线索,以捕捉编辑的几何细节。然后,将编辑视图的潜变量表示映射到这些3D线索,引导其他视图的潜变量优化。该过程确保编辑在多个视图间无缝传播,保持多视图一致性。最后,从编辑后的多视图图像重建目标3D场景。大量实验表明,DragScene促进了3D场景的精确灵活拖拽式编辑,支持跨多种3D表示的广泛适用性。

英文摘要:

3D editing has shown remarkable capability in editing scenes based on various instructions. However, existing methods struggle with achieving intuitive, localized editing, such as selectively making flowers blossom. Drag-style editing has shown exceptional capability to edit images with direct manipulation instead of ambiguous text commands. Nevertheless, extending drag-based editing to 3D scenes presents substantial challenges due to multi-view inconsistency. To this end, we introduce DragScene, a framework that integrates drag-style editing with diverse 3D representations. First, latent optimization is performed on a reference view to generate 2D edits based on user instructions. Subsequently, coarse 3D clues are reconstructed from the reference view using a point-based representation to capture the geometric details of the edits. The latent representation of the edited view is then mapped to these 3D clues, guiding the latent optimization of other views. This process ensures that edits are propagated seamlessly across multiple views, maintaining multi-view consistency. Finally, the target 3D scene is reconstructed from the edited multi-view images. Extensive experiments demonstrate that DragScene facilitates precise and flexible drag-style editing of 3D scenes, supporting broad applicability across diverse 3D representations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑