Chat-Edit-3D: 通过文本提示进行交互式3D场景编辑
Chat-Edit-3D: Interactive 3D Scene Editing via Text Prompts
AI总结：
针对现有3D场景编辑方案输入模式固定、模型集成复杂的问题，提出以大语言模型为核心的对话式编辑方法CE3D，通过Hash-Atlas实现2D编辑与3D重建解耦，可灵活集成多视觉模型，具备强场景理解与多轮对话能力。
摘要：
基于视觉-语言预训练模型的图像内容操控相关最新研究，已被有效拓展至文本驱动的3D场景编辑领域。但现有的3D场景编辑方案仍存在一定缺陷，阻碍了其进一步的交互式设计：这类方案通常遵循固定的输入模式，限制了用户文本输入的灵活性；此外，其编辑能力受限于单个或少数几个2D视觉模型，还需要复杂的流水线设计才能将这些模型集成到3D重建流程中。针对上述问题，我们提出了一种基于对话的3D场景编辑方法，名为CE3D。该方法以大语言模型为核心，支持用户输入任意文本并解析其意图，随后自主调用对应的视觉专家模型。此外，我们设计了一种利用Hash-Atlas（哈希图集）表示3D场景视图的方案，将3D场景的编辑转移到2D图集图像上。这一设计实现了2D编辑与3D重建流程的完全解耦，使得CE3D能够灵活集成大量现有的2D或3D视觉模型，无需复杂的融合设计。实验结果表明，CE3D可有效集成多个视觉模型以实现多样化的编辑视觉效果，具备较强的场景理解能力和多轮对话能力。代码可在https://sk-fun.fun/CE3D获取。
英文摘要：
Recent work on image content manipulation based on vision-language pre-training models has been effectively extended to text-driven 3D scene editing. However, existing schemes for 3D scene editing still exhibit certain shortcomings, hindering their further interactive design. Such schemes typically adhere to fixed input patterns, limiting users' flexibility in text input. Moreover, their editing capabilities are constrained by a single or a few 2D visual models and require intricate pipeline design to integrate these models into 3D reconstruction processes. To address the aforementioned issues, we propose a dialogue-based 3D scene editing approach, termed CE3D, which is centered around a large language model that allows for arbitrary textual input from users and interprets their intentions, subsequently facilitating the autonomous invocation of the corresponding visual expert models. Furthermore, we design a scheme utilizing Hash-Atlas to represent 3D scene views, which transfers the editing of 3D scenes onto 2D atlas images. This design achieves complete decoupling between the 2D editing and 3D reconstruction processes, enabling CE3D to flexibly integrate a wide range of existing 2D or 3D visual models without necessitating intricate fusion designs. Experimental results demonstrate that CE3D effectively integrates multiple visual models to achieve diverse editing visual effects, possessing strong scene comprehension and multi-round dialog capabilities. The code is available at https://sk-fun.fun/CE3D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
