Instruct 4D-to-4D：使用2D扩散将4D场景作为伪3D场景进行编辑
Instruct 4D-to-4D: Editing 4D Scenes as Pseudo-3D Scenes Using 2D Diffusion
AI总结：
提出 Instruct 4D-to-4D，将4D场景视为伪3D场景，结合增强IP2P、光流传播与深度投影，实现时空一致的高质量指令引导动态场景编辑。
摘要：
本文提出 Instruct 4D-to-4D，使2D扩散模型具备4D感知能力和时空一致性，以生成高质量的指令引导动态场景编辑结果。2D扩散模型在动态场景编辑中的传统应用往往导致不一致，这主要源于其固有的逐帧编辑方式。为应对将指令引导编辑扩展到4D的复杂性，我们的核心洞见是把4D场景视为伪3D场景，并将其解耦为两个子问题：在视频编辑中实现时间一致性，以及将这些编辑应用于伪3D场景。据此，我们首先通过锚点感知注意力模块增强 Instruct-Pix2Pix（IP2P）模型，以实现批处理和一致编辑。此外，我们以滑动窗口方式集成光流引导的外观传播，用于更精确的帧间编辑，并引入基于深度的投影来管理伪3D场景的大量数据，随后进行迭代编辑以达到收敛。我们在多种场景和编辑指令上广泛评估了所提方法，证明其实现了空间和时间一致的编辑结果，并且相较现有技术显著增强了细节和锐度。值得注意的是，Instruct 4D-to-4D 具有通用性，适用于单目场景以及具有挑战性的多相机场景。代码和更多结果可在 immortalco.github.io/Instruct-4D-to-4D 获取。
英文摘要：
This paper proposes Instruct 4D-to-4D that achieves 4D awareness and spatial-temporal consistency for 2D diffusion models to generate high-quality instruction-guided dynamic scene editing results. Traditional applications of 2D diffusion models in dynamic scene editing often result in inconsistency, primarily due to their inherent frame-by-frame editing methodology. Addressing the complexities of extending instruction-guided editing to 4D, our key insight is to treat a 4D scene as a pseudo-3D scene, decoupled into two sub-problems: achieving temporal consistency in video editing and applying these edits to the pseudo-3D scene. Following this, we first enhance the Instruct-Pix2Pix (IP2P) model with an anchor-aware attention module for batch processing and consistent editing. Additionally, we integrate optical flow-guided appearance propagation in a sliding window fashion for more precise frame-to-frame editing and incorporate depth-based projection to manage the extensive data of pseudo-3D scenes, followed by iterative editing to achieve convergence. We extensively evaluate our approach in various scenes and editing instructions, and demonstrate that it achieves spatially and temporally consistent editing results, with significantly enhanced detail and sharpness over the prior art. Notably, Instruct 4D-to-4D is general and applicable to both monocular and challenging multi-camera scenes. Code and more results are available at immortalco.github.io/Instruct-4D-to-4D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
