InstructPix2NeRF：基于单张图像的指令式3D人像编辑
InstructPix2NeRF: Instructed 3D Portrait Editing from a Single Image
AI总结：
针对现有3D感知人像编辑依赖逐提示优化、端到端开放世界编辑探索不足的问题，提出基于扩散模型的端到端框架InstructPix2NeRF，通过条件潜在3D扩散、token位置随机化和身份一致性模块，实现单图指令式多语义3D人像编辑，效果优于强基线。
摘要：
随着神经辐射场（Neural Radiance Field, NeRF）在3D感知人像编辑领域取得成功，已有诸多工作在编辑质量和3D一致性方面均取得了可观成果。但这类方法在以自然语言作为编辑指令时，严重依赖逐提示优化过程。由于缺乏带标注的人脸3D数据集与高效架构，面向开放世界人像、采用端到端模式的人类指令式3D感知编辑领域仍未得到充分探索。\n为解决这一问题，我们提出了一种基于扩散模型的端到端框架——InstructPix2NeRF，可基于单张开放世界图像与人类指令，实现指令式3D感知人像编辑。该框架的核心是条件潜在3D扩散过程，它通过三元组数据学习配对图像的差异与指令之间的关联，将2D编辑提升至3D空间。借助我们提出的token位置随机化策略，我们甚至能在单次前向传播中实现多语义编辑，同时很好地保留人像身份。此外，我们还进一步提出了身份一致性模块，将提取的身份信号直接调制到扩散过程中，提升了多视角3D身份一致性。\n大量实验验证了我们方法的有效性，且在定量和定性层面均展现出相较于强基线方法的优越性。源代码与预训练模型可在我们的项目页面获取：https://mybabyyh.github.io/InstructPix2NeRF。
英文摘要：
With the success of Neural Radiance Field (NeRF) in 3D-aware portrait editing, a variety of works have achieved promising results regarding both quality and 3D consistency. However, these methods heavily rely on per-prompt optimization when handling natural language as editing instructions. Due to the lack of labeled human face 3D datasets and effective architectures, the area of human-instructed 3D-aware editing for open-world portraits in an end-to-end manner remains under-explored. To solve this problem, we propose an end-to-end diffusion-based framework termed InstructPix2NeRF, which enables instructed 3D-aware portrait editing from a single open-world image with human instructions. At its core lies a conditional latent 3D diffusion process that lifts 2D editing to 3D space by learning the correlation between the paired images' difference and the instructions via triplet data. With the help of our proposed token position randomization strategy, we could even achieve multi-semantic editing through one single pass with the portrait identity well-preserved. Besides, we further propose an identity consistency module that directly modulates the extracted identity signals into our diffusion process, which increases the multi-view 3D identity consistency. Extensive experiments verify the effectiveness of our method and show its superiority against strong baselines quantitatively and qualitatively. Source code and pre-trained models can be found on our project page: \url{https://mybabyyh.github.io/InstructPix2NeRF}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
