面向人像图像动效的深度与外观联合学习
Joint Learning of Depth and Appearance for Portrait Image Animation
AI总结：
针对现有2D人像动效扩散方法多仅生成RGB图像、缺乏一致3D输出的问题，提出联合学习视觉外观与深度的扩散生成框架，含参考网络与通道扩展骨干，可适配多种带一致3D输出的下游人像应用。
摘要：
近年来，2D人像动效领域取得了显著进展。大量研究利用大型生成扩散模型中嵌入的先验知识来提升高质量图像操控效果。然而，多数方法仅聚焦于生成RGB图像作为输出，一致的视觉输出与3D输出的协同生成仍未得到充分探索。本研究提出在基于扩散的人像图像生成器中同时联合学习视觉外观与深度。该方法采用端到端扩散范式，并引入一种适用于学习该条件联合分布的新架构，由参考网络和通道扩展的扩散骨干网络组成。训练完成后，本框架可高效适配多种下游应用，例如人脸深度到图像生成、图像到深度生成、人像重打光，以及具备一致3D输出的音频驱动数字人头部动效。
英文摘要：
2D portrait animation has experienced significant advancements in recent years. Much research has utilized the prior knowledge embedded in large generative diffusion models to enhance high-quality image manipulation. However, most methods only focus on generating RGB images as output, and the co-generation of consistent visual plus 3D output remains largely under-explored. In our work, we propose to jointly learn the visual appearance and depth simultaneously in a diffusion-based portrait image generator. Our method embraces the end-to-end diffusion paradigm and introduces a new architecture suitable for learning this conditional joint distribution, consisting of a reference network and a channel-expanded diffusion backbone. Once trained, our framework can be efficiently adapted to various downstream applications, such as facial depth-to-image and image-to-depth generation, portrait relighting, and audio-driven talking head animation with consistent 3D output.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
