VLOGGER：用于具身化身合成的多模态扩散模型
VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis
AI总结：
VLOGGER提出一种基于多模态扩散的音频驱动视频生成方法，从单张图像合成高质量人物视频，无需逐人训练，并在三个基准上超越现有方法，同时引入大规模数据集MENTOR以提升多样性。
摘要：
我们提出了VLOGGER，一种从单张人物输入图像生成音频驱动的人类视频的方法，该方法建立在近期生成扩散模型成功的基础上。我们的方法包括：1）一个随机的人体到3D运动扩散模型，以及2）一种新颖的基于扩散的架构，该架构为文本到图像模型增加了空间和时间控制。这支持生成可变长度的高质量视频，并且可以通过人类面部和身体的高级表示轻松控制。与以往工作相比，我们的方法不需要针对每个人进行训练，不依赖人脸检测和裁剪，生成完整图像（不仅仅是面部或嘴唇），并考虑了广泛场景（例如可见躯干或多样主体身份），这些对于正确合成交流中的人类至关重要。我们还策划了MENTOR，一个带有3D姿态和表情注释的新的多样化数据集，比以往数据集大一个数量级（800,000个身份），并包含动态手势，我们在此数据集上训练并消融了我们的主要技术贡献。VLOGGER在三个公开基准上超越了最先进方法，在图像质量、身份保持和时间一致性方面表现出色，同时还能生成上半身手势。我们分析了VLOGGER在多个多样性指标上的表现，表明我们的架构选择和使用MENTOR有助于在规模上训练公平且无偏的模型。最后，我们展示了在视频编辑和个性化方面的应用。
英文摘要：
We propose VLOGGER, a method for audio-driven human video generation from a single input image of a person, which builds on the success of recent generative diffusion models. Our method consists of 1) a stochastic human-to-3d-motion diffusion model, and 2) a novel diffusion-based architecture that augments text-to-image models with both spatial and temporal controls. This supports the generation of high quality video of variable length, easily controllable through high-level representations of human faces and bodies. In contrast to previous work, our method does not require training for each person, does not rely on face detection and cropping, generates the complete image (not just the face or the lips), and considers a broad spectrum of scenarios (e.g. visible torso or diverse subject identities) that are critical to correctly synthesize humans who communicate. We also curate MENTOR, a new and diverse dataset with 3d pose and expression annotations, one order of magnitude larger than previous ones (800,000 identities) and with dynamic gestures, on which we train and ablate our main technical contributions. VLOGGER outperforms state-of-the-art methods in three public benchmarks, considering image quality, identity preservation and temporal consistency while also generating upper-body gestures. We analyze the performance of VLOGGER with respect to multiple diversity metrics, showing that our architectural choices and the use of MENTOR benefit training a fair and unbiased model at scale. Finally we show applications in video editing and personalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
