StyleTalker：基于风格的一次性音频驱动说话头视频生成
StyleTalker: One-shot Style-based Audio-driven Talking Head Video Generation
AI总结：
提出StyleTalker，一种基于单张参考图像的音频驱动说话头视频生成模型，通过对比唇形同步判别器、条件序列变分自编码器和归一化流增强的自回归先验，实现精确唇形同步、逼真头部姿态和眨眼，支持运动可控和纯音频驱动两种生成方式，性能优于现有方法。
摘要：
我们提出了StyleTalker，一种新颖的音频驱动的说话头生成模型，它可以从单张参考图像合成一个说话人的视频，并具有精确的音频同步唇形、逼真的头部姿态和眨眼。具体来说，通过利用预训练的图像生成器和图像编码器，我们估计了忠实反映给定音频的说话头视频的潜在编码。这得益于几个新设计的组件：1）一个对比唇形同步判别器，用于精确的唇形同步；2）一个条件序列变分自编码器，学习与唇部运动解耦的潜在运动空间，从而我们可以在保持身份的同时独立操控运动和唇部运动；3）一个用归一化流增强的自回归先验，用于学习复杂的音频到运动的多模态潜在空间。配备这些组件，StyleTalker不仅能以运动可控的方式生成说话头视频（当给定另一个运动源视频时），还能完全以音频驱动的方式，通过从输入音频推断逼真的运动来生成视频。通过大量实验和用户研究，我们表明我们的模型能够合成具有令人印象深刻的感知质量的说话头视频，这些视频与输入音频精确唇形同步，大大优于最先进的基线方法。
英文摘要：
We propose StyleTalker, a novel audio-driven talking head generation model that can synthesize a video of a talking person from a single reference image with accurately audio-synced lip shapes, realistic head poses, and eye blinks. Specifically, by leveraging a pretrained image generator and an image encoder, we estimate the latent codes of the talking head video that faithfully reflects the given audio. This is made possible with several newly devised components: 1) A contrastive lip-sync discriminator for accurate lip synchronization, 2) A conditional sequential variational autoencoder that learns the latent motion space disentangled from the lip movements, such that we can independently manipulate the motions and lip movements while preserving the identity. 3) An auto-regressive prior augmented with normalizing flow to learn a complex audio-to-motion multi-modal latent space. Equipped with these components, StyleTalker can generate talking head videos not only in a motion-controllable way when another motion source video is given but also in a completely audio-driven manner by inferring realistic motions from the input audio. Through extensive experiments and user studies, we show that our model is able to synthesize talking head videos with impressive perceptual quality which are accurately lip-synced with the input audios, largely outperforming state-of-the-art baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
