SVP：风格增强型生动肖像会说话头像扩散模型
SVP: Style-Enhanced Vivid Portrait Talking Head Diffusion Model
AI总结：
针对音频驱动的会说话头像生成中扩散模型方法忽略视频内在风格导致内容缺乏多样性和生动性的问题，提出SVP框架，通过概率风格先验学习和SD模型微调，生成更优视频。
摘要：
会说话头像生成（THG）通常由音频驱动，是一项重要且具有挑战性的任务，在数字人、电影制作和虚拟现实等多个领域具有广泛的应用前景。虽然基于扩散模型的THG方法能生成高质量且稳定的内容，但它们往往忽略了包含视频中说话习惯、面部表情等个性化特征的内在风格。因此，生成的视频内容缺乏多样性和生动性，在现实场景中应用受限。为解决这些问题，我们提出名为风格增强型生动肖像（SVP）的新框架，充分利用THG中的风格相关信息。具体而言，我们首先引入新的概率风格先验学习，利用面部表情和音频嵌入将内在风格建模为高斯分布；该分布通过“定制”对比目标学习，有效捕捉每个视频中的动态风格信息。然后，我们微调预训练的Stable Diffusion（SD）模型，通过交叉注意力注入学习到的内在风格作为控制信号。实验表明，我们的模型生成的视频多样、生动且高质量，能灵活控制内在风格，性能优于现有最先进方法。
英文摘要：
Talking Head Generation (THG), typically driven by audio, is an important and challenging task with broad application prospects in various fields such as digital humans, film production, and virtual reality. While diffusion model-based THG methods present high quality and stable content generation, they often overlook the intrinsic style which encompasses personalized features such as speaking habits and facial expressions of a video. As consequence, the generated video content lacks diversity and vividness, thus being limited in real life scenarios. To address these issues, we propose a novel framework named Style-Enhanced Vivid Portrait (SVP) which fully leverages style-related information in THG. Specifically, we first introduce the novel probabilistic style prior learning to model the intrinsic style as a Gaussian distribution using facial expressions and audio embedding. The distribution is learned through the 'bespoked' contrastive objective, effectively capturing the dynamic style information in each video. Then we finetune a pretrained Stable Diffusion (SD) model to inject the learned intrinsic style as a controlling signal via cross attention. Experiments show that our model generates diverse, vivid, and high-quality videos with flexible control over intrinsic styles, outperforming existing state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
