FLOAT：面向音频驱动说话人肖像的生成式运动潜空间流匹配
FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait
AI总结：
FLOAT利用学习到的正交运动潜空间和流匹配生成模型，结合Transformer向量场预测器与逐帧条件机制，实现高效、时间一致且支持情感增强的音频驱动说话人肖像视频生成，在视觉质量、运动保真度和效率上优于现有方法。
摘要：
随着基于扩散的生成模型的快速发展，肖像图像动画已取得显著成果。然而，由于其迭代采样特性，它在时间一致的视频生成和快速采样方面仍面临挑战。本文提出了FLOAT，一种基于流匹配生成模型的音频驱动说话人肖像视频生成方法。我们没有采用基于像素的潜空间，而是利用学习到的正交运动潜空间，从而能够高效生成和编辑时间一致的运动。为实现这一目标，我们引入了一个基于Transformer的向量场预测器，并配备了有效的逐帧条件机制。此外，我们的方法支持语音驱动的情感增强，能够自然地融入富有表现力的动作。大量实验表明，我们的方法在视觉质量、运动保真度和效率方面均优于最先进的音频驱动说话人肖像方法。
英文摘要：
With the rapid advancement of diffusion-based generative models, portrait image animation has achieved remarkable results. However, it still faces challenges in temporally consistent video generation and fast sampling due to its iterative sampling nature. This paper presents FLOAT, an audio-driven talking portrait video generation method based on flow matching generative model. Instead of a pixel-based latent space, we take advantage of a learned orthogonal motion latent space, enabling efficient generation and editing of temporally consistent motion. To achieve this, we introduce a transformer-based vector field predictor with an effective frame-wise conditioning mechanism. Additionally, our method supports speech-driven emotion enhancement, enabling a natural incorporation of expressive motions. Extensive experiments demonstrate that our method outperforms state-of-the-art audio-driven talking portrait methods in terms of visual quality, motion fidelity, and efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
