arXiv 2510.228102025-10-26cs.CV

MAGIC-Talk：具有可定制身份控制的运动感知音频驱动说话人脸生成

MAGIC-Talk: Motion-aware Audio-Driven Talking Face Generation with Customizable Identity Control

School of Computer Science and Electronic Engineering, University of Surrey（Surrey大学计算机科学与电子工程学院）；School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；Centre for Vision, Speech and Signal Processing, University of Surrey（Surrey大学视觉、语音和信号处理中心）

Fatemeh Nazarieh, Zhenhua Feng, Diptesh Kanojia, Muhammad Awais, Josef Kittler

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出MAGIC-Talk单次扩散框架，通过ReferenceNet保持身份和AnimateNet增强运动连贯性，结合渐进式潜在融合，实现长视频中稳定且可定制的音频驱动说话人脸生成。

摘要：

音频驱动的说话人脸生成在数字媒体和虚拟化身应用中引起了广泛关注。尽管近期方法提高了音频-嘴唇同步性能，但它们往往在时间一致性、身份保持和定制化方面存在困难，尤其是在长视频生成中。为解决这些问题，我们提出了MAGIC-Talk，一个基于单次扩散的框架，用于可定制且时间稳定的说话人脸生成。MAGIC-Talk由ReferenceNet和AnimateNet组成，其中ReferenceNet通过文本提示保持身份并实现细粒度面部编辑，而AnimateNet利用结构化运动先验增强运动连贯性。与先前需要多张参考图像或微调的方法不同，MAGIC-Talk仅从单张图像保持身份，同时确保帧间平滑过渡。此外，我们引入了一种渐进式潜在融合策略，通过减少运动不一致和闪烁来提高长视频质量。大量实验表明，MAGIC-Talk在视觉质量、身份保持和同步准确性方面优于最先进的方法，为说话人脸生成提供了稳健的解决方案。

英文摘要：

Audio-driven talking face generation has gained significant attention for applications in digital media and virtual avatars. While recent methods improve audio-lip synchronization, they often struggle with temporal consistency, identity preservation, and customization, especially in long video generation. To address these issues, we propose MAGIC-Talk, a one-shot diffusion-based framework for customizable and temporally stable talking face generation. MAGIC-Talk consists of ReferenceNet, which preserves identity and enables fine-grained facial editing via text prompts, and AnimateNet, which enhances motion coherence using structured motion priors. Unlike previous methods requiring multiple reference images or fine-tuning, MAGIC-Talk maintains identity from a single image while ensuring smooth transitions across frames. Additionally, a progressive latent fusion strategy is introduced to improve long-form video quality by reducing motion inconsistencies and flickering. Extensive experiments demonstrate that MAGIC-Talk outperforms state-of-the-art methods in visual quality, identity preservation, and synchronization accuracy, offering a robust solution for talking face generation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。