FREAK：频率调制的高保真、实时音频驱动说话肖像合成
FREAK: Frequency-modulated High-fidelity and Real-time Audio-driven Talking Portrait Synthesis
AI总结：
提出频率调制框架FREAK，从频域建模说话肖像，引入VEFM和AVFM模块，联合像素域与频域优化，实现高保真、实时且唇部同步的合成。
摘要：
在音频驱动的说话肖像合成中，实现高保真的唇语同步仍然具有挑战性。虽然多阶段流水线或扩散模型能产生高质量结果，但它们遭受高计算成本的困扰。一些方法在低资源下对特定个体表现良好，但仍表现出不匹配的唇部运动。上述方法都是在像素域中建模的。我们观察到，合成说话视频与自然视频在频域中存在明显差异。目前，没有关于说话肖像合成的研究考虑这一方面。为了解决这个问题，我们提出了一种频率调制的高保真、实时音频驱动说话肖像合成框架，命名为FREAK，它从频域角度对说话肖像进行建模，增强合成肖像的保真度和自然度。FREAK引入了两个新颖的基于频率的模块：1）视觉编码频率调制器（VEFM），在频域中耦合多尺度视觉特征，更好地保留视觉频率信息，减少合成帧与自然帧之间频谱的差距；2）音频视觉频率调制器（AVFM），帮助模型在频域中学习说话模式，提高音视频同步性。此外，我们在像素域和频域中联合优化模型。而且，FREAK支持在单次和视频配音设置之间无缝切换，提供增强的灵活性。由于其优越的性能，它可以同时支持高分辨率视频结果和实时推理。大量实验表明，我们的方法能够实时合成具有精细面部纹理和精确唇部同步的高保真说话肖像，优于最先进的方法。
英文摘要：
Achieving high-fidelity lip-speech synchronization in audio-driven talking portrait synthesis remains challenging. While multi-stage pipelines or diffusion models yield high-quality results, they suffer from high computational costs. Some approaches perform well on specific individuals with low resources, yet still exhibit mismatched lip movements. The aforementioned methods are modeled in the pixel domain. We observed that there are noticeable discrepancies in the frequency domain between the synthesized talking videos and natural videos. Currently, no research on talking portrait synthesis has considered this aspect. To address this, we propose a FREquency-modulated, high-fidelity, and real-time Audio-driven talKing portrait synthesis framework, named FREAK, which models talking portraits from the frequency domain perspective, enhancing the fidelity and naturalness of the synthesized portraits. FREAK introduces two novel frequency-based modules: 1) the Visual Encoding Frequency Modulator (VEFM) to couple multi-scale visual features in the frequency domain, better preserving visual frequency information and reducing the gap in the frequency spectrum between synthesized and natural frames. and 2) the Audio Visual Frequency Modulator (AVFM) to help the model learn the talking pattern in the frequency domain and improve audio-visual synchronization. Additionally, we optimize the model in both pixel domain and frequency domain jointly. Furthermore, FREAK supports seamless switching between one-shot and video dubbing settings, offering enhanced flexibility. Due to its superior performance, it can simultaneously support high-resolution video results and real-time inference. Extensive experiments demonstrate that our method synthesizes high-fidelity talking portraits with detailed facial textures and precise lip synchronization in real-time, outperforming state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
