SINGER：基于多尺度频谱扩散模型的生动音频驱动歌唱视频生成
SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model
AI总结：
针对说话与歌唱在音频频谱及行为表达上的差异，本文提出融合多尺度频谱与频谱滤波模块的扩散模型 SINGER，并构建野外歌唱音视频数据集，实现更生动的歌唱视频生成。
摘要：
近期生成模型的进展显著提升了说话人脸视频生成，但歌唱视频生成仍未得到充分研究。人类说话与歌唱之间的差异限制了现有说话人脸视频生成模型应用于歌唱时的性能。说话与歌唱之间的根本差异——具体体现在音频特征和行为表达方面——限制了现有模型的有效性。我们观察到，歌唱音频与说话音频之间的差异表现在频率和振幅上。为解决这一问题，我们设计了一个多尺度频谱模块，以帮助模型在频谱域中学习歌唱模式。此外，我们开发了一个频谱滤波模块，帮助模型学习与歌唱音频相关的人类行为。这两个模块被集成到扩散模型中，以增强歌唱视频生成性能，由此形成我们提出的模型 SINGER。此外，缺乏高质量真实歌唱人脸视频阻碍了歌唱视频生成领域的发展。为弥补这一空白，我们收集了一个野外音视频歌唱数据集，以促进该领域的研究。实验表明，SINGER 能够生成生动的歌唱视频，并在客观和主观评估中均优于最先进方法。
英文摘要：
Recent advancements in generative models have significantly enhanced talking face video generation, yet singing video generation remains underexplored. The differences between human talking and singing limit the performance of existing talking face video generation models when applied to singing. The fundamental differences between talking and singing-specifically in audio characteristics and behavioral expressions-limit the effectiveness of existing models. We observe that the differences between singing and talking audios manifest in terms of frequency and amplitude. To address this, we have designed a multi-scale spectral module to help the model learn singing patterns in the spectral domain. Additionally, we develop a spectral-filtering module that aids the model in learning the human behaviors associated with singing audio. These two modules are integrated into the diffusion model to enhance singing video generation performance, resulting in our proposed model, SINGER. Furthermore, the lack of high-quality real-world singing face videos has hindered the development of the singing video generation community. To address this gap, we have collected an in-the-wild audio-visual singing dataset to facilitate research in this area. Our experiments demonstrate that SINGER is capable of generating vivid singing videos and outperforms state-of-the-art methods in both objective and subjective evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
