arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.00320cs.SDcs.CVcs.MMeess.AS
收藏

Frieren:基于整流流匹配的高效视频到音频生成网络

Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching

Zhejiang University(浙江大学)

Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, Zhou Zhao

AI总结:

本文提出基于整流流匹配的V2A模型Frieren,通过非自回归向量场估计、通道级跨模态融合及reflow与一步蒸馏,实现高质量、高同步且高效的视频到音频生成。

摘要:

视频到音频(V2A)生成旨在从无声视频合成与内容匹配的音频,而构建兼具高生成质量、高效率和视觉-音频时间同步性的V2A模型仍具挑战性。我们提出Frieren,这是一种基于整流流匹配的V2A模型。Frieren沿直线路径回归从噪声到频谱图潜变量的条件传输向量场,并通过求解ODE进行采样,在音频质量方面优于自回归模型和基于分数的模型。通过采用基于前馈transformer的非自回归向量场估计器,以及具有强时间对齐能力的通道级跨模态特征融合,我们的模型生成的音频与输入视频高度同步。此外,通过reflow和带引导向量场的一步蒸馏,我们的模型能够在少数几个、甚至仅一个采样步骤内生成质量尚可的音频。实验表明,Frieren在VGGSound上的生成质量和时间对齐均达到最先进性能,对齐准确率达到97.22%,并且inception score相比强大的基于扩散的基线提升6.2%。音频样本可在http://frieren-v2a.github.io获取。

英文摘要:

Video-to-audio (V2A) generation aims to synthesize content-matching audio from silent video, and it remains challenging to build V2A models with high generation quality, efficiency, and visual-audio temporal synchrony. We propose Frieren, a V2A model based on rectified flow matching. Frieren regresses the conditional transport vector field from noise to spectrogram latent with straight paths and conducts sampling by solving ODE, outperforming autoregressive and score-based models in terms of audio quality. By employing a non-autoregressive vector field estimator based on a feed-forward transformer and channel-level cross-modal feature fusion with strong temporal alignment, our model generates audio that is highly synchronized with the input video. Furthermore, through reflow and one-step distillation with guided vector field, our model can generate decent audio in a few, or even only one sampling step. Experiments indicate that Frieren achieves state-of-the-art performance in both generation quality and temporal alignment on VGGSound, with alignment accuracy reaching 97.22%, and 6.2% improvement in inception score over the strong diffusion-based baseline. Audio samples are available at http://frieren-v2a.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑