Video-to-audio (V2A) generation aims to synthesize content-matching audio from silent video, and it remains challenging to build V2A models with high generation quality, efficiency, and visual-audio temporal synchrony. We propose Frieren, a V2A model based on rectified flow matching. Frieren regresses the conditional transport vector field from noise to spectrogram latent with straight paths and conducts sampling by solving ODE, outperforming autoregressive and score-based models in terms of audio quality. By employing a non-autoregressive vector field estimator based on a feed-forward transformer and channel-level cross-modal feature fusion with strong temporal alignment, our model generates audio that is highly synchronized with the input video. Furthermore, through reflow and one-step distillation with guided vector field, our model can generate decent audio in a few, or even only one sampling step. Experiments indicate that Frieren achieves state-of-the-art performance in both generation quality and temporal alignment on VGGSound, with alignment accuracy reaching 97.22%, and 6.2% improvement in inception score over the strong diffusion-based baseline. Audio samples are available at http://frieren-v2a.github.io.
AudioTurbo: Fast Text-to-Audio Generation with Rectified Diffusion
University of SurreyUK(英国萨里大学);Institute of Acoustics, Chinese Academy of SciencesChina(中国科学院声学研究所);Centre for Vision, Speech and Signal Processing (CVSSP) Laboratory of Noise and Audio Research(噪声与音频研究实验室)
FlashAudio: Rectified Flows for Fast and High-Fidelity Text-to-Audio Generation
Hong Kong University of Science and Technology(香港科技大学);Tongyi Lab, Alibaba Group(阿里云实验室);Zhejiang University(浙江大学);Fundamental AI Research (FAIR), Meta(Meta基础人工智能研究(FAIR))
RFM-Editing: Rectified Flow Matching for Text-guided Audio Editing
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, United Kingdom(视觉、语音和信号处理中心(CVSSP),英国萨里大学);College of Information and Electrical Engineering, China Agricultural University, China(信息与电气工程学院,中国农业大学);Fisheries College, Ocean University of China, China(海洋大学渔业学院,中国)
RFM-Editing 2: Text-Guided Audio Editing with Rectified Flow Matching and Coarse-to-Fine Diffusion Transformers
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey(萨里大学视觉、语音与信号处理中心);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);Fisheries College, Ocean University of China(中国海洋大学水产学院);College of Information and Electrical Engineering, China Agricultural University(中国农业大学信息与电气工程学院)
FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation
School of Computer Science and Electronic Engineering, University of Surrey(萨里大学计算机科学与电子工程学院);Meta Superintelligence Labs(元宇宙超级智能实验室);Department of Informatics, King’s College London(伦敦国王学院信息学系)
CogSR: Semantic-Aware Speech Super-Resolution via Chain-of-Thought Guided Flow Matching
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, Wuhan, China(多媒体软件国家工程研究中心、计算机科学学院、武汉大学、武汉);School of Artificial Intelligence, Jianghan University, Wuhan, China(人工智能学院、江汉大学、武汉);Xiaomi Corporation, Beijing, China(小米公司、北京)