基于多基础模型映射器的高效视频转音频生成
Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper
AI总结：
针对视频转音频生成从头训练模型资源消耗大的问题，提出融合双视觉编码器特征、用GPT-2替代线性映射的MFM-Mapper，仅用16%训练规模就实现更优的语义与时间一致性，训练效率大幅提升。
摘要：
当前的视频转音频（Video-to-Audio, V2A）生成依赖从视频中提取语义与时间特征，以此为生成模型提供条件。从头训练这类模型资源消耗极大，因此利用具备跨模态知识迁移与泛化能力的基础模型（Foundation Models, FMs）逐渐成为研究热点。已有研究探索了通过微调轻量映射网络，将预训练视觉编码器与文本转音频生成模型连接以实现V2A的方案。受此启发，我们提出多基础模型映射器（Multiple Foundation Model Mapper, MFM-Mapper）。与此前的映射方法相比，MFM-Mapper通过融合双视觉编码器的特征，获得了更丰富的语义与时间信息。此外，我们用GPT-2替换线性映射器，将跨模态特征映射与自回归翻译任务类比，提升了特征对齐效果。MFM-Mapper训练效率显著：仅需此前基于映射器的工作16%的训练规模，就能在语义一致性与时间一致性上取得更优表现，且性能可与训练规模大得多的模型相媲美。
英文摘要：
Recent Video-to-Audio (V2A) generation relies on extracting semantic and temporal features from video to condition generative models. Training these models from scratch is resource intensive. Consequently, leveraging foundation models (FMs) has gained traction due to their cross-modal knowledge transfer and generalization capabilities. One prior work has explored fine-tuning a lightweight mapper network to connect a pre-trained visual encoder with a text-to-audio generation model for V2A. Inspired by this, we introduce the Multiple Foundation Model Mapper (MFM-Mapper). Compared to the previous mapper approach, MFM-Mapper benefits from richer semantic and temporal information by fusing features from dual visual encoders. Furthermore, by replacing a linear mapper with GPT-2, MFM-Mapper improves feature alignment, drawing parallels between cross-modal features mapping and autoregressive translation tasks. Our MFM-Mapper exhibits remarkable training efficiency. It achieves better performance in semantic and temporal consistency with fewer training consuming, requiring only 16\% of the training scale compared to previous mapper-based work, yet achieves competitive performance with models trained on a much larger scale.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
