告诉你从所见中听到了什么——通过文本实现从视频到音频的生成
Tell What You Hear From What You See -- Video to Audio Generation Through Text
AI总结：
提出多模态生成框架VATT，通过文本提示和音频字幕实现可控的视频到音频生成，在客观和主观评估中均表现出色。
摘要：
视觉和音频场景的内容是多方面的，因此一个视频可以与各种音频配对，反之亦然。因此，在视频到音频生成任务中，引入控制生成音频的引导方法是势在必行的。虽然视频到音频生成是一项成熟的生成任务，但现有方法缺乏这种可控性。在这项工作中，我们提出了VATT，这是一个多模态生成框架，它以视频和可选的文本提示作为输入，并生成音频以及可选的音频文本描述。这样的框架有两个优势：i）视频到音频的生成过程可以通过文本进行细化和控制，这补充了视觉信息的上下文；ii）模型可以通过生成音频字幕来建议为视频生成什么音频。VATT由两个关键模块组成：VATT Converter，这是一个针对指令进行微调并包含一个投影层的LLM（大语言模型），该投影层将视频特征映射到LLM向量空间；以及VATT Audio，这是一个transformer，它使用迭代并行解码从视觉帧和可选的文本提示生成音频令牌。音频令牌通过预训练的神经编解码器转换为波形。实验表明，当VATT在客观指标上与现有的视频到音频生成方法进行比较时，在未提供音频字幕的情况下，它取得了具有竞争力的性能。当提供音频字幕作为提示时，VATT取得了更加精细的性能（最低的KLD分数为1.41）。此外，主观研究表明，VATT Audio被选为比现有方法生成的音频更受青睐的生成音频。VATT通过文本实现了可控的视频到音频生成，并通过音频字幕为视频建议文本提示，从而解锁了文本引导的视频到音频生成和视频到音频字幕生成等新应用。
英文摘要：
The content of visual and audio scenes is multi-faceted such that a video can be paired with various audio and vice-versa. Thereby, in video-to-audio generation task, it is imperative to introduce steering approaches for controlling the generated audio. While Video-to-Audio generation is a well-established generative task, existing methods lack such controllability. In this work, we propose VATT, a multi-modal generative framework that takes a video and an optional text prompt as input, and generates audio and optional textual description of the audio. Such a framework has two advantages: i) Video-to-Audio generation process can be refined and controlled via text which complements the context of visual information, and ii) The model can suggest what audio to generate for the video by generating audio captions. VATT consists of two key modules: VATT Converter, a LLM that is fine-tuned for instructions and includes a projection layer that maps video features to the LLM vector space; and VATT Audio, a transformer that generates audio tokens from visual frames and from optional text prompt using iterative parallel decoding. The audio tokens are converted to a waveform by pretrained neural codec. Experiments show that when VATT is compared to existing video-to-audio generation methods in objective metrics, it achieves competitive performance when the audio caption is not provided. When the audio caption is provided as a prompt, VATT achieves even more refined performance (lowest KLD score of 1.41). Furthermore, subjective studies show that VATT Audio has been chosen as preferred generated audio than audio generated by existing methods. VATT enables controllable video-to-audio generation through text as well as suggesting text prompts for videos through audio captions, unlocking novel applications such as text-guided video-to-audio generation and video-to-audio captioning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
