FoleyCrafter：用逼真且同步的声音让无声视频栩栩如生
FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds
AI总结：
本文提出FoleyCrafter框架，通过语义适配器和时间控制器解决神经拟音中音效质量与音视频同步难题，并支持文本提示实现可控多样的视频到音频生成。
摘要：
我们研究神经拟音（Neural Foley），即自动生成与视频同步的高质量音效，从而实现沉浸式视听体验。尽管其应用范围广泛，现有方法在同时合成高质量且与视频对齐（即语义相关且时间同步）的声音方面仍存在局限。为克服这些局限，我们提出FoleyCrafter，一个利用预训练文本到音频模型来保证高质量音频生成的新框架。FoleyCrafter包含两个关键组件：用于语义对齐的语义适配器和用于精确音视频同步的时间控制器。语义适配器利用并行交叉注意力层，以视频特征为条件进行音频生成，产生与视觉内容语义相关的逼真音效。同时，时间控制器结合了起始检测器和基于时间戳的适配器，以实现精确的音视频对齐。FoleyCrafter的一个显著优势是其与文本提示的兼容性，使得能够根据用户意图使用文本描述来实现可控且多样化的视频到音频生成。我们在标准基准上进行了大量的定量和定性实验，以验证FoleyCrafter的有效性。模型和代码可在https://github.com/open-mmlab/FoleyCrafter获取。
英文摘要：
We study Neural Foley, the automatic generation of high-quality sound effects synchronizing with videos, enabling an immersive audio-visual experience. Despite its wide range of applications, existing approaches encounter limitations when it comes to simultaneously synthesizing high-quality and video-aligned (i.e.,, semantic relevant and temporal synchronized) sounds. To overcome these limitations, we propose FoleyCrafter, a novel framework that leverages a pre-trained text-to-audio model to ensure high-quality audio generation. FoleyCrafter comprises two key components: the semantic adapter for semantic alignment and the temporal controller for precise audio-video synchronization. The semantic adapter utilizes parallel cross-attention layers to condition audio generation on video features, producing realistic sound effects that are semantically relevant to the visual content. Meanwhile, the temporal controller incorporates an onset detector and a timestampbased adapter to achieve precise audio-video alignment. One notable advantage of FoleyCrafter is its compatibility with text prompts, enabling the use of text descriptions to achieve controllable and diverse video-to-audio generation according to user intents. We conduct extensive quantitative and qualitative experiments on standard benchmarks to verify the effectiveness of FoleyCrafter. Models and codes are available at https://github.com/open-mmlab/FoleyCrafter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
