通过语言桥接高质量音频与视频，实现基于视觉查询的音效检索
Bridging High-Quality Audio and Video via Language for Sound Effects Retrieval from Visual Queries
AI总结：
针对视频帧检索高质量音效缺乏高质量训练数据的问题，本文提出用LLM与视觉-语言模型自动构建HQ音视频对，并训练对比学习检索系统，显著提升跨域泛化与用户偏好。
摘要：
为视频中的时刻找到合适的音效（SFX）是一项困难且耗时的任务，并且高度依赖文本元数据的质量和完整性。直接使用视频帧作为查询来检索高质量（HQ）SFX是一种有吸引力的替代方案，它消除了对文本元数据的依赖，并为非专业人士提供了较低的入门门槛。由于缺乏高质量音视频训练数据，以往关于音视频检索的工作依赖质量参差不齐的YouTube（野外）视频进行训练，其中音频通常有噪声，视频也属于业余质量。因此，尚不清楚这些系统能否泛化到将高质量音频与制作级视频匹配的任务。为解决这一问题，我们提出了一个多模态框架，用于在给定视频帧的情况下推荐高质量SFX：（1）利用大型语言模型和基础视觉-语言模型桥接高质量音频与视频以创建音视频对，从而形成一个高度可扩展的自动化音视频数据整理流程；（2）使用预训练的音频和视觉编码器训练一个基于对比学习的检索系统。我们表明，使用我们的自动化数据整理流程训练的系统，在面向视频的高质量SFX检索任务上显著优于在野外数据上训练的基线。此外，尽管基线无法泛化到该任务，我们的系统却能从干净数据良好泛化到野外数据；尽管仅在高质量音视频对上训练，它仍在一个YouTube视频数据集上优于基线。一项用户研究证实，对于高质量数据和野外数据，人们在67%的情况下更偏好我们系统检索到的SFX，而非基线结果。最后，我们给出消融实验，以确定模型和数据流程设计选择对下游检索性能的影响。请访问我们的项目网站，收听并查看我们的SFX检索结果。
英文摘要：
Finding the right sound effects (SFX) to match moments in a video is a difficult and time-consuming task, and relies heavily on the quality and completeness of text metadata. Retrieving high-quality (HQ) SFX using a video frame directly as the query is an attractive alternative, removing the reliance on text metadata and providing a low barrier to entry for non-experts. Due to the lack of HQ audio-visual training data, previous work on audio-visual retrieval relies on YouTube (in-the-wild) videos of varied quality for training, where the audio is often noisy and the video of amateur quality. As such it is unclear whether these systems would generalize to the task of matching HQ audio to production-quality video. To address this, we propose a multimodal framework for recommending HQ SFX given a video frame by (1) leveraging large language models and foundational vision-language models to bridge HQ audio and video to create audio-visual pairs, resulting in a highly scalable automatic audio-visual data curation pipeline; and (2) using pre-trained audio and visual encoders to train a contrastive learning-based retrieval system. We show that our system, trained using our automatic data curation pipeline, significantly outperforms baselines trained on in-the-wild data on the task of HQ SFX retrieval for video. Furthermore, while the baselines fail to generalize to this task, our system generalizes well from clean to in-the-wild data, outperforming the baselines on a dataset of YouTube videos despite only being trained on the HQ audio-visual pairs. A user study confirms that people prefer SFX retrieved by our system over the baseline 67% of the time both for HQ and in-the-wild data. Finally, we present ablations to determine the impact of model and data pipeline design choices on downstream retrieval performance. Please visit our project website to listen to and view our SFX retrieval results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
