SonicVisionLM:使用视觉语言模型播放声音
SonicVisionLM: Playing Sound with Vision Language Models
AI总结:
本文提出 SonicVisionLM 框架,利用 VLM 将无声视频中的事件转为文本声音描述,再经扩散模型和时间受控音频适配器生成音效,提升音画同步与对齐效果。
摘要:
为无声视频生成声音的任务正受到越来越多的关注,这主要是因为它在简化视频后期制作方面具有实用性。然而,现有的视频声音生成方法试图直接从视觉表征创建声音,由于视觉表征与音频表征之间难以对齐,这可能颇具挑战。本文提出 SonicVisionLM,这是一个旨在利用视觉语言模型(VLM)生成多种音效的新颖框架。我们并非直接从视频生成音频,而是借助强大 VLM 的能力。给定一段无声视频时,我们的方法首先使用 VLM 识别视频中的事件,以提出与视频内容相匹配的可能声音。这种方法上的转变,将困难的图像与音频对齐任务,转化为研究更充分的子问题:通过流行的扩散模型实现图像到文本以及文本到音频的对齐。为提升使用 LLM 进行音频推荐的质量,我们收集了一个将文本描述映射到特定音效的大规模数据集,并开发了一个时间受控音频适配器。我们的方法在视频转音频方面超越了当前最先进的方法,增强了与视觉内容的同步性,并改善了音频与视频组成部分之间的对齐。项目页面:https://yusiissy.github.io/SonicVisionLM.github.io/
英文摘要:
There has been a growing interest in the task of generating sound for silent videos, primarily because of its practicality in streamlining video post-production. However, existing methods for video-sound generation attempt to directly create sound from visual representations, which can be challenging due to the difficulty of aligning visual representations with audio representations. In this paper, we present SonicVisionLM, a novel framework aimed at generating a wide range of sound effects by leveraging vision-language models(VLMs). Instead of generating audio directly from video, we use the capabilities of powerful VLMs. When provided with a silent video, our approach first identifies events within the video using a VLM to suggest possible sounds that match the video content. This shift in approach transforms the challenging task of aligning image and audio into more well-studied sub-problems of aligning image-to-text and text-to-audio through the popular diffusion models. To improve the quality of audio recommendations with LLMs, we have collected an extensive dataset that maps text descriptions to specific sound effects and developed a time-controlled audio adapter. Our approach surpasses current state-of-the-art methods for converting video to audio, enhancing synchronization with the visuals, and improving alignment between audio and video components. Project page: https://yusiissy.github.io/SonicVisionLM.github.io/
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。