通过观看电影学习音频高亮处理
Learning to Highlight Audio by Watching Movies
AI总结：
针对音视频显著性脱节问题，提出视觉引导的声学高亮新任务，构建基于transformer的多模态框架与muddy mix数据集，通过三步伪数据生成训练，在多类评估中优于基线。
摘要：
近年来，视频内容的创作与消费量大幅增长。打造有吸引力的内容需要精心策划视觉与音频元素。尽管通过最佳视角选择或后期编辑等技术进行的视觉线索策划一直是媒体制作的核心，但与之对应的音频领域却未取得同等进展，这往往导致视觉显著性与听觉显著性之间脱节。为弥补这一差距，我们提出一项新任务：视觉引导的声学高亮，旨在通过配套视频引导对音频进行转换，以呈现恰当的高亮效果，最终打造更和谐的视听体验。我们提出了一个灵活的、基于transformer的多模态框架来解决该任务。为训练模型，我们还引入了一个新数据集——muddy mix数据集，该数据集利用电影中精心制作的音视频内容，提供了一种免费监督信号。我们开发了一套伪数据生成流程来模拟混音效果不佳的音频，通过分离、调整、再混合三步过程模拟真实场景。我们的方法在定量评估和主观评估中均始终优于多个基线模型。我们还系统研究了不同类型的上下文引导以及数据集难度等级带来的影响。我们的项目页面位于：https://wikichao.github.io/VisAH/。
英文摘要：
Recent years have seen a significant increase in video content creation and consumption. Crafting engaging content requires the careful curation of both visual and audio elements. While visual cue curation, through techniques like optimal viewpoint selection or post-editing, has been central to media production, its natural counterpart, audio, has not undergone equivalent advancements. This often results in a disconnect between visual and acoustic saliency. To bridge this gap, we introduce a novel task: visually-guided acoustic highlighting, which aims to transform audio to deliver appropriate highlighting effects guided by the accompanying video, ultimately creating a more harmonious audio-visual experience. We propose a flexible, transformer-based multimodal framework to solve this task. To train our model, we also introduce a new dataset -- the muddy mix dataset, leveraging the meticulous audio and video crafting found in movies, which provides a form of free supervision. We develop a pseudo-data generation process to simulate poorly mixed audio, mimicking real-world scenarios through a three-step process -- separation, adjustment, and remixing. Our approach consistently outperforms several baselines in both quantitative and subjective evaluation. We also systematically study the impact of different types of contextual guidance and difficulty levels of the dataset. Our project page is here: https://wikichao.github.io/VisAH/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
