具备增强同步性的掩码式生成型视频转音频Transformer
Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity
AI总结：
针对视频转音频生成中同步性与音质难以兼顾的问题，提出MaskVAT模型，结合高质量音频编解码器与序列到序列掩码生成结构，同时实现高同步性、优音质与语义匹配，性能达同期最优水平。
摘要：
视频转音频（V2A）生成任务利用纯视觉视频特征生成与场景匹配的合理声音。关键在于生成的声音起始点需与对应的视觉动作对齐，否则会出现不自然的同步伪影。现有研究先后探索了以静态图像、视频特征为条件的声音生成器，要么侧重生成质量与语义匹配而忽略同步性，要么牺牲部分质量仅专注于提升同步效果。本文提出一种名为MaskVAT的V2A生成模型，将全频段高质量通用音频编解码器与序列到序列掩码生成模型互联。该组合可同时实现高音频质量、语义匹配与时间同步性建模。实验结果表明，通过将高质量编解码器与合适的预训练视听特征、序列到序列并行结构相结合，我们一方面能生成高度同步的结果，另一方面性能可与非编解码器式生成音频模型的当前最优水平相媲美。示例视频与生成音频可访问https://maskvat.github.io 获取。
英文摘要：
Video-to-audio (V2A) generation leverages visual-only video features to render plausible sounds that match the scene. Importantly, the generated sound onsets should match the visual actions that are aligned with them, otherwise unnatural synchronization artifacts arise. Recent works have explored the progression of conditioning sound generators on still images and then video features, focusing on quality and semantic matching while ignoring synchronization, or by sacrificing some amount of quality to focus on improving synchronization only. In this work, we propose a V2A generative model, named MaskVAT, that interconnects a full-band high-quality general audio codec with a sequence-to-sequence masked generative model. This combination allows modeling both high audio quality, semantic matching, and temporal synchronicity at the same time. Our results show that, by combining a high-quality codec with the proper pre-trained audio-visual features and a sequence-to-sequence parallel structure, we are able to yield highly synchronized results on one hand, whilst being competitive with the state of the art of non-codec generative audio models. Sample videos and generated audios are available at https://maskvat.github.io .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
