发表机构
Simon Fraser University; University of Cyprus; CYENS Centre of Excellence; Tel Aviv University（西蒙弗雷泽大学; 塞浦路斯大学; CYENS卓越中心; 特拉维夫大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
Soundwich提出无需训练的框架，将冻结的联合音视频流匹配模型转化为生成多个同步、可独立编辑音频主干并与共享视频耦合的生成器，实现源级时间控制和跨模态交互，提升视听一致性、时间控制、源分离和自然度，支持灵活编辑。
AI 中文摘要
最近的联合音视频生成模型可以合成带有同步声音的逼真视频，但通常将音频生成为单个混合音轨。这限制了源级控制，并且不同于实际的视听工作流程，在实际工作流程中，语音、音乐、音效和环境声音被表示为单独的可编辑音轨。我们引入了Soundwich，一个无需训练的框架，它将冻结的联合音视频流匹配模型转换为生成多个同步、可独立编辑的音频主干（stems）的生成器，这些主干与共享视频耦合。Soundwich生成单独的音频主干，并对其时间活动进行显式控制。为了保持单独生成的声音的连贯性，我们引入了一种共享场景表示，该表示在主干之间传递全局视听上下文，同时保持其源级分离。我们进一步路由每个音频主干与其对应视觉源之间的跨模态交互，提高了视听一致性。生成的音频主干保持与视频同步，并且可以独立地重新定时、静音、替换或重新混合。实验和人工评估表明，在连贯的视听生成中，时间控制、源分离和自然度得到了改善，同时实现了灵活的源级编辑。代码可在以下网址获取：https://this https URL。
英文摘要
Recent joint audio-video generative models can synthesize realistic videos with synchronized sound, but typically generate audio as a single mixed track. This limits source-level control and differs from practical audiovisual workflows, where speech, music, sound effects, and ambient sounds are represented as separate editable tracks. We introduce Soundwich, a training-free framework that transforms a frozen joint audio-video flow-matching model into a generator of multiple synchronized, independently editable audio stems coupled to a shared video. Soundwich generates separate audio stems with explicit control over their temporal activity. To keep separately generated sounds coherent, we introduce a shared scene representation that communicates global audiovisual context across stems while preserving their source-level separation. We further route cross-modal interactions between each audio stem and its corresponding visual source, improving audiovisual consistency. The resulting stems remain synchronized with the video and can be independently retimed, muted, replaced, or remixed. Experiments and human evaluations show improved temporal control, source separation, and naturalness, while enabling flexible source-level editing within coherent audiovisual generation. Code is available at https://github.com/CodyNing/Soundwich.
Comments34 pages. Code: https://github.com/CodyNing/Soundwich
