具有多流时间控制的音频同步视频生成
Audio-Sync Video Generation with Multi-Stream Temporal Control
AI总结：
本文提出 MTV 框架，通过分离语音、音效和音乐多流音频实现细粒度音视频同步控制，并构建 DEMIX 数据集，在六项指标上达到最先进性能。
摘要：
音频本质上具有时间性，并与视觉世界紧密同步，这使其成为可控视频生成（例如电影）中自然对齐且富有表现力的控制信号。除控制之外，将音频直接转化为视频对于理解和可视化丰富的音频叙事（例如播客或历史录音）也至关重要。然而，现有方法在生成具有精确音视频同步的高质量视频方面存在不足，尤其是在处理多样且复杂的音频类型时。在本工作中，我们提出了 MTV，这是一个用于音频同步视频生成的通用框架。MTV 显式地将音频分离为语音、音效和音乐轨道，从而能够分别对口唇运动、事件时机和视觉氛围进行解耦控制——进而实现细粒度且语义对齐的视频生成。为支持该框架，我们还提出了 DEMIX，这是一个包含高质量电影视频和解混音频轨道的数据集。DEMIX 被组织为五个重叠子集，可为多样的生成场景支持可扩展的多阶段训练。大量实验表明，MTV 在涵盖视频质量、文本-视频一致性和音频-视频对齐的六项标准指标上均达到最先进性能。项目页面：https://hjzheng.net/projects/MTV/。
英文摘要：
Audio is inherently temporal and closely synchronized with the visual world, making it a naturally aligned and expressive control signal for controllable video generation (e.g., movies). Beyond control, directly translating audio into video is essential for understanding and visualizing rich audio narratives (e.g., Podcasts or historical recordings). However, existing approaches fall short in generating high-quality videos with precise audio-visual synchronization, especially across diverse and complex audio types. In this work, we introduce MTV, a versatile framework for audio-sync video generation. MTV explicitly separates audios into speech, effects, and music tracks, enabling disentangled control over lip motion, event timing, and visual mood, respectively -- resulting in fine-grained and semantically aligned video generation. To support the framework, we additionally present DEMIX, a dataset comprising high-quality cinematic videos and demixed audio tracks. DEMIX is structured into five overlapped subsets, enabling scalable multi-stage training for diverse generation scenarios. Extensive experiments demonstrate that MTV achieves state-of-the-art performance across six standard metrics spanning video quality, text-video consistency, and audio-video alignment. Project page: https://hjzheng.net/projects/MTV/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
