ALIVE: 用逼真音频视频生成动画你的世界
ALIVE: Animate Your World with Lifelike Audio-Video Generation
AI总结：
ALIVE通过结合预训练文本到视频模型与新的音频视频生成技术，实现了逼真的音频视频生成和动画，展示了优于开源和商业解决方案的性能。
摘要：
视频生成正迅速向统一的音频视频生成发展。在本文中，我们提出了ALIVE，一种能够将预训练的文本到视频（T2V）模型适应到Sora风格的音频视频生成和动画中的生成模型。特别是，该模型解锁了文本到视频和音频（T2VA）以及参考到视频和音频（动画）的能力，这比T2V基础模型有所提升。为了支持音频视觉同步和参考动画，我们增强了流行的MMDiT架构，加入了联合音频视频分支，其中包括TA-CrossAttn用于时间对齐的跨模态融合和UniTemp-RoPE用于精确的音频视觉对齐。同时，一个包含音频视频描述、质量控制等的综合性数据管道被精心设计，以收集高质量的微调数据。此外，我们引入了一个新的基准测试，以进行全面的模型测试和比较。在继续预训练和微调千万级高质量数据后，ALIVE表现出卓越的性能，持续优于开源模型，并与或超越了最先进的商业解决方案。通过详细的配方和基准测试，我们希望ALIVE能帮助社区更高效地开发音频视频生成模型。官方页面：https://github.com/FoundationVision/Alive。
英文摘要：
Video generation is rapidly evolving towards unified audio-video generation. In this paper, we present ALIVE, a generation model that adapts a pretrained Text-to-Video (T2V) model to Sora-style audio-video generation and animation. In particular, the model unlocks the Text-to-Video&Audio (T2VA) and Reference-to-Video&Audio (animation) capabilities compared to the T2V foundation models. To support the audio-visual synchronization and reference animation, we augment the popular MMDiT architecture with a joint audio-video branch which includes TA-CrossAttn for temporally-aligned cross-modal fusion and UniTemp-RoPE for precise audio-visual alignment. Meanwhile, a comprehensive data pipeline consisting of audio-video captioning, quality control, etc., is carefully designed to collect high-quality finetuning data. Additionally, we introduce a new benchmark to perform a comprehensive model test and comparison. After continue pretraining and finetuning on million-level high-quality data, ALIVE demonstrates outstanding performance, consistently outperforming open-source models and matching or surpassing state-of-the-art commercial solutions. With detailed recipes and benchmarks, we hope ALIVE helps the community develop audio-video generation models more efficiently. Official page: https://github.com/FoundationVision/Alive.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
