看见声音：使用 Mirage 从音频生成 A-Roll 视频
Seeing Voices: Generating A-Roll Video from Audio with Mirage
AI总结：
本文提出音频到视频基础模型 Mirage，利用统一的自注意力训练方法从语音音频生成逼真且富有表现力的人物 A-roll 视频，并在通用性和主观质量上优于专用架构方法。
摘要：
从专业电影制作到用户生成内容，创作者和消费者长期以来都认识到，视频的力量取决于我们听到的内容（视频的音轨）与我们看到的内容（视频的图像序列）之间的和谐融合。当前的视频生成方法要么忽略声音，专注于通用但无声的图像序列生成；要么同时处理视觉和音频元素，但聚焦于重新配音等受限的应用领域。我们提出 Mirage，这是一个音频到视频基础模型，擅长在给定音频输入的情况下从头生成逼真、富有表现力的输出图像。当与现有的语音合成（text-to-speech，即 TTS）方法结合时，Mirage 可生成引人入胜的多模态视频。在人们说话的音视频素材（A-roll）上进行训练，并以包含语音的音频为条件时，Mirage 会生成人物视频，对输入音频中隐含的表演作出可信演绎。我们的核心技术贡献是一种统一方法，用于训练基于自注意力的音频到视频生成模型，既可以从头训练，也可以基于现有权重训练。该方法使 Mirage 保持了作为音频到视频生成方法的通用性，同时相比那些纳入音频专用架构，或针对人物、语音、图像或音频采集细节的专用损失组件的方法，能够产生主观质量更优的输出。我们鼓励读者亲自观看并聆听 Mirage 的结果（链接见论文和评论）。
英文摘要：
From professional filmmaking to user-generated content, creators and consumers have long recognized that the power of video depends on the harmonious integration of what we hear (the video's audio track) with what we see (the video's image sequence). Current approaches to video generation either ignore sound to focus on general-purpose but silent image sequence generation or address both visual and audio elements but focus on restricted application domains such as re-dubbing. We introduce Mirage, an audio-to-video foundation model that excels at generating realistic, expressive output imagery from scratch given an audio input. When integrated with existing methods for speech synthesis (text-to-speech, or TTS), Mirage results in compelling multimodal video. When trained on audio-video footage of people talking (A-roll) and conditioned on audio containing speech, Mirage generates video of people delivering a believable interpretation of the performance implicit in input audio. Our central technical contribution is a unified method for training self-attention-based audio-to-video generation models, either from scratch or given existing weights. This methodology allows Mirage to retain generality as an approach to audio-to-video generation while producing outputs of superior subjective quality to methods that incorporate audio-specific architectures or loss components specific to people, speech, or details of how images or audio are captured. We encourage readers to watch and listen to the results of Mirage for themselves (see paper and comments for links).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
