In this work we introduce NWT, an expressive speech-to-video model. Unlike approaches that use domain-specific intermediate representations such as pose keypoints, NWT learns its own latent representations, with minimal assumptions about the audio and video content. To this end, we propose a novel discrete variational autoencoder with adversarial loss, dVAE-Adv, which learns a new discrete latent representation we call Memcodes. Memcodes are straightforward to implement, require no additional loss terms, are stable to train compared with other approaches, and show evidence of interpretability. To predict on the Memcode space, we use an autoregressive encoder-decoder model conditioned on audio. Additionally, our model can control latent attributes in the generated video that are not annotated in the data. We train NWT on clips from HBO's Last Week Tonight with John Oliver. NWT consistently scores above other approaches in Mean Opinion Score (MOS) on tests of overall video naturalness, facial naturalness and expressiveness, and lipsync quality. This work sets a strong baseline for generalized audio-to-video synthesis. Samples are available at https://next-week-tonight.github.io/NWT/.
WorldCraft: From Camera Navigation to Object Manipulation in Interactive Video World Models
The Hong Kong University of Science and Technology(香港科学与技术大学);AI Technology Center, Tencent Video, Tencent(腾讯视频AI技术中心,腾讯);Wuhan University(武汉大学);Peking University(北京大学)
VABench: A Comprehensive Benchmark for Audio-Video Generation
Peking University(北京大学);Zhongguancun Academy(中关村学院);Beijing Key Laboratory of Data Intelligence (Peking University)(北京市数据智能重点实验室(北京大学));Ant Group(蚂蚁集团);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Huazhong University of Science and Technology(华中科技大学)
Tencent AI Lab(腾讯AI实验室);Beijing Institute of Technology(北京理工大学);University of California, Berkeley(加州大学伯克利分校);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Audio-Sync Video Generation with Multi-Stream Temporal Control
Beijing Academy of Artificial Intelligence(北京人工智能研究院);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);Nat’l Key Lab of General AI, School of Intelligence Science and Technology, Peking University(国家通用人工智能实验室,北京大学智能科学与技术学院);Nat’l Eng. Research Ctr. of Visual Tech., School of Computer Science, Peking University(国家视觉技术工程研究中心,北京大学计算机学院)
本文提出 MTV 框架,通过分离语音、音效和音乐多流音频实现细粒度音视频同步控制,并构建 DEMIX 数据集,在六项指标上达到最先进性能。
Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Lets Play Music: Audio-driven Performance Video Generation
Anhui University(安徽大学);University of Chinese Academy of Sciences(中国科学院大学);Center for Excellence in Brain Science and Intelligence Technology, CAS(中国科学院脑科学与智能技术卓越创新中心)
Multi Modal Adaptive Normalization for Audio to Video Generation
Hike(Hike公司)
语音驱动的人脸视频生成是一个复杂的问题,由于其多模态方面,即音频和视频域。音频包括许多基本特征,例如表情、音高、响度、韵律(说话风格),并且面部视频在头部运动、眨眼、嘴唇同步和各种面部动作单元的运动以及时间平滑度方面具有许多可变性。对于生成对抗网络来说,从音频输入和静态图像中合成高度表达的面部视频仍然是一项具有挑战性的任务。在本文中,我们提出了一种多模态自适应归一化(MAN)的基础架构,合成一个说话的人的视频的任意长度作为输入:一个音频信号和一个人的单个图像。该架构使用多模态自适应归一化、关键点热图预测器、光流预测器和基于类激活图[58]的层来学习表情面部组件的运动,从而生成给定人的高度表情化的讲话头部视频。多模态自适应归一化使用音频和视频的各种特征(诸如来自音频信号的Mel频谱图、音高、能量和预测的关键点热图/光流)和单个图像来学习相应的仿射参数以生成高度表达的视频。实验评估表明,与使用GANs的真实语音驱动面部动画(RSDGAN)[53],Speech 2 Vid [10]和其他方法相比,所提出的方法在多个定量指标上具有优异的性能,包括:SSIM(结构相似性指数),PSNR(峰值信噪比),CPBD(图像清晰度),WER(单词出错率),眨眼/秒和LMD(地标距离)。此外,定性评估和在线图灵测试证明了我们的方法的有效性。
UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions
State Key Laboratory for Novel Software Technology(新型软件技术国家重点实验室);Tencent Hunyuan(腾讯文英);Shanghai Jiao Tong University(上海交通大学);Renmin University of China(中国人民大学);Tsinghua University(清华大学);Shanghai AI Lab(上海人工智能实验室)
Audio-Driven Talking Face Video Generation with Dynamic Convolution Kernels
Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);University of Science and Technology of China(中国科学技术大学);Cardiff University(卡迪夫大学);Kwai Inc.(快手公司)