We consider the challenging problem of audio to animated video generation. We propose a novel method OneShotAu2AV to generate an animated video of arbitrary length using an audio clip and a single unseen image of a person as an input. The proposed method consists of two stages. In the first stage, OneShotAu2AV generates the talking-head video in the human domain given an audio and a person's image. In the second stage, the talking-head video from the human domain is converted to the animated domain. The model architecture of the first stage consists of spatially adaptive normalization based multi-level generator and multiple multilevel discriminators along with multiple adversarial and non-adversarial losses. The second stage leverages attention based normalization driven GAN architecture along with temporal predictor based recycle loss and blink loss coupled with lipsync loss, for unsupervised generation of animated video. In our approach, the input audio clip is not restricted to any specific language, which gives the method multilingual applicability. OneShotAu2AV can generate animated videos that have: (a) lip movements that are in sync with the audio, (b) natural facial expressions such as blinks and eyebrow movements, (c) head movements. Experimental evaluation demonstrates superior performance of OneShotAu2AV as compared to U-GAT-IT and RecycleGan on multiple quantitative metrics including KID(Kernel Inception Distance), Word error rate, blinks/sec
SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion
University of Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学);Nanyang Technological University(南洋理工大学);City University of Hong Kong(香港城市大学)
UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating
The Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong(香港中文大学);Kling Team, Kuaishou Technology(Kling团队,快手科技);Tsinghua University(清华大学);Sun Yat-sen University(中山大学)
Character-Centric Understanding of Animated Movies
Visual Geometry Group Dept.\ of Engineering Science University of Oxford, UK;School of Artifitial Intelligence\ Jiao Tong University Shanghai China;School of Artifitial Intelligence\ Jiao Tong University
CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation
Shanghai Jiao Tong University(上海交通大学);University of Electronic Science and Technology of China(电子科技大学);Zhejiang University(浙江大学);The University of Tokyo(东京大学);Nanyang Technological University(南洋理工大学)
Decoding while Adapting: Zero-Shot Online Speaker Adaptation via Audio-Textual Prompts for Elderly Speech Recognition
The Chinese University of Hong Kong, Hong Kong SAR, China(香港中文大学);Institute of Software, Chinese Academy of Sciences, China(中国科学院软件研究所);National Research Council Canada, Canada(加拿大国家研究理事会)
Zero-Shot Federated Learning with New Classes for Audio Classification
Global AI Accelerator, Ericsson(爱立信全球人工智能加速器);Ericsson Research(爱立信研究院)
联邦学习是一种从不同用户设备中提取见解同时保护用户隐私的有效方法。然而,具有完全不可见的数据分布的新类可以在联邦学习设置中的任何设备上流式传输,其数据不能被全局服务器或其他用户访问。为此,我们提出了一个统一的zero-shot框架来处理联邦学习过程中的这些挑战。我们在这里模拟了两种情况- 1)当用户没有报告新的类别标签时,使用传统的FL设置; 2)当用户报告新的类别标签时,我们通过计算与每个设备的新类别对应的类别相似性矩阵来合成Ancesterized Data Imperial,然后进行无监督聚类以区分不同用户的新类别。此外,我们提出的框架还可以处理参与用户的标签和模型中的统计异质性。我们在两个广泛使用的音频分类应用程序(关键字识别和城市声音分类)上,通过本地和全局更新中的新类别以及异构标签和模型,在不同的通信回合(FL迭代)中对我们的设备上框架进行了经验评估,并观察到平均确定性准确度分别增加了~4.041%和~4.258%。
AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Alibaba Group(阿里巴巴集团);Nanjing University(南京大学)
AudioStory: Generating Long-Form Narrative Audio with Large Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);ARC Lab, Tencent PCG(腾讯PCG ARC实验室);MAIS, Institute of Automation, CAS, Beijing(自动化研究所北京研究所MAIS)
MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Control
Peking University(北京大学);Kling Team, Kuaishou Technology(快手团队);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Sun Yat-sen University(中山大学)
Exploiting Temporal Audio-Visual Correlation Embedding for Audio-Driven One-Shot Talking Head Animation
School of Information Engineering, Guangdong University of Technology(广东工业大学信息工程学院);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院)
On Aligning Hierarchical Standardized Embedding for Audio-visual Generalized Zero-shot Learning
Southeast University(东南大学);The University of Hong Kong(香港大学);Beijing Institute of Technology(北京理工大学);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education(新一代人工智能技术及其跨学科应用重点实验室(东南大学),教育部);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院)
SpurAudio: A Benchmark for Studying Shortcut Learning in Few-Shot Audio Classification
Department of Computer Science, University of Haifa(海法大学计算机科学系);Independent Researcher(独立研究者);University of Stuttgart, Germany(斯图加特大学,德国);IMPRS-IS, Germany(智能系统国际Max Planck研究学校,德国)
Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models
University of Tübingen(蒂宾根大学);Tübingen AI Center(蒂宾根人工智能中心);Helmholtz Munich(慕尼黑亥姆霍兹中心);Technical University of Munich(慕尼黑工业大学);Localyzer GmbH(Localyzer有限公司)
SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM
Department of Computer Science, University at Albany - SUNY(University at Albany - SUNY 计算机科学系);School of Software & Microelectronics, Peking University(北京大学软件与微电子学院);Nanjing University(南京大学);Xiamen University(厦门大学);Department of Mathematics and Statistics, University at Albany - SUNY(University at Albany - SUNY 数学与统计学系)
PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation
School of Computer Science and Electronic Engineering, University of Surrey(计算机科学与电子工程系,塞夫顿大学);Centre for Vision, Speech and Signal Processing, University of Surrey(视觉、语音与信号处理中心,塞夫顿大学);School of Artificial Intelligence and Computer Science, Jiangnan University(人工智能与计算机科学系,江南大学)