Sound and movement are closely coupled, particularly in dance. Certain audio features have been found to affect the way we move to music. Is this relationship between sound and movement something which can be modelled using machine learning? This work presents initial experiments wherein high-level audio features calculated from a set of music pieces are included in a movement generation model trained on motion capture recordings of improvised dance. Our results indicate that the model learns to generate realistic dance movements which vary depending on the audio features.
Multidisciplinary Institute of Artificial Intelligence (MIAI)（多学科人工智能研究所）
This paper investigates the impact of head movements on audio-visual speech enhancement (AVSE). Although being a common conversational feature, head movements have been ignored by past and recent studies: they challenge today's learning-based methods as they often degrade the performance of models that are trained on clean, frontal, and steady face images. To alleviate this problem, we propose to use robust face frontalization (RFF) in combination with an AVSE method based on a variational auto-encoder (VAE) model. We briefly describe the basic ingredients of the proposed pipeline and we perfo…
We propose an end to end deep learning approach for generating real-time facial animation from just audio. Specifically, our deep architecture employs deep bidirectional long short-term memory network and attention mechanism to discover the latent representations of time-varying contextual information within the speech and recognize the significance of different information contributed to certain face status. Therefore, our model is able to drive different levels of facial movements at inference and automatically keep up with the corresponding pitch and latent speaking style in the input audio…
Synthesizing human motion through learning techniques is becoming an increasingly popular approach to alleviating the requirement of new data capture to produce animations. Learning to move naturally from music, i.e., to dance, is one of the more complex motions humans often perform effortlessly. Each dance movement is unique, yet such movements maintain the core characteristics of the dance style. Most approaches addressing this problem with classical convolutional and recursive neural models undergo training and variability issues due to the non-Euclidean geometry of the motion manifold stru…
Tianjin University（天津大学）；University of Maryland at College Park（马里兰大学学院公园分校）
We present a learning-based approach for generating binaural audio from mono audio using multi-task learning. Our formulation leverages additional information from two related tasks: the binaural audio generation task and the flipped audio classification task. Our learning model extracts spatialization features from the visual and audio input, predicts the left and right audio channels, and judges whether the left and right channels are flipped. First, we extract visual features using ResNet from the video frames. Next, we perform binaural audio generation and flipped audio classification usin…
We consider the challenging problem of audio to animated video generation. We propose a novel method OneShotAu2AV to generate an animated video of arbitrary length using an audio clip and a single unseen image of a person as an input. The proposed method consists of two stages. In the first stage, OneShotAu2AV generates the talking-head video in the human domain given an audio and a person's image. In the second stage, the talking-head video from the human domain is converted to the animated domain. The model architecture of the first stage consists of spatially adaptive normalization based mu…
Academia Sinica（中央研究院）；National Taiwan University（国立台湾大学）；Taiwan AI Labs（台湾人工智能实验室）
While generative adversarial networks (GANs) have been widely used in research on audio generation, the training of a GAN model is known to be unstable, time consuming, and data inefficient. Among the attempts to ameliorate the training process of GANs, the idea of Projected GAN emerges as an effective solution for GAN-based image generation, establishing the state-of-the-art in different image applications. The core idea is to use a pre-trained classifier to constrain the feature space of the discriminator to stabilize and improve GAN training. This paper investigates whether Projected GAN ca…
Research Center for IT Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）
We present in this paper PerformacnceNet, a neural network model we proposed recently to achieve score-to-audio music generation. The model learns to convert a music piece from the symbolic domain to the audio domain, assigning performance-level attributes such as changes in velocity automatically to the music and then synthesizing the audio. The model is therefore not just a neural audio synthesizer, but an AI performer that learns to interpret a musical score in its own way. The code and sample outputs of the model can be found online at https://github.com/bwang514/PerformanceNet.
Unified Audio Generation and Editing via Joint Condition Modeling and Progressive Training
Department of Electronic Engineering and Information Science, University of Science and Technology of China（电子工程与信息科学系，中国科学技术大学）；Tianjin Key Laboratory of Cognitive Computing and Application, School of Artificial Intelligence, Tianjin University（认知计算与应用重点实验室，天津大学人工智能学院）；Institute of Artificial Intelligence of China Telecom (TeleAI)（中国电信人工智能研究院（TeleAI））
LAFMA: A Latent Flow Matching Model for Text-to-Audio Generation
School of Informatics, Xiamen University（厦门大学信息学院）；School of Electronic Science and Engineering, Xiamen University（厦门大学电子科学与工程学院）；Graduate School of Informatics, Kyoto University（京都大学信息学研究科）；Kuaishou Technology（快手科技）；Nankai University（南开大学）
Korea University（高丽大学）；Netmarble Corporation（网石公司）
In recent works, a flow-based neural vocoder has shown significant improvement in real-time speech generation task. The sequence of invertible flow operations allows the model to convert samples from simple distribution to audio samples. However, training a continuous density model on discrete audio data can degrade model performance due to the topological difference between latent and actual distribution. To resolve this problem, we propose audio dequantization methods in flow-based neural vocoder for high fidelity audio generation. Data dequantization is a well-known method in image generati…
Tokyo Institute of Technology（东京工业大学）；Honda Research Institute Japan Co., Ltd.（本田日本研究所）
We describe a novel metric-based learning approach that introduces a multimodal framework and uses deep audio and geophone encoders in siamese configuration to design an adaptable and lightweight supervised model. This framework eliminates the need for expensive data labeling procedures and learns general-purpose representations from low multisensory data obtained from omnipresent sensing systems. These sensing systems provide numerous applications and various use cases in activity recognition tasks. Here, we intend to explore the human footstep movements from indoor environments and analyze r…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Audio-visual target speech extraction, which aims to extract a certain speaker's speech from the noisy mixture by looking at lip movements, has made significant progress combining time-domain speech separation models and visual feature extractors (CNN). One problem of fusing audio and video information is that they have different time resolutions. Most current research upsamples the visual features along the time dimension so that audio and video features are able to align in time. However, we believe that lip movement should mostly contain long-term, or phone-level information. Based on this …
CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation
Australian Institute for Machine Learning, University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学）；Sony Group Corporation（索尼集团）；Sony AI, Sony Group Corporation（索尼人工智能，索尼集团）
The recent surge in popularity of diffusion models for image generation has brought new attention to the potential of these models in other areas of media generation. One area that has yet to be fully explored is the application of diffusion models to audio generation. Audio generation requires an understanding of multiple aspects, such as the temporal dimension, long term structure, multiple layers of overlapping sounds, and the nuances that only trained listeners can detect. In this work, we investigate the potential of diffusion models for audio generation. We propose a set of models to tac…
We propose a novel system that takes as an input body movements of a musician playing a musical instrument and generates music in an unsupervised setting. Learning to generate multi-instrumental music from videos without labeling the instruments is a challenging problem. To achieve the transformation, we built a pipeline named 'Multi-instrumentalistNet' (MI Net). At its base, the pipeline learns a discrete latent representation of various instruments music from log-spectrogram using a Vector Quantized Variational Autoencoder (VQ-VAE) with multi-band residual blocks. The pipeline is then traine…
AudioRAG+: Feedback-driven Retrieval-augmented Audio Generation with Large Audio Language Models
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK（视觉、语音和信号处理中心（CVSSP），英国萨里大学）；Tencent AI Lab, Beijing, China（腾讯人工智能实验室，中国北京）；Tencent AI Lab, Seattle, USA（腾讯人工智能实验室，美国西雅图）
Text Prompt is Not Enough: Sound Event Enhanced Prompt Adapter for Target Style Audio Generation
SDU-ANU Joint Science College, Shandong University（山东大学SDU-ANU联合科学学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tsinghua University（清华大学）；AI Lab, Tencent（腾讯AI实验室）
DreamAudio: Customized Text-to-Audio Generation with Diffusion Models
School of Computer Science and Electronic Engineering, University of Surrey（Surrey大学计算机科学与电子工程学院）；Department of Informatics, King’s College London（伦敦国王学院信息学院）；Seed Group, ByteDance Inc.（字节跳动Seed团队）
AudioTurbo: Fast Text-to-Audio Generation with Rectified Diffusion
University of SurreyUK（英国萨里大学）；Institute of Acoustics, Chinese Academy of SciencesChina（中国科学院声学研究所）；Centre for Vision, Speech and Signal Processing (CVSSP) Laboratory of Noise and Audio Research（噪声与音频研究实验室）
