We propose a novel task for generating 3D dance movements that simultaneously incorporate both text and music modalities. Unlike existing works that generate dance movements using a single modality such as music, our goal is to produce richer dance movements guided by the instructive information provided by the text. However, the lack of paired motion data with both music and text modalities limits the ability to generate dance movements that integrate both. To alleviate this challenge, we propose to utilize a 3D human motion VQ-VAE to project the motions of the two datasets into a latent space consisting of quantized vectors, which effectively mix the motion tokens from the two datasets with different distributions for training. Additionally, we propose a cross-modal transformer to integrate text instructions into motion generation architecture for generating 3D dance movements without degrading the performance of music-conditioned dance generation. To better evaluate the quality of the generated motion, we introduce two novel metrics, namely Motion Prediction Distance (MPD) and Freezing Score (FS), to measure the coherence and freezing percentage of the generated motion. Extensive experiments show that our approach can generate realistic and coherent dance movements conditioned on both text and music while maintaining comparable performance with the two single modalities. Code is available at https://garfield-kh.github.io/TM2D/.
University of California, Merced（加利福尼亚大学默塞德分校）；NVIDIA（英伟达公司）
Dancing to music is an instinctive move by humans. Learning to model the music-to-dance generation process is, however, a challenging problem. It requires significant efforts to measure the correlation between music and dance as one needs to simultaneously consider multiple aspects, such as style and beat of both music and dance. Additionally, dance is inherently multimodal and various following movements of a pose at any moment are equally likely. In this paper, we propose a synthesis-by-analysis learning framework to generate dance from music. In the analysis phase, we decompose a dance into…
Close your eyes and listen to music, one can easily imagine an actor dancing rhythmically along with the music. These dance movements are usually made up of dance movements you have seen before. In this paper, we propose to reproduce such an inherent capability of the human-being within a computer vision system. The proposed system consists of three modules. To explore the relationship between music and dance movements, we propose a cross-modal alignment module that focuses on dancing video clips, accompanied on pre-designed music, to learn a system that can judge the consistency between the v…
Synthesize human motions from music, i.e., music to dance, is appealing and attracts lots of research interests in recent years. It is challenging due to not only the requirement of realistic and complex human motions for dance, but more importantly, the synthesized motions should be consistent with the style, rhythm and melody of the music. In this paper, we propose a novel autoregressive generative model, DanceNet, to take the style, rhythm and melody of music as the control signals to generate 3D dance motions with high realism and diversity. To boost the performance of our proposed model, …
Spatial-Temporal Graph Mamba for Music-Guided Dance Video Synthesis
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；UCAS-Terminus AI Lab, University of Chinese Academy of Sciences（中国科学院大学Terminus AI实验室，中国科学院大学）；School of Intelligent Science and Technology, Nanjing University（智能科学与技术学院，南京大学）；Department of Information Technology and Electrical Engineering, ETH Zurich（信息科技与电气工程系，苏黎世联邦理工学院）；Department of Electrical Engineering, KU Leuven（电气工程系，鲁汶大学）；INSAIT, Sofia Un.（INSAIT，索菲亚大学）；Department of Information Engineering and Computer Science (DISI), University of Trento（信息工程与计算机科学系（DISI），特伦托大学）
DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心）；ByteDance（字节跳动）；University of Rochester（罗切斯特大学）
针对音乐与舞蹈的相机运动合成缺乏成对数据的挑战，提出新多模态数据集 DCM 及基于 transformer 的扩散模型 DanceCamera3D，结合 body attention loss 与条件分离策略，并设计新评估指标验证了有效性。
NetEase Fuxi AI Lab（网易伏羲AI实验室）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；Zhejiang University（浙江大学）
Music-to-dance translation is a brand-new and powerful feature in recent role-playing games. Players can now let their characters dance along with specified music clips and even generate fan-made dance videos. Previous works of this topic consider music-to-dance as a supervised motion generation problem based on time-series data. However, these methods suffer from limited training data pairs and the degradation of movements. This paper provides a new perspective for this task where we re-formulate the translation problem as a piece-wise dance phrase retrieval problem based on the choreography …
Black Sesame Technologies（黑芝麻科技）；Nanyang Technological University（南洋理工大学）；University of Alberta（阿尔伯塔大学）
Dance choreography for a piece of music is a challenging task, having to be creative in presenting distinctive stylistic dance elements while taking into account the musical theme and rhythm. It has been tackled by different approaches such as similarity retrieval, sequence-to-sequence modeling and generative adversarial networks, but their generated dance sequences are often short of motion realism, diversity and music consistency. In this paper, we propose a Music-to-Dance with Optimal Transport Network (MDOT-Net) for learning to generate 3D dance choreographies from music. We introduce an o…
We present a learning-based approach with pose perceptual loss for automatic music video generation. Our method can produce a realistic dance video that conforms to the beats and rhymes of almost any given music. To achieve this, we firstly generate a human skeleton sequence from music and then apply the learned pose-to-appearance mapping to generate the final video. In the stage of generating skeleton sequences, we utilize two discriminators to capture different aspects of the sequence and propose a novel pose perceptual loss to produce natural dances. Besides, we also provide a new cross-mod…
Illinois Institute of Technology（伊利诺伊理工学院）；Snap Inc.（Snap公司）；Princeton University（普林斯顿大学）
We present Dance2Music-GAN (D2M-GAN), a novel adversarial multi-modal framework that generates complex musical samples conditioned on dance videos. Our proposed framework takes dance video frames and human body motions as input, and learns to generate music samples that plausibly accompany the corresponding input. Unlike most existing conditional music generation works that generate specific types of mono-instrumental sounds using symbolic audio representations (e.g., MIDI), and that usually rely on pre-defined musical synthesizers, in this work we generate dance music in complex styles (e.g.,…
Dance is an important human art form, but creating new dances can be difficult and time-consuming. In this work, we introduce Editable Dance GEneration (EDGE), a state-of-the-art method for editable dance generation that is capable of creating realistic, physically-plausible dances while remaining faithful to the input music. EDGE uses a transformer-based diffusion model paired with Jukebox, a strong music feature extractor, and confers powerful editing capabilities well-suited to dance, including joint-wise conditioning, and in-betweening. We introduce a new metric for physical plausibility, …
Google Research（谷歌研究院）；University of California, Berkeley（加州大学伯克利分校）；University of Southern California（南加州大学）
We present AIST++, a new multi-modal dataset of 3D dance motion and music, along with FACT, a Full-Attention Cross-modal Transformer network for generating 3D dance motion conditioned on music. The proposed AIST++ dataset contains 5.2 hours of 3D dance motion in 1408 sequences, covering 10 dance genres with multi-view videos with known camera poses -- the largest dataset of this kind to our knowledge. We show that naively applying sequence models such as transformers to this dataset for the task of music conditioned 3D motion generation does not produce satisfactory 3D motion that is well corr…
Exploring Multi-Modal Control in Music-Driven Dance Generation
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
Tempo as the Stable Cue: Hierarchical Mixture of Tempo and Beat Experts for Music to 3D Dance Generation
School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；Hohai university（河海大学）；School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Institute for Infocomm Research (I 2 R), A*STAR（新加坡科技研究局信息与通信研究所）
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Electronics Engineering and Computer Science, Peking University（北京大学电子工程与计算机科学学院）；National Institute of Health Data Science, Peking University（北京大学健康数据科学研究所）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）
GCDance: Genre-Controlled Music-Driven 3D Full Body Dance Generation
School of Computer Science and Electronic Engineering, University of Surrey（萨里大学计算机科学与电子工程学院）；Department of Music and Media, University of Surrey（萨里大学音乐与媒体系）；Department of Informatics, University of Oslo（奥斯陆大学信息学系）；Centre for Vision, Speech and Signal Processing, University of Surrey（萨里大学视觉、语音与信号处理中心）；School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）
PAMD: Plausibility-Aware Motion Diffusion Model for Long Dance Generation
School of Computer Science and Engineering, Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, Southeast University（计算机科学与工程学院，新一代人工智能技术及交叉应用重点实验室，教育部，东南大学）；State Key Laboratory of Media Convergence and Communication, Communication University of China（媒体融合与传播国家重点实验室，中国传媒大学）；School of Computer Science and Software Engineering, National Engineering Laboratory for Big Data System Computing Technology, Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University（计算机科学与软件工程学院，大数据系统计算技术国家工程实验室，广东智能信息处理重点实验室，深圳大学）；School of Computer Science and Engineering, Nanjing University of Science and Technology（计算机科学与工程学院，南京理工大学）
Align Your Rhythm: Generating Highly Aligned Dance Poses with Gating-Enhanced Rhythm-Aware Feature Representation
College of Computer Science and Technology, Harbin Engineering University（哈尔滨工程大学计算机科学与技术学院）；Shanghai Academy of Artificial Intelligence for Science（上海人工智能科学研究院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）；State Key Laboratory of Space-Ground Integrated Information Technology（空间地面一体化信息技术创新实验室）
Driving 3D characters to dance following a piece of music is highly challenging due to the spatial constraints applied to poses by choreography norms. In addition, the generated dance sequence also needs to maintain temporal coherency with different music genres. To tackle these challenges, we propose a novel music-to-dance framework, Bailando, with two powerful components: 1) a choreographic memory that learns to summarize meaningful dancing units from 3D pose sequence to a quantized codebook, 2) an actor-critic Generative Pre-trained Transformer (GPT) that composes these units to a fluent da…
TCDiff++: An End-to-end Trajectory-Controllable Diffusion Model for Harmonious Music-Driven Group Choreography
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China（教育部高维信息智能感知与系统重点实验室，南京理工大学计算机科学与工程学院）；Shenzhen International Graduate School, Tsinghua University, Shenzhen, China（清华大学深圳国际研究生院）；Nanjing University, Suzhou, China（南京大学）
Synthesizing human motion through learning techniques is becoming an increasingly popular approach to alleviating the requirement of new data capture to produce animations. Learning to move naturally from music, i.e., to dance, is one of the more complex motions humans often perform effortlessly. Each dance movement is unique, yet such movements maintain the core characteristics of the dance style. Most approaches addressing this problem with classical convolutional and recursive neural models undergo training and variability issues due to the non-Euclidean geometry of the motion manifold stru…
QEAN: Quaternion-Enhanced Attention Network for Visual Dance Generation
Guangdong University of Technology（广东工业大学）；Huizhou University（惠州学院）；Guangdong Mechanical and Electrical College（广东机电职业技术学院）；University of Western Australia（西澳大学）
Not Like Transformers: Drop the Beat Representation for Dance Generation with Mamba-Based Diffusion Model
Ulsan National Institute of Science and Technology, South Korea（乌山国立科学技术研究院，韩国）；Daegu Gyeongbuk Institute of Science and Technology, South Korea（大邱庆北科学技术研究院，韩国）
