UniMuMo: Unified Text, Music and Motion Generation
The Chinese University of Hong Kong（香港中文大学）；University of Washington（华盛顿大学）；The University of British Columbia（不列颠哥伦比亚大学）；UMass Amherst（马萨诸塞大学阿默斯特分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Cisco Research（思科研究部门）
Han Yang, Kun Su, Yutong Zhang, Jiaben Chen, Kaizhi Qian, Gaowen Liu, Chuang Gan
We introduce UniMuMo, a unified multimodal model capable of taking arbitrary text, music, and motion data as input conditions to generate outputs across all three modalities. To address the lack of time-synchronized data, we align unpaired music and motion data based on rhythmic patterns to leverage existing large-scale music-only and motion-only datasets. By converting music, motion, and text into token-based representation, our model bridges these modalities through a unified encoder-decoder transformer architecture. To support multiple generation tasks within a single framework, we introduce several architectural improvements. We propose encoding motion with a music codebook, mapping motion into the same feature space as music. We introduce a music-motion parallel generation scheme that unifies all music and motion generation tasks into a single transformer decoder architecture with a single training task of music-motion joint generation. Moreover, the model is designed by fine-tuning existing pre-trained single-modality models, significantly reducing computational demands. Extensive experiments demonstrate that UniMuMo achieves competitive results on all unidirectional generation benchmarks across music, motion, and text modalities. Quantitative results are available in the \href{https://hanyangclarence.github.io/unimumo_demo/}{project page}.
GCDance: Genre-Controlled Music-Driven 3D Full Body Dance Generation
School of Computer Science and Electronic Engineering, University of Surrey（萨里大学计算机科学与电子工程学院）；Department of Music and Media, University of Surrey（萨里大学音乐与媒体系）；Department of Informatics, University of Oslo（奥斯陆大学信息学系）；Centre for Vision, Speech and Signal Processing, University of Surrey（萨里大学视觉、语音与信号处理中心）；School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）
Art2Mus: Artwork-to-Music Generation via Visual Conditioning and Large-Scale Cross-Modal Alignment
University of Bari Aldo Moro（巴里大学阿尔多·莫罗大学）；Catalonia's Telecommunications Technology Centre（加泰罗尼亚电信技术中心）；University of Picardie Jules Verne（皮卡第大学朱利·瓦内大学）
Microsoft Research Asia（微软亚洲研究院）；Dalian University of Technology（大连理工大学）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）；Nanjing University（南京大学）
MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies
LAION；University of California San Diego（加州大学圣迭戈分校）；Mila, Quebec Artificial Intelligence Institute（Mila魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；University of Surrey（萨里大学）
Multimodal Music Generation with Explicit Bridges and Retrieval Augmentation
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；University of Edinburgh（爱丁堡大学）；Meitu Inc（美图公司）；The Chinese University of Hong Kong（香港中文大学）；Beihang University（北京航空航天大学）
MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation
College of Artificial Intelligence, Zhejiang University（浙江大学人工智能学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Innovation Center of Yangtze River Delta, Zhejiang University（浙江大学长三角创新中心）；The Chinese University of Hong Kong（香港中文大学）；Shandong University（山东大学）；Chu Kochen Honors College, Zhejiang University（浙江大学竺可桢学院）；Ant Group（蚂蚁集团）
MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response
University of Pennsylvania（宾夕法尼亚大学）；Queen Mary University of London（伦敦大学玛丽皇后学院）；Carnegie Mellon University（卡内基梅隆大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Waterloo（滑铁卢大学）
MusiLingo 通过单一投影层将冻结音乐音频模型 MERT 与冻结 LLM 对齐，并构建 MusicInstruct 数据集，实现音乐描述生成与音乐查询响应的性能提升。
Dance-to-Music Generation with Encoder-based Textual Inversion
Institute of Automation, CAS（中国科学院自动化研究所）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；University of Chinese Academy of Sciences（中国科学院大学）；Kuaishou Technology（快手科技）；University of Konstanz（康斯坦茨大学）；National Cheng-Kung University（成功大学）
