This paper presents the crossing scheme (X-scheme) for improving the performance of deep neural network (DNN)-based music source separation (MSS) with almost no increasing calculation cost. It consists of three components: (i) multi-domain loss (MDL), (ii) bridging operation, which couples the individual instrument networks, and (iii) combination loss (CL). MDL enables the taking advantage of the frequency- and time-domain representations of audio signals. We modify the target network, i.e., the network architecture of the original DNN-based MSS, by adding bridging paths for each output instrument to share their information. MDL is then applied to the combinations of the output sources as well as each independent source; hence, we called it CL. MDL and CL can easily be applied to many DNN-based separation methods as they are merely loss functions that are only used during training and do not affect the inference step. Bridging operation does not increase the number of learnable parameters in the network. Experimental results showed that the validity of Open-Unmix (UMX), densely connected dilated DenseNet (D3Net) and convolutional time-domain audio separation network (Conv-TasNet) extended with our X-scheme, respectively called X-UMX, X-D3Net and X-Conv-TasNet, by comparing them with their original versions. We also verified the effectiveness of X-scheme in a large-scale data regime, showing its generality with respect to data size. X-UMX Large (X-UMXL), which was trained on large-scale internal data and used in our experiments, is newly available at https://github.com/asteroid-team/asteroid/tree/master/egs/musdb18/X-UMX.
Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generation
The Hong Kong University of Science and Technology(香港科技大学);Multimodal Art Projection(多模态艺术投影);Shenzhen University(深圳大学);Queen Mary University of London(伦敦玛丽女王大学)
Sony Group Corporation(索尼集团公司);Sony Europe B.V.(索尼欧洲公司);Facebook AI Research Paris(巴黎Facebook人工智能研究院);Korea University(高丽大学);Singapore University of Technology and Design(新加坡科技设计大学)
MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response
University of Pennsylvania(宾夕法尼亚大学);Queen Mary University of London(伦敦大学玛丽皇后学院);Carnegie Mellon University(卡内基梅隆大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Waterloo(滑铁卢大学)
MusiLingo 通过单一投影层将冻结音乐音频模型 MERT 与冻结 LLM 对齐,并构建 MusicInstruct 数据集,实现音乐描述生成与音乐查询响应的性能提升。
Dissecting Performance Degradation in Audio Source Separation under Sampling Frequency Mismatch
The University of Tokyo, Japan(东京大学);The National Institute of Advanced Industrial Science and Technology (AIST), Japan(日本国家先进工业科学与技术研究院);Tokyo University of Agriculture and Technology, Japan(东京农工大学)