Deep Neural Network-based source separation methods usually train independent models to optimize for the separation of individual sources. Although this can lead to good performance for well-defined targets, it can also be computationally expensive. The multitask alternative of a single network jointly optimizing for all targets simultaneously usually requires the availability of all target sources for each input. This requirement hampers the ability to create large training databases. In this paper, we present a model that decomposes the learnable parameters into a shared parametric model (encoder) and independent components (decoders) specific to each source. We propose an interleaved training procedure that optimizes the sub-task decoders independently and thus does not require each sample to possess a ground truth for all of its composing sources. Experimental results on MUSDB18 with the proposed method show comparable performance to independently trained models, with less trainable parameters, more efficient inference, and an encoder transferable to future target objectives. The results also show that using the proposed interleaved training procedure leads to better Source-to-Interference energy ratios when compared to the simultaneous optimization of all training objectives, even when all composing sources are available.
While deep learning has been incredibly successful in modeling tasks with large, carefully curated labeled datasets, its application to problems with limited labeled data remains a challenge. The aim of the present work is to improve the label efficiency of large neural networks operating on audio data through a combination of multitask learning and self-supervised learning on unlabeled data. We trained an end-to-end audio feature extractor based on WaveNet that feeds into simple, yet versatile task-specific neural networks. We describe several easily implemented self-supervised learning tasks…
Deep learning models trained on audio-visual data have been successfully used to achieve state-of-the-art performance for emotion recognition. In particular, models trained with multitask learning have shown additional performance improvements. However, such multitask models entangle information between the tasks, encoding the mutual dependencies present in label distributions in the real world data used for training. This work explores the disentanglement of multimodal signal representations for the primary task of emotion recognition and a secondary person identification task. In particular,…
Echo: Towards Advanced Audio Comprehension via Audio-Interleaved Reasoning
IIE, Chinese Academy of Sciences（中国科学院信息工程研究所）；Zhejiang University（浙江大学）；ByteDance China（字节跳动中国）；VCIP & TMCC & DISSec, College of Computer Science, Nankai University（VCIP与TMCC与DISSec，南开大学计算机学院）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Department of Psychological and Cognitive Sciences, Tsinghua University（清华大学心理学与认知科学系）；University of Chinese Academy of Sciences（中国科学院大学）
Although several research works have been reported on audio-visual sound source localization in unconstrained videos, no datasets and metrics have been proposed in the literature to quantitatively evaluate its performance. Defining the ground truth for sound source localization is difficult, because the location where the sound is produced is not limited to the range of the source object, but the vibrations propagate and spread through the surrounding objects. Therefore we propose a new concept, Sounding Object, to reduce the ambiguity of the visual location of sound, making it possible to ann…
An Evaluation of Interleaved Instruction Tuning on Semantic Reasoning Performance in an Audio MLLM
The Graduate Center, CUNY（CUNY研究生中心）；Brooklyn College, CUNY（CUNY布鲁克林学院）；Borough of Manhattan Community College, CUNY（CUNY曼哈顿社区学院）；The City College of New York, CUNY（CUNY纽约城市学院）
MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoders
Institute for Infocomm Research (I 2 2 {}^{\text{2}} start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT R)（信息与通信研究 institute）；Agency for Science, Technology and Research (A*STAR)（科技研究局）；Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR)（前沿人工智能研究 center）
Imperial College London（伦敦帝国学院）；Chinese University HongKong, Shenzhen, China（香港中文大学（深圳））；Huawei, Singapore（新加坡华为）；Technical University Munich, Munich, German（慕尼黑技术大学）
Self-supervised-learning-based pre-trained models for speech data, such as Wav2Vec 2.0 (W2V2), have become the backbone of many speech tasks. In this paper, to achieve speaker diarisation and speech recognition using a single model, a tandem multitask training (TMT) method is proposed to fine-tune W2V2. For speaker diarisation, the tasks of voice activity detection (VAD) and speaker classification (SC) are required, and connectionist temporal classification (CTC) is used for ASR. The multitask framework implements VAD, SC, and ASR using an early layer, middle layer, and late layer of W2V2, whi…
This paper presents a methodology for early detection of audio events from audio streams. Early detection is the ability to infer an ongoing event during its initial stage. The proposed system consists of a novel inference step coupled with dual parallel tailored-loss deep neural networks (DNNs). The DNNs share a similar architecture except for their loss functions, i.e. weighted loss and multitask loss, which are designed to efficiently cope with issues common to audio event detection. The inference step is newly introduced to make use of the network outputs for recognizing ongoing events. Th…
This paper proposes a Region-based Convolutional Recurrent Neural Network (R-CRNN) for audio event detection (AED). The proposed network is inspired by Faster-RCNN, a well known region-based convolutional network framework for visual object detection. Different from the original Faster-RCNN, a recurrent layer is added on top of the convolutional network to capture the long-term temporal context from the extracted high level features. While most of the previous works on AED generate predictions at frame level first, and then use post-processing to predict the onset/offset timestamps of events f…
UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information
International Digital Economy Academy（国际数字经济学院）；IDEA&Emdoor Collaborative Laboratory（IDEA与Emdoor协同实验室）；Emdoor Information Co., Ltd（Emdoor信息有限公司）；Shenzhen Yijiayiban Information Technology Co., Ltd（深圳艺佳iban信息技术有限公司）
The performance of music source separation (MSS) models has been greatly improved in recent years thanks to the development of novel neural network architectures and training pipelines. However, recent model designs for MSS were mainly motivated by other audio processing tasks or other research fields, while the intrinsic characteristics and patterns of the music signals were not fully discovered. In this paper, we propose band-split RNN (BSRNN), a frequency-domain model that explictly splits the spectrogram of the mixture into subbands and perform interleaved band-level and sequence-level mod…
