TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation
Department of Computer Science and Technology, Institute for AI, BNRist, Tsinghua University(计算机科学与技术系、人工智能研究院、BNRist、清华大学);Tsinghua Laboratory of Brain and Intelligence (THBI), IDG/McGovern Institute for Brain Research, Tsinghua University(脑智能实验室(THBI)、IDG/麦克戈文脑科学研究院、清华大学);Chinese Institute for Brain Research (CIBR), Beijing(中国脑科学研究院)
In recent years, much speech separation research has focused primarily on improving model performance. However, for low-latency speech processing systems, high efficiency is equally important. Therefore, we propose a speech separation model with significantly reduced parameters and computational costs: Time-frequency Interleaved Gain Extraction and Reconstruction network (TIGER). TIGER leverages prior knowledge to divide frequency bands and compresses frequency information. We employ a multi-scale selective attention module to extract contextual features while introducing a full-frequency-frame attention module to capture both temporal and frequency contextual information. Additionally, to more realistically evaluate the performance of speech separation models in complex acoustic environments, we introduce a dataset called EchoSet. This dataset includes noise and more realistic reverberation (e.g., considering object occlusions and material properties), with speech from two speakers overlapping at random proportions. Experimental results showed that models trained on EchoSet had better generalization ability than those trained on other datasets compared to the data collected in the physical world, which validated the practical value of the EchoSet. On EchoSet and real-world data, TIGER significantly reduces the number of parameters by 94.3% and the MACs by 95.3% while achieving performance surpassing the state-of-the-art (SOTA) model TF-GridNet.
Gelina: Unified Speech and Gesture Synthesis via Interleaved Token Prediction
1 ISIR, Sorbonne Universit\'e, Paris, France 2 STMS Lab -- IRCAM, Sorbonne Universit\'e, Paris, France 3 CNRS, France 4 Department of Speech, Music;Hearing, KTH Royal Institute of Technology, Stockholm, Sweden
Transformer在自动语音识别(ASR)中实现了与最先进的端到端模型竞争的性能,并且比基于RNN的模型需要更少的训练时间。原始的Transformer采用编码器-解码器架构,仅适用于离线ASR。它依赖于注意力机制来学习对齐,并对输入音频进行双向编码。Transformer解码的高计算成本也限制了其在生产流传输系统中的使用。为了使Transformer适用于流式ASR,我们探索了Transducer框架作为学习对齐的可流式方式。对于音频编码,我们应用具有交错卷积层的单向Transformer。交错的卷积层用于对未来的上下文进行建模,这对性能很重要。为了降低计算成本,我们逐渐降低声音输入的采样,也使用交织卷积层。此外,我们限制了自注意中的历史上下文的长度,以保持每个解码步骤的计算成本恒定。我们表明,这种架构,名为Conv-Transformer Transducer,在LibriSpeech数据集上实现了有竞争力的性能(3.6\% WER on test-clean),而无需外部语言模型。该性能与先前发布的可流式传输的Transformer Transducer和强混合流式ASR系统相当,并且以更小的前瞻窗口(140 ms)、更少的参数和更低的帧速率实现。
CleanMel: Mel-Spectrogram Enhancement for Improving Both Speech Quality and ASR
Zhejiang University(浙江大学);Westlake University(西湖大学);School of Engineering(工程学院);Institute of Advanced Technology(先进技术研究院);Westlake Institute for Advanced Study(西湖研究学院)