Rethinking Flow and Diffusion Bridge Models for Speech Enhancement
Key Laboratory of Modern Acoustics, Institute of Acoustics, Nanjing University(南京大学声学研究所现代声学重点实验室);NJU-Horizon Intelligent Audio Lab, Horizon Robotics(地平线智能音频实验室);Tencent AI Lab(腾讯AI实验室)
Dahan Wang, Jun Gao, Tong Lei, Yuxiang Hu, Changbao Zhu, Kai Chen, Jing Lu
Flow matching and diffusion bridge models have emerged as leading paradigms in generative speech enhancement, modeling stochastic processes between paired noisy and clean speech signals based on principles such as flow matching, score matching, and Schrödinger bridge. In this paper, we present a framework that unifies existing flow and diffusion bridge models by interpreting them as constructions of Gaussian probability paths with varying means and variances between paired data. Furthermore, we investigate the underlying consistency between the training/inference procedures of these generative models and conventional predictive models. Our analysis reveals that each sampling step of a well-trained flow or diffusion bridge model optimized with a data prediction loss is theoretically analogous to executing predictive speech enhancement. Motivated by this insight, we introduce an enhanced bridge model that integrates an effective probability path design with key elements from predictive paradigms, including improved network architecture, tailored loss functions, and optimized training strategies. Experiments on denoising and dereverberation tasks demonstrate that the proposed method outperforms existing flow and diffusion baselines with fewer parameters and reduced computational complexity. The results also highlight that the inherently predictive nature of this generative framework imposes limitations on its achievable upper-bound performance.
Rethinking Speech Representation Aggregation in Speech Enhancement: A Phonetic Mutual Information Perspective
Department of Intelligence and Information, Seoul National University, Republic of Korea(智能与信息系,首尔国立大学,韩国);Artificial Intelligence Institute, Seoul National University, Republic of Korea(人工智能研究所,首尔国立大学,韩国)
Rethinking Speaker Embeddings for Speech Generation: Sub-Center Modeling for Capturing Intra-Speaker Diversity
Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA(语言与语音处理中心(CLSP),约翰霍普金斯大学,美国);Center for Robust Speech Systems (CRSS), The University of Texas at Dallas, USA(稳健语音系统中心(CRSS),德克萨斯大学达拉斯分校,美国);Language Technologies Institute (LTI), Carnegie Mellon University, USA(语言技术研究所(LTI),卡内基梅隆大学,美国)
Is Word Error Rate Enough? Rethinking Privacy Evaluation in Speech with Entity-Aware Metrics
Fraunhofer Institute for Integrated Circuits (IIS)(弗劳恩霍夫集成电路研究所 (IIS));Jade University of Applied Sciences(亚德应用科学大学);Carl von Ossietzky Universität Oldenburg(奥尔登堡卡尔·冯·奥西茨基大学)
Rethinking Speech Recognition with A Multimodal Perspective via Acoustic and Semantic Cooperative Decoding
School of Computer and Communication Engineering, University of Science and Technology Beijing(北京科技大学计算机与通信工程学院);USTB-EEasyTech Joint Lab of Artificial Intelligence, University of Science and Technology Beijing(北京科技大学USTB-易智瑞人工智能联合实验室);EEasy Technology Company Ltd.(易智科技有限公司)
M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation
Monash University(莫纳什大学)
端到端的语音到文本翻译模型通常使用预训练的语音编码器和预训练的文本解码器进行初始化。这导致预训练和微调之间的显著训练间隙,这主要是由于来自编码器的语音输出和到解码器的文本输入之间的模态差异。在这项工作中,我们的目标是弥合语音和文本之间的模态差距,以提高翻译质量。我们提出了M-Adapter,一个新的基于transformer的模块,以适应语音表示文本。在缩小语音序列的同时,M-Adapter通过对语音序列的全局和局部依赖关系进行建模,产生语音到文本翻译所需的特征。我们的实验结果表明,我们的模型在Must-C En $\rightarrow$ DE数据集上的性能优于强基线,最高可达1个BLEU分数。\footnote{Our code is available at https://github.com/mingzi151/w2v2-st.}