弥合差距：Speech-LLM与端到端架构在多语言对话ASR中的比较探索
Bridging the gap: A comparative exploration of Speech-LLM and end-to-end architecture for multilingual conversational ASR
AI总结：
本文提出一种结合微调Whisper和mHuBERT编码器与LLM的ASR框架，通过交叉注意力机制提升多语言对话ASR性能，实验表明其在性能上接近端到端模型，但仍有提升空间。
摘要：
INTERSPEECH 2025挑战赛上的多语言对话语音语言模型（MLC-SLM）促进了利用大型语言模型（LLMs）的多语言对话ASR。我们先前的SHNU-mASR系统采用了一种竞争性的并行语音编码器架构，整合了Whisper和mHuBERT与一个LLM。然而，它面临两个挑战：简单的特征拼接可能无法充分利用互补信息，且LLM基于的ASR与端到端（E2E）编码器-解码器ASR之间的性能差距仍未被探索。在本工作中，我们提出了一种增强的LLM基于的ASR框架，将微调的Whisper和mHuBERT编码器与LLM结合以丰富语音表示。我们首先在MLC-SLM ASR任务上评估了带有LoRA和全微调的E2E Whisper模型，然后提出了基于交叉注意力的融合机制用于并行语音编码器。在MLC-SLM挑战赛的官方评估集上，我们的系统实现了10.69%的CER/WER，排名与Track 1系统相当，尽管其仅使用了1,500小时的基线训练数据，与他们的大规模训练集相比。然而，我们发现我们的最终LLM基于的ASR仍然无法与微调的E2E Whisper模型相媲美，为未来Speech-LLM设计提供了宝贵的实证指导。我们的代码可在https://github.com/1535176727/MLC-SLM上公开获取。
英文摘要：
The INTERSPEECH 2025 Challenge on Multilingual Conversational Speech Language Models (MLC-SLM) promotes multilingual conversational ASR with large language models (LLMs). Our previous SHNU-mASR system adopted a competitive parallel-speech-encoder architecture that integrated Whisper and mHuBERT with an LLM. However, it faced two challenges: simple feature concatenation may not fully exploit complementary information, and the performance gap between LLM-based ASR and end-to-end(E2E) encoder-decoder ASR remained unexplored. In this work, we present an enhanced LLM-based ASR framework that combines fine-tuned Whisper and mHuBERT encoders with an LLM to enrich speech representations. We first evaluate E2E Whisper models with LoRA and full fine-tuning on the MLC-SLM ASR task, and then propose cross-attention-based fusion mechanisms for the parallel-speech-encoder. On the official evaluation set of the MLC-SLM Challenge, our system achieves a CER/WER of 10.69%, ranking on par with the top-ranked Track 1 systems, even though it uses only 1,500 hours of baseline training data compared with their large-scale training sets. Nonetheless, we find that our final LLM-based ASR still does not match the performance of a fine-tuned E2E Whisper model, providing valuable empirical guidance for future Speech-LLM design. Our code is publicly available at https://github.com/1535176727/MLC-SLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
