In this work, we study different approaches to utilize large language models (LLMs) for automatic speech recognition (ASR). Specifically, we compare the tight integration of an acoustic model (AM) with the LLM ("speech LLM") to the traditional way of combining AM and LLM via shallow fusion and provide ablations on the effect of different label units and LLM sizes. For tight integration, we further examine the effect of attention interfaces, encoder downsampling, and length normalization. Furthermore, we investigate joint recognition with a CTC model to mitigate hallucinations of speech LLMs and present effective optimizations. We train and evaluate on LibriSpeech and Loquacious and additionally evaluate on the HuggingFace ASR leaderboard. Across model sizes, we find that shallow fusion consistently outperforms tight integration of AM and LLM on in-domain data, highlighting the importance of strong shallow-fusion baselines when evaluating speech LLMs for ASR. On the more heterogeneous HuggingFace ASR leaderboard, however, the integrated prefix LLM achieves lower average WER than shallow fusion, with gains concentrated on out-of-domain corpora.
A Survey on Speech Large Language Models for Understanding
X-LANCE Lab, School of Computer Science, MoE Key Lab of Artificial Intelligence Shanghai Jiao Tong University(上海交通大学计算机学院人工智能教育部重点实验室);Jiangsu Key Lab of Language Computing(江苏省语言计算重点实验室);AISpeech Co., Ltd.(思必驰科技股份有限公司);Huazhong University of Science and Technology(华中科技大学);School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)
AdaLTM: Adaptive Layer-wise Task Vector Merging for Categorical Speech Emotion Recognition with ASR Knowledge Integration
National Tsing Hua University(国立清华大学);Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California(南加州大学信号分析与解释实验室);Graduate Institute of Communication Engineering, National Taiwan University(国立台湾大学通信工程研究所);University of Edinburgh(爱丁堡大学)
ESPnet-se: end-to-end speech enhancement and separation toolkit designed for asr integration
Shanghai Jiao Tong University(上海交通大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Johns Hopkins University(约翰斯·霍普金斯大学);Facebook AI(脸书人工智能);Nagoya University(名古屋大学);Human Dataware Lab. Co., Ltd.(人类数据仓库实验室有限公司);Paderborn University(帕德博恩大学);Microsoft Research(微软研究院)
End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representation
Carnegie Mellon University(卡内基梅隆大学);Yahoo Japan Corporation(雅虎日本公司)
本文提出了一种端到端(E2 E)的自动语音识别(ASR)模型,该模型以鲁棒的语音识别为目标,称为增强语音输入的集成语音识别自监督学习表示(IRIS)。与传统的E2 E ASR模型相比,该模型集成了语音增强(SE)和自监督学习表示(SSLR)两个重要模块。SE模块增强带噪语音。然后,SSLR模块从增强的语音中提取用于语音识别(ASR)的特征。为了对模型进行训练,我们建立了一个有效的学习方案。单声道CHiME-4任务的评估结果表明,IRIS模型在单声道CHiME-4基准测试中实现了文献报道的最佳性能(实际开发为2.0%,实际测试为3.9%),这要归功于强大的预训练SSLR模块和微调的SE模块。
VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs
Department of Computer Science, University of Sheffield, United Kingdom(英国谢菲尔德大学计算机科学系);Signal Analysis and Interpretation Laboratory (SAIL), Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA(美国南加州大学电气与计算机工程系信号分析与解释实验室(SAIL))
Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuning
Huazhong University of Science and Technology, School of Electronic Information and Communications(华中科技大学电子信息与通信学院);MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China(人工智能MoE重点实验室、AI研究院、X-LANCE实验室、上海交通大学、上海,中国);Jiangsu Key Lab of Language Computing, Suzhou, China(江苏省语言计算重点实验室、苏州,中国);AISpeech Co., Ltd., Suzhou, China(AISpeech公司、苏州,中国)
A Study of Joint Effect on Denoising Techniques and Visual Cues to Improve Speech Intelligibility in Cochlear Implant Simulation
Research Center for Information Technology Innovation, Academia Sinica(中央研究院资讯科技创新研究中心);National Taiwan University(台湾大学);Institute of Linguistics, Academia Sinica(中央研究院语言学研究所)
SOVA-Bench: Benchmarking the Speech Conversation Ability for LLM-based Voice Assistant
Shanghai Jiao Tong University(上海交通大学);Ant Group(蚂蚁集团);Wuhan University(武汉大学);School of Artificial Intelligence(人工智能学院);School of Cyber Science and Engineering(网络科学与工程学院)