Speech-XL: Towards Long-Form Speech Understanding in Large Speech Language Models
TMCC, College of Computer Science, Nankai University, Tianjin, China(TMCC,计算机科学学院,南开大学,天津,中国);Alibaba International Digital Commerce(阿里巴巴国际数字商业);University of Exeter, Exeter, United Kingdom(埃克塞特大学,埃克塞特,英国)
Despite the growing success of Large Speech Language Models (LSLMs) in processing short-term acoustic signals, their extension to long-form audio understanding is severely bottlenecked. This limitation stems from the limited context length and the exorbitant memory footprints required for long-form inference. In this work, we propose Speech-XL, a new model that capitalizes on the intrinsic key-value (KV) sparsification capacity of Large Language Models (LLMs) to achieve high-ratio speech input compression. Specifically, we introduce a novel special token, the Speech Summarization Token (SST), for each speech interval to encapsulate the intra-interval speech information into its associated KV pairs. The SST module is trained via instruction fine-tuning, employing a curriculum learning strategy where the SST learns to compress information in a progressive manner--advancing from low-ratio (simple) to high-ratio (challenging) compression. Despite utilizing significantly less training data than other baselines, our model achieves highly competitive performance on major benchmarks, including LongSpeech and AUDIOMARATHON. By addressing the long-standing bottlenecks in long-form audio modeling, our approach offers a novel perspective on the condensation of extensive acoustic sequences.
LongSpeech: A Scalable Benchmark for Transcription, Translation and Understanding in Long Speech
Alibaba International Digital Commerce(阿里巴巴国际数字商务);Shanghai Jiao Tong University(上海交通大学);Delft University of Technology(代尔夫特理工大学);Linköping University(林肯大学);University of Groningen(格罗宁根大学)
ChildVox: A Speech, Audio, and Large Audio-Language Model Benchmark in Understanding and Characterizing Sound across Childhood
University of Southern California(南加州大学);The Ohio State University(俄亥俄州立大学);University of California, Los Angeles(加州大学洛杉矶分校);Harvard University(哈佛大学);Boston University(波士顿大学);University of Miami(迈阿密大学)
Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models
The University of Tokyo(东京大学);Keio University(.keio大学);Graduate School of Information Science and Technology(信息科学与技术研究生院);Faculty of Science and Technology(科学与技术学部)
Unlocking Foundation Models for Privacy-Enhancing Speech Understanding: An Early Study on Low Resource Speech Training Leveraging Label-guided Synthetic Speech Content
The NVVSpeech Challenge 2026: Overview and Lessons for Speech Understanding and Generation with Nonverbal Vocalizations
School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院);School of Computer Science, Northwestern Polytechnical University(西北工业大学计算机学院);Shenzhen Pimei Technology(深圳湃美科技);National Taiwan University(台湾大学);Tongji University(同济大学);Hong Kong University of Science and Technology(香港科技大学);Yutuzhineng(语知能);AISHELL;State Key Laboratory of Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室)
Optimizing Alignment of Speech and Language Latent Spaces for End-to-End Speech Recognition and Understanding
Microsoft(微软)
基于注意力的编码器-解码器(AED)网络的发展为端到端(E2 E)自动语音识别(ASR)带来了巨大的进步。进一步提高基于AED的E2 E ASR性能的一种方法是引入额外的文本编码器,以利用大量的文本数据,从而捕获更多的上下文感知语言信息。然而,由于用于建模的不同单元,这种方法带来了语音编码器和文本编码器之间的失配问题。在本文中,我们提出了一种嵌入对齐器和模态切换训练,以更好地对齐语音和文本潜在空间。嵌入对齐器是文本编码器和语音编码器之间的共享线性投影,分别由掩蔽语言建模(MLM)损失和连接主义时间分类(CTC)训练。模态切换训练基于强制对齐结果随机交换语音和文本嵌入,以学习联合表示空间。实验结果表明,我们提出的方法实现了相对14%至19%的字出错率(WER)减少Librispeech ASR任务。我们进一步验证了它对口语语言理解(SLU)的有效性,即,SNIPS插槽填充任务的F1分数绝对提高了2.5%至2.8%。
Joint Automatic Speech Recognition And Structure Learning For Better Speech Understanding
School of Cyber Science and Engineering, Wuhan University(武汉大学网络空间安全学院);School of Computer Science, Wuhan University(武汉大学计算机学院);Wuhan Second Ship Design and Research Institute(武汉第二船舶设计研究所);National Institute of Information and Communications Technology, Japan(日本信息通信研究机构)
Speech-language Pre-training for End-to-end Spoken Language Understanding
Microsoft(微软)
端到端(E2 E)口语语言理解(SLU)可以直接从语音信号中推断语义,而无需将自动语音识别器(ASR)与自然语言理解(NLU)模块级联。然而,成对的话语记录和对应的语义可能并不总是可用的或不足以在真实的生产环境中训练E2 E SLU模型。在本文中,我们提出将优化的E2 E ASR编码器(语音)和预训练的语言模型编码器(语言)统一到Transformer解码器中。通过使用条件掩蔽语言模型(MLM)目标,统一的语音-语言预训练模型(SLP)在来自目标域的有限标记数据上不断增强,并且因此可以有效地生成用于推理中的给定输入语音的意图、槽类型和槽值的序列。在两个公开语料库上的实验结果表明,该方法在E2 E SLU上的性能优于传统的级联方法。它还优于目前最先进的E2 E SLU方法,配对数据少得多。
Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study
Carnegie Mellon University(卡内基梅隆大学);Yahoo Japan Corporation(雅虎日本公司);Northwestern Polytechnical University(西北工业大学);University of Washington(华盛顿大学);University of Stuttgart(斯图加特大学);Waseda University(早稻田大学);National Taiwan University(国立台湾大学)