arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.03310eess.AS
收藏

TASU:用于语音理解的纯文本对齐

TASU: Text-Only Alignment for Speech Understanding

Shanghai Jiao Tong University(上海交通大学);Huazhong University of Science and Technology(华中科技大学);Soochow University(苏州大学);AISpeech Ltd(思必驰科技股份有限公司)

Jing Peng, Yi Yang, Xu Li, Yu Xi, Quanwei Tang, Yangui Fang, Junjie Li, Kai Yu

AI总结:

TASU通过纯文本数据实现语音理解的跨模态对齐,提升了零样本语音识别性能,并在多个基准测试中优于现有模型。

摘要:

近年来,语音大语言模型(Speech LLMs)的进展为跨多样语音理解任务的统一架构铺平了道路。然而,现有的对齐范式严重依赖大规模的音频-文本配对数据和计算密集型训练,但往往在未见过的领域或任务上泛化能力有限。为了解决这些限制,我们提出了TASU(用于语音理解的纯文本对齐),这是一种新的对齐范式,可以仅利用未配对的文本数据来指导跨模态对齐。实验表明,TASU在零样本语音识别中实现了具有竞争力的性能。利用这一特性,它可以进一步作为课程学习中的预训练阶段,增强语音识别中的领域泛化能力。最终,TASU可以将零样本泛化能力扩展到广泛的声音理解任务,并在MMSU基准上显著优于GLM-4-Voice和Step-Audio等主流语音大语言模型,使TASU成为语音大语言模型的一种高效且可扩展的对齐范式。

英文摘要:

Recent advances in Speech Large Language Models (Speech LLMs) have paved the way for unified architectures across diverse speech understanding tasks. However, prevailing alignment paradigms rely heavily on large-scale audio-text paired data and computationally intensive training, yet often exhibit limited generalization to unseen domains or tasks. To address these limitations, we propose TASU (Text-only Alignment for Speech Understanding), a novel alignment paradigm that can leverage only unpaired text data to guide cross-modal alignment. Experiments show that TASU achieves competitive zero-shot speech recognition. Leveraging this property, it can further function as a pre-training stage in curriculum learning, enhancing domain generalization in speech recognition. Ultimately, TASU can extend its zero-shot generalization to a wide range of speech understanding tasks and notably outperforms prominent Speech LLMs including GLM-4-Voice and Step-Audio on the MMSU benchmark, establishing TASU as an efficient and scalable alignment paradigm for Speech LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑