arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.05373cs.SD
收藏

Speech-XL: 向大语言模型中的长形式语音理解迈进

Speech-XL: Towards Long-Form Speech Understanding in Large Speech Language Models

TMCC, College of Computer Science, Nankai University, Tianjin, China(TMCC,计算机科学学院,南开大学,天津,中国);Alibaba International Digital Commerce(阿里巴巴国际数字商业);University of Exeter, Exeter, United Kingdom(埃克塞特大学,埃克塞特,英国)

Haoqin Sun, Chenyang Lyu, Shiwan Zhao, Xuanfan Ni, Xiangyu Kong, Longyue Wang, Weihua Luo, Yong Qin

AI总结:

Speech-XL通过引入语音摘要标记和课程学习策略,实现长形式语音压缩,提升大语言模型在长音频理解中的性能。

摘要:

尽管大型语音语言模型(LSLMs)在处理短期音频信号方面取得了显著成功,但将其扩展到长形式音频理解却受到严重限制。这种限制源于上下文长度有限以及长形式推理所需的庞大内存占用。在本工作中,我们提出了Speech-XL,一个新的模型,利用大型语言模型(LLMs)内在的关键值(KV)稀疏化能力,实现高比率的语音输入压缩。具体来说,我们为每个语音区间引入了一个新的特殊标记,即语音摘要标记(SST),用于将区间内的语音信息封装到其关联的KV对中。SST模块通过指令微调进行训练,采用课程学习策略,使SST以渐进的方式学习压缩信息——从低比率(简单)到高比率(挑战性)压缩。尽管使用比其他基线少得多的训练数据,我们的模型在主要基准上实现了具有竞争力的性能,包括LongSpeech和AUDIOMARATHON。通过解决长形式音频建模中的长期瓶颈,我们的方法为大规模音频序列的压缩提供了新的视角。

英文摘要:

Despite the growing success of Large Speech Language Models (LSLMs) in processing short-term acoustic signals, their extension to long-form audio understanding is severely bottlenecked. This limitation stems from the limited context length and the exorbitant memory footprints required for long-form inference. In this work, we propose Speech-XL, a new model that capitalizes on the intrinsic key-value (KV) sparsification capacity of Large Language Models (LLMs) to achieve high-ratio speech input compression. Specifically, we introduce a novel special token, the Speech Summarization Token (SST), for each speech interval to encapsulate the intra-interval speech information into its associated KV pairs. The SST module is trained via instruction fine-tuning, employing a curriculum learning strategy where the SST learns to compress information in a progressive manner--advancing from low-ratio (simple) to high-ratio (challenging) compression. Despite utilizing significantly less training data than other baselines, our model achieves highly competitive performance on major benchmarks, including LongSpeech and AUDIOMARATHON. By addressing the long-standing bottlenecks in long-form audio modeling, our approach offers a novel perspective on the condensation of extensive acoustic sequences.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑