arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.13632cs.CLcs.AIeess.AS
收藏

弥合语言模型中文本与语音理解之间的差距

Closing the Gap Between Text and Speech Understanding in LLMs

Université de Toulon, Aix Marseille Université, CNRS, LIS(法国图卢兹大学、马赛大学、CNRS、LIS)

Santiago Cuervo, Skyler Seto, Maureen de Seyssel, Richard He Bai, Zijin Gu, Tatiana Likhomanenko, Navdeep Jaitly, Zakaria Aldeneh

AI总结:

SALAD通过主动选择和跨模态蒸馏提高语音与文本对齐,以更高效的方式弥合LLMs在语音理解上的性能差距。

摘要:

大型语言模型(LLMs)可以被调整以扩展其文本能力到语音输入。然而,这些语音适应的LLMs在语言理解任务上始终不如其文本基于的对应物——甚至级联流程——表现。我们称这种不足为文本-语音理解差距:当语音适应的LLM处理语音输入时相对于原始文本基于的LLM处理等效文本时观察到的性能下降。最近缩小这一差距的方法要么依赖于大规模文本语料的语音合成,这成本高昂且严重依赖合成数据,要么依赖于大规模专有语音数据集,这些数据集不可复现。因此,仍然需要更数据高效的替代方法来弥合文本-语音理解差距。在本工作中,我们分析了差距是由两个因素驱动的:(i)在适应过程中文本能力的遗忘,以及(ii)语音和文本之间的跨模态对齐问题。基于此分析,我们引入了SALAD——通过主动选择和跨模态蒸馏进行的学习对齐——它结合了跨模态蒸馏和针对性合成数据以提高对齐并减轻遗忘。应用于3B和7B LLMs,SALAD在广泛领域基准测试中与强大的开放式权重模型表现相当,在知识、语言理解和推理方面,尽管仅使用公共语料库中数量级多的语音数据进行训练。

英文摘要:

Large Language Models (LLMs) can be adapted to extend their text capabilities to speech inputs. However, these speech-adapted LLMs consistently underperform their text-based counterparts--and even cascaded pipelines--on language understanding tasks. We term this shortfall the text-speech understanding gap: the performance drop observed when a speech-adapted LLM processes spoken inputs relative to when the original text-based LLM processes the equivalent text. Recent approaches to narrowing this gap either rely on large-scale speech synthesis of text corpora, which is costly and heavily dependent on synthetic data, or on large-scale proprietary speech datasets, which are not reproducible. As a result, there remains a need for more data-efficient alternatives for closing the text-speech understanding gap. In this work, we analyze the gap as driven by two factors: (i) forgetting of text capabilities during adaptation, and (ii) cross-modal misalignment between speech and text. Based on this analysis, we introduce SALAD--Sample-efficient Alignment with Learning through Active selection and cross-modal Distillation--which combines cross-modal distillation with targeted synthetic data to improve alignment while mitigating forgetting. Applied to 3B and 7B LLMs, SALAD achieves competitive performance with a strong open-weight model across broad-domain benchmarks in knowledge, language understanding, and reasoning, while training on over an order of magnitude less speech data from public corpora.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑