arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.09332eess.AS
收藏

音素与投影器:针对基于LLM的ASR的语音-语言接口研究

Phonemes vs. Projectors: An Investigation of Speech-Language Interfaces for LLM-based ASR

Tsinghua University(清华大学);TasiTech Co., Ltd.(塔思科技);Xinjiang University(新疆大学)

Ziwei Li, Lukuang Dong, Saierdaer Yusuyin, Xianyu Zhao, Zhijian Ou

AI总结:

本文比较了基于音素和投影器的接口在英语和塔塔尔语ASR中的性能,提出BPE-音素接口提升生成效果。

摘要:

将预训练语音编码器与大语言模型(LLM)结合在ASR中具有潜力,但性能和数据效率依赖于语音-语言接口。常见选择是学习投影器,将编码器特征映射到LLM嵌入空间,而另一种选择是将离散音素序列暴露给LLM。使用相同的编码器和LLM主干,我们在高资源英语和低资源塔塔尔语中比较了基于音素和普通投影器接口。我们还提出了一种BPE-音素接口,将频繁的局部音素模式分组,同时保留显式的词边界提示以生成音素到字母的转换。在LibriSpeech上,基于音素的接口与普通投影器相当,而BPE-音素接口进一步提升性能。在塔塔尔语上,基于音素的接口显著优于普通投影器。我们进一步发现,音素监督产生了一种音素指导的混合接口,其性能优于普通投影器。

英文摘要:

Integrating pretrained speech encoders with large language models (LLMs) is promising for ASR, but performance and data efficiency depend on the speech-language interface. A common choice is a learned projector that maps encoder features into the LLM embedding space, whereas an alternative is to expose discrete phoneme sequences to the LLM. Using the same encoder and LLM backbones, we compare phoneme-based and vanilla projector-based interfaces in high-resource English and low-resource Tatar. We also propose a BPE-phoneme interface that groups frequent local phoneme patterns while preserving explicit word-boundary cues for phoneme-to-grapheme generation. On LibriSpeech, the phoneme-based interface is competitive with the vanilla projector, and the BPE-phoneme interface yields further gains. On Tatar, the phoneme-based interface substantially outperforms the vanilla projector. We further find that phoneme supervision yields a phoneme-informed hybrid interface that is stronger than the vanilla projector.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑