通过上下文化稀有和歧义词增强基于大型语言模型的语音识别
Enhancing Large Language Model-based Speech Recognition by Contextualization for Rare and Ambiguous Words
AI总结：
本文提出一种基于LLM的ASR系统，通过在文本提示中提供关键词作为先验信息进行上下文化，利用Whisper编码器和PLaMo-100B解码器，在不修改架构的情况下显著提升了稀有和歧义词的识别性能。
摘要：
我们开发了一种基于大型语言模型（LLM）的自动语音识别（ASR）系统，该系统可以通过在文本提示中提供关键词作为先验信息来进行上下文化。我们采用纯解码器架构，并使用我们的内部LLM，PLaMo-100B，作为解码器，该模型是从头开始使用以日语和英语文本为主的数据集进行预训练的。我们采用预训练的Whisper编码器作为音频编码器，来自音频编码器的音频嵌入通过适配器层投影到文本嵌入空间，并与从文本提示转换的文本嵌入连接，以形成解码器的输入。通过在文本提示中提供关键词作为先验信息，我们可以在不修改模型架构的情况下，对我们的基于LLM的ASR系统进行上下文化，以准确转录输入音频中的歧义词。实验结果表明，向解码器提供关键词可以显著提高稀有和歧义词的识别性能。
英文摘要：
We develop a large language model (LLM) based automatic speech recognition (ASR) system that can be contextualized by providing keywords as prior information in text prompts. We adopt decoder-only architecture and use our in-house LLM, PLaMo-100B, pre-trained from scratch using datasets dominated by Japanese and English texts as the decoder. We adopt a pre-trained Whisper encoder as an audio encoder, and the audio embeddings from the audio encoder are projected to the text embedding space by an adapter layer and concatenated with text embeddings converted from text prompts to form inputs to the decoder. By providing keywords as prior information in the text prompts, we can contextualize our LLM-based ASR system without modifying the model architecture to transcribe ambiguous words in the input audio accurately. Experimental results demonstrate that providing keywords to the decoder can significantly improve the recognition performance of rare and ambiguous words.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
