利用非配对语音与文本提升 Whisper 对代表性不足语言哈萨克语的识别性能
Improving Whisper's Recognition Performance for Under-Represented Language Kazakh Leveraging Unpaired Speech and Text
AI总结：
本文针对 Whisper 在哈萨克语等低资源语言上表现不佳的问题，提出结合非配对语音与文本、GPT 语言模型、EOT 修正、幻觉惩罚及伪标注微调的方法，在多个实验中实现超过 10% 的绝对 WER 降低。
摘要：
Whisper 及其他大规模自动语音识别模型在性能上已取得显著进步，但它们在许多低资源语言（如哈萨克语）上的表现仍不理想。如何利用低成本数据提升 Whisper 在代表性不足语言上的性能值得研究。本研究利用易于获取的非配对语音和文本数据，并将语言模型 GPT 与 Whisper 在哈萨克语上结合。我们实现了转录结束（EOT）判断修正和幻觉惩罚，以提高语音识别性能。此外，我们采用解码平均 token 对数概率作为准则，从未标注语音数据中选择样本，并使用伪标注数据微调模型，以进一步提升性能。最终，我们在多个实验中实现了超过 10% 的绝对 WER 降低，整个流程有潜力推广到其他代表性不足的语言。
英文摘要：
Whisper and other large-scale automatic speech recognition models have made significant progress in performance. However, their performance on many low-resource languages, such as Kazakh, is not satisfactory. It is worth researching how to utilize low-cost data to improve the performance of Whisper on under-represented languages. In this study, we utilized easily accessible unpaired speech and text data and combined the language model GPT with Whisper on Kazakh. We implemented end of transcript (EOT) judgment modification and hallucination penalty to improve the performance of speech recognition. Further, we employed the decoding average token log probability as a criterion to select samples from unlabeled speech data and used pseudo-labeled data to fine-tune the model to further improve its performance. Ultimately, we achieved more than 10\% absolute WER reduction in multiple experiments, and the whole process has the potential to be generalized to other under-represented languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
