WER We Stand：乌尔都语自动语音识别模型基准评测
WER We Stand: Benchmarking Urdu ASR Models
AI总结：
本文通过WER评测Whisper、MMS和Seamless-M4T三类乌尔都语ASR模型，并发布首个乌尔都语对话语音基准数据集，揭示不同模型在朗读与对话场景下的性能差异及文本规范化需求。
摘要：
本文对乌尔都语自动语音识别（ASR）模型进行了全面评估。我们使用词错误率（WER）分析了三个ASR模型家族——Whisper、MMS和Seamless-M4T的性能，并详细考察了最常见的错误词以及插入、删除和替换等错误类型。我们的分析基于两类数据集：朗读语音和对话语音。值得注意的是，我们提出了首个专为乌尔都语ASR模型基准评测设计的对话语音数据集。我们发现，seamless-large在朗读语音数据集上优于其他ASR模型，而whisper-large在对话语音数据集上表现最佳。此外，该评估凸显了仅使用定量指标评估乌尔都语等低资源语言ASR模型的复杂性，并强调需要建立稳健的乌尔都语文本规范化系统。我们的研究结果为开发面向乌尔都语等低资源语言的稳健ASR系统提供了有价值的见解。
英文摘要：
This paper presents a comprehensive evaluation of Urdu Automatic Speech Recognition (ASR) models. We analyze the performance of three ASR model families: Whisper, MMS, and Seamless-M4T using Word Error Rate (WER), along with a detailed examination of the most frequent wrong words and error types including insertions, deletions, and substitutions. Our analysis is conducted using two types of datasets, read speech and conversational speech. Notably, we present the first conversational speech dataset designed for benchmarking Urdu ASR models. We find that seamless-large outperforms other ASR models on the read speech dataset, while whisper-large performs best on the conversational speech dataset. Furthermore, this evaluation highlights the complexities of assessing ASR models for low-resource languages like Urdu using quantitative metrics alone and emphasizes the need for a robust Urdu text normalization system. Our findings contribute valuable insights for developing robust ASR systems for low-resource languages like Urdu.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
