arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.04598cs.CL
收藏

在普什图语上对多语言语音模型进行基准测试：零样本ASR、脚本失败和跨领域评估

Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation

Independent Researcher（独立研究员）

Hanif Rahman

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文首次在公开普什图语数据上评估多模型，探讨零样本ASR、脚本失败和跨领域评估，发现零样本ASR的WER差异显著，脚本失败问题难以通过WER检测，跨领域评估显示模型性能下降，且普什图语独特音素导致误差集中。

摘要：

普什图语由约60-80百万人使用，但尚未有公开的多语言自动语音识别（ASR）基准测试。本文报告了首个可复现的多模型评估，涵盖零样本ASR、脚本层面的失败以及微调模型的跨领域评估。对于零样本ASR，十种模型（包括七种Whisper型号、MMS-1B、SeamlessM4T-v2-large和OmniASR-CTC-300M）在FLEURS普什图语测试集和过滤后的Common Voice~24子集上进行评估；零样本Whisper的WER范围从90%到297%，中等型号在Common Voice~24上崩溃至461%，与解码器循环一致。SeamlessM4T在Common Voice~24上达到39.7%的WER（目前最佳零样本结果）。MMS-1B在FLEURS上达到43.8%。对于脚本失败，语言识别审计显示，没有任何Whisper模型在超过0.8%的语句中产生普什图语脚本输出，而MMS-1B、SeamlessM4T和OmniASR的脚本保真度均超过93%；WER本身无法揭示此失败，因为生成阿拉伯语脚本输出的模型在普什图语音上未实现可解释的ASR。对于跨领域评估，五种微调的普什图语ASR模型在两个测试集上进行评估：公布的WER为14%的性能在非分布数据集上降到了32.5-59%，而一个增强的模型在两个数据集上达到了35.1%，无跨领域降解。字符类误差分层确认，普什图语独特的音素（回声系列和侧摩擦音）导致了不成比例的误差量。所有评估仅涵盖朗读语音。五种阻碍累积进展的结构性障碍被识别，并提出了五项研究优先级。

英文摘要：

Pashto is spoken by approximately 60--80 million people but has no published benchmarks for multilingual automatic speech recognition (ASR) on any shared public test set. This paper reports the first reproducible multi-model evaluation on public Pashto data, covering zero-shot ASR, script-level failure, and cross-domain evaluation of fine-tuned models. For zero-shot ASR, ten models (all seven Whisper sizes, MMS-1B, SeamlessM4T-v2-large, and OmniASR-CTC-300M) are evaluated on the FLEURS Pashto test set and a filtered Common Voice~24 subset; zero-shot Whisper WER ranges from 90% to 297%, with the medium model collapsing to 461% on Common Voice~24 consistent with decoder looping. SeamlessM4T achieves 39.7% WER on Common Voice~24 (the best zero-shot result reported to date, as of submission); MMS-1B achieves 43.8% on FLEURS. For script failure, a language-identification audit shows that no Whisper model produces Pashto-script output in more than 0.8% of utterances, while MMS-1B, SeamlessM4T, and OmniASR each exceed 93% Pashto-script fidelity; WER alone does not reveal this failure, since a model generating Arabic-script output on Pashto audio has not achieved ASR in any interpretable sense. For cross-domain evaluation, five fine-tuned Pashto ASR models are evaluated on both test sets: published WER figures of 14% degrade to 32.5--59% on out-of-distribution sets, while one augmented model achieves 35.1% on both sets with zero cross-domain degradation. Character-class error stratification confirms that Pashto-unique phonemes (the retroflex series and lateral fricatives) account for disproportionate error mass. All evaluations cover read speech only. Five structural impediments to cumulative progress are identified and five ordered research priorities are argued.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑