arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.10591eess.AScs.CLcs.LG
收藏

基于BERTScore的无序语音ASR模型质量评估

Assessing ASR Model Quality on Disordered Speech using BERTScore

Google Research(谷歌研究院);University of Washington(华盛顿大学);Leonard Florence Center for Living(伦纳德·弗洛伦斯生活中心);MND Association(运动神经元病协会)

Jimmy Tobin, Qisheng Li, Subhashini Venugopalan, Katie Seaver, Richard Cave, Katrin Tomanek

摘要:

字错误率(WER)是用于评估自动语音识别(ASR)模型质量的主要度量。研究表明,ASR模型对有言语障碍的说话者的WER往往比典型的英语说话者高得多。很难确定模型在如此高的错误率下是否有用。本研究调查使用BERTScore,文本生成的评价指标,提供了一个更翔实的衡量ASR模型的质量和有用性。BERTScore和WER都与语音语言病理学家手动注释的预测错误进行了比较,以确定错误类型和评估。BERTScore被发现与人类对错误类型和评估的评估更相关。BERTScore特别对保留意义的正字法变化(收缩和归一化错误)更鲁棒。此外,BERTScore比WER更适合错误评估,使用有序逻辑回归和赤池信息标准(AIC)进行测量。总的来说,我们的研究结果表明,BERTScore可以从实际的角度评估ASR模型性能时,补充WER,特别是对于无障碍应用程序,其中模型是有用的,即使在较低的准确性比典型的语音。

英文摘要:

Word Error Rate (WER) is the primary metric used to assess automatic speech recognition (ASR) model quality. It has been shown that ASR models tend to have much higher WER on speakers with speech impairments than typical English speakers. It is hard to determine if models can be be useful at such high error rates. This study investigates the use of BERTScore, an evaluation metric for text generation, to provide a more informative measure of ASR model quality and usefulness. Both BERTScore and WER were compared to prediction errors manually annotated by Speech Language Pathologists for error type and assessment. BERTScore was found to be more correlated with human assessment of error type and assessment. BERTScore was specifically more robust to orthographic changes (contraction and normalization errors) where meaning was preserved. Furthermore, BERTScore was a better fit of error assessment than WER, as measured using an ordinal logistic regression and the Akaike's Information Criterion (AIC). Overall, our findings suggest that BERTScore can complement WER when assessing ASR model performance from a practical perspective, especially for accessibility applications where models are useful even at lower accuracy than for typical speech.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑