归一化中丢失了什么？探索多语言ASR模型评估中的陷阱
What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations
AI总结：
本文揭示多语言ASR评估中文本归一化对印度语系脚本的缺陷，导致性能指标虚高，并提出利用本土语言学知识改进归一化流程。
摘要：
本文探讨了评估多语言自动语音识别（ASR）模型时的陷阱，特别关注印度语系文字。我们调查了领先ASR模型（包括OpenAI Whisper、Meta的MMS、Seamless和Assembly AI的Conformer）所采用的文本归一化流程及其对性能指标的意外影响。我们的研究揭示，当前的文本归一化实践虽然旨在通过移除拼写、标点和特殊字符等不一致性来标准化ASR输出以实现公平比较，但在应用于印度语系文字时存在根本性缺陷。通过使用文本相似度分数和深入的语言学检查进行的实证分析，我们证明这些缺陷导致印度语系语言的性能指标被人为提升。最后，我们提出转向开发利用本土语言学专业知识的文本归一化流程，以确保多语言ASR模型更稳健和准确的评估。
英文摘要：
This paper explores the pitfalls in evaluating multilingual automatic speech recognition (ASR) models, with a particular focus on Indic language scripts. We investigate the text normalization routine employed by leading ASR models, including OpenAI Whisper, Meta's MMS, Seamless, and Assembly AI's Conformer, and their unintended consequences on performance metrics. Our research reveals that current text normalization practices, while aiming to standardize ASR outputs for fair comparison, by removing inconsistencies such as variations in spelling, punctuation, and special characters, are fundamentally flawed when applied to Indic scripts. Through empirical analysis using text similarity scores and in-depth linguistic examination, we demonstrate that these flaws lead to artificially improved performance metrics for Indic languages. We conclude by proposing a shift towards developing text normalization routines that leverage native linguistic expertise, ensuring more robust and accurate evaluations of multilingual ASR models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
