重新思考人类对齐评估：超越WER的语义度量分析

中文摘要 AI 辅助

词错误率（WER）是自动语音识别（ASR）中最常用的度量标准，它将与参考文本的每个词汇偏差视为同等代价，无论该偏差是否改变语义。这引发了一个问题：WER是否真正反映了人类对ASR转录质量的判断？我们引入了HATS-en，一个用于以人为中心的ASR评估的英语数据集。利用该数据集，我们将词汇度量与BERTScore和SemDist的多种配置进行基准比较，这些配置在语言模型、层和池化策略上有所不同。我们发现，在所有测试的度量中，WER与人类判断的一致性最低；表现最佳的SemDist配置实现了最高的总体一致性，领先于CER和BERTScore；且没有任何单一模型在所有设置中表现最佳。CER尽管简单且成本低，但仍与这些最佳配置非常接近。与先前的建议一致，我们的结果支持将ASR评估转向CER，既适用于英语也适用于形态音节书写系统，因为它对于评估实际应捕获的内容而言是一个更具可解释性和低成本的度量，并将SemDist作为补充评估。

英文摘要

Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equally costly, regardless of whether it changes meaning. This raises the question: does WER actually track how humans judge ASR transcript quality? We introduce HATS-en, an English dataset for human-centered ASR evaluation. Using this dataset, we benchmark lexical metrics against several configurations of BERTScore and SemDist, varying the language model, layer, and pooling strategy. We find that WER agrees least with human judgment among all metrics tested, that the best-performing SemDist configurations achieve the highest overall agreement, ahead of CER and BERTScore, and that no single model is best across settings. CER, despite its simplicity and low cost, remains remarkably close to these best configurations. In line with prior recommendations, our results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation.

发表机构

• Algoma University（阿尔戈马大学）
• IDIAP Research Institute（IDIAP研究所）
• Seoul National University（首尔大学）

机构由 AI 辅助整理，请以论文原文为准。