WER是无意识的：评估ASR错误如何扭曲患者面对对话中的临床理解
WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialogue
AI总结：
本文提出通过LLM作为判断者评估ASR错误的临床影响，发现WER等指标与临床风险标签相关性低，引入优化的LLM模型提升评估准确性。
摘要：
随着自动语音识别（ASR）在临床对话中的应用日益增加，标准评估仍然严重依赖于词错误率（WER）。本文挑战这一标准，研究WER或其他常见指标是否与转录错误的临床影响相关。我们通过让专家临床医生将真实语音与ASR生成的语音进行比较，建立了一个金标准基准，对两种不同的医生-患者对话数据集中的差异进行标注，以标记发现的任何差异的临床影响。我们的分析表明，WER和一组全面的现有指标与临床医生分配的风险标签（无、最小或显著影响）相关性极低。为弥合这一评估差距，我们引入了LLM作为判断者，通过DSPy使用GEPA进行程序化优化，以复制专家的临床评估。优化的判断者（Gemini-2.5-Pro）实现了与人类相当的性能，准确率达到90%，并获得强Cohen's kappa值0.816。本研究提供了一个经过验证的、自动化的框架，使ASR评估能够超越简单的文本忠实度，进入临床对话中必要的、可扩展的安全性评估。
英文摘要：
As Automatic Speech Recognition (ASR) is increasingly deployed in clinical dialogue, standard evaluations still rely heavily on Word Error Rate (WER). This paper challenges that standard, investigating whether WER or other common metrics correlate with the clinical impact of transcription errors. We establish a gold-standard benchmark by having expert clinicians compare ground-truth utterances to their ASR-generated counterparts, labeling the clinical impact of any discrepancies found in two distinct doctor-patient dialogue datasets. Our analysis reveals that WER and a comprehensive suite of existing metrics correlate poorly with the clinician-assigned risk labels (No, Minimal, or Significant Impact). To bridge this evaluation gap, we introduce an LLM-as-a-Judge, programmatically optimized using GEPA through DSPy to replicate expert clinical assessment. The optimized judge (Gemini-2.5-Pro) achieves human-comparable performance, obtaining 90% accuracy and a strong Cohen's kappa of 0.816. This work provides a validated, automated framework for moving ASR evaluation beyond simple textual fidelity to a necessary, scalable assessment of safety in clinical dialogue.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
