字幕指标是否失效？延迟、聋人与听障用户评分以及跨技术的偏差
Are Caption Metrics Broken? Latency, Deaf and Hard of Hearing User Ratings, and Bias across Technologies
- Gallaudet University（加劳德特大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本研究通过大规模用户调查发现，现有字幕质量指标（WER、ACE2、NER）对电视字幕有效，但对ASR字幕偏差明显，且延迟显著影响聋人与听障用户的观看体验。
AI中文摘要：
电视上的实时字幕经常包含错误和时序问题，使得聋人和听障（DHH）观众难以跟上对话。字幕质量指标必须反映DHH观众的真实电视观看体验，这一点至关重要。为此，我们描述了一项基于美国的大规模在线调查，共有216名经过验证的参与者，他们提供了302份回复，累计包含4,832个数据点。参与者观看了从现场电视录制的70个片段中抽取的视频，并被要求对四种条件下的字幕质量和主观理解程度进行评分：按原始录制状态（延迟高达7-12秒）的电视字幕、与音频同步的电视字幕、与音频同步的自动语音识别（ASR）生成的字幕，以及平均延迟两秒的ASR字幕。所有字幕均根据词错误率（WER）、自动字幕评估（ACE2）以及数字、编辑和识别（NER）指标进行评估。结果显示，电视字幕和ASR字幕的评分相似。对于电视字幕，所有三个指标与观众评分的相关性为中等到高度相关，但对于ASR字幕，相关性远低得多，这使得这些指标远非技术中立。此外，字幕延迟显著影响观众体验，尤其是典型的7-12秒电视延迟。我们讨论了采用字幕质量指标的影响。
英文摘要：
Live captions on TV often contain errors and timing issues, making it hard for deaf and hard-of-hearing (DHH) viewers to follow dialog. It is essential that caption quality metrics reflect the lived DHH TV viewing experience. To this end, we describe a U.S.-based large-scale online survey with 216 validated participants, who provided 302 responses containing a cumulative 4,832 data points. Participants viewed videos drawn from a pool of 70 clips recorded from live TV, and were asked to rate the caption quality and subjective understanding of the content across four conditions: TV captions as originally recorded with up to 7-12 seconds delay, TV captions synchronized with audio, Automatic Speech Recognition (ASR)-generated captions synchronized with audio, and ASR captions with an average two-second delay. All captions were evaluated against the Word Error Rate (WER), Automated Caption Evaluation (ACE2) and Number, Edition and Recognition (NER) metrics. Results show that TV and ASR captions were rated similarly. For TV captions, all three metrics were moderately-to-highly correlated with viewer ratings, but far less so for ASR captions, making them far from technology-neutral. Additionally, caption latencies significantly impact the viewer experience, especially typical 7-12-second TV delays. We discuss the implications for the adoption of caption quality metrics.
