PROFASR-BENCH:面向高风险专业演讲的上下文条件ASR基准
PROFASR-BENCH: A Benchmark for Context-Conditioned ASR in High-Stakes Professional Speech
AI总结:
ProfASR-Bench 是一个针对高风险专业演讲的ASR基准测试,通过上下文条件识别评估,揭示了当前系统在利用上下文信息方面的不足。
摘要:
在专业场合中,自动语音识别(ASR)面临现有基准测试忽视的挑战:密集的领域术语、正式语体变化以及对关键实体错误的接近零容忍度。我们提出了ProfASR-Bench,一个适用于金融、医学、法律和技术等高风险应用的专业演讲评估套件。每个示例配对一个自然语言提示(领域提示和/或说话人资料)与一个富含实体的目标发言,使能够受控地测量上下文条件的识别能力。该语料库支持传统的ASR指标,以及实体感知得分和按口音和性别切片的报告。使用代表性的家庭Whisper(编码器-解码器ASR)和Qwen-Omni(音频语言模型)在匹配的无上下文、资料、领域+资料、 oracle 和对抗性条件下进行测试,我们发现一致的模式:轻量级文本上下文对平均词错误率(WER)几乎没有或没有影响,即使在oracle提示下,对抗性提示也不可靠地降低性能。我们将此现象称为上下文利用差距(CUG):当前系统名义上可提示但未充分利用 readily 可用的辅助信息。ProfASR-Bench 提供了标准化的上下文阶梯、实体和切片感知的报告以及带有置信区间和可重复的测试平台,用于比较不同模型家族的融合策略。
英文摘要:
Automatic Speech Recognition (ASR) in professional settings faces challenges that existing benchmarks underplay: dense domain terminology, formal register variation, and near-zero tolerance for critical entity errors. We present ProfASR-Bench, a professional-talk evaluation suite for high-stakes applications across finance, medicine, legal, and technology. Each example pairs a natural-language prompt (domain cue and/or speaker profile) with an entity-rich target utterance, enabling controlled measurement of context-conditioned recognition. The corpus supports conventional ASR metrics alongside entity-aware scores and slice-wise reporting by accent and gender. Using representative families Whisper (encoder-decoder ASR) and Qwen-Omni (audio language models) under matched no-context, profile, domain+profile, oracle, and adversarial conditions, we find a consistent pattern: lightweight textual context produces little to no change in average word error rate (WER), even with oracle prompts, and adversarial prompts do not reliably degrade performance. We term this the context-utilization gap (CUG): current systems are nominally promptable yet underuse readily available side information. ProfASR-Bench provides a standardized context ladder, entity- and slice-aware reporting with confidence intervals, and a reproducible testbed for comparing fusion strategies across model families. Dataset: https://huggingface.co/datasets/prdeepakbabu/ProfASR-Bench Code: https://github.com/prdeepakbabu/ProfASR-Bench
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。