看得更少，听得更好：联合奖励的 GRPO 用于流式语音识别

中文摘要 AI 辅助

流式自动语音识别（ASR）必须在转录内容和每个词提交的速度上被联合评估。延迟流建模（DSM）已成为流式大型音频语言模型的主导范式，它暴露了一个结构延迟 $\ au$，该延迟限制了解码器的前瞻范围。我们表明，$\ au$ 是用户感知延迟的一个不佳代理指标，并且 DSM 的基于对齐的监督留下了延迟问题：在每一个 $\ au$ 下都使用相同的强制对齐转录，迫使模型扣留它本可以提交的词。我们引入了 AWED，一个相对于每个词的声学结束来定义的词级发射延迟指标，并使用 GRPO 在联合评分转录准确性和测量延迟的奖励下对 DSM 识别器进行后训练。在单一操作点（$\ au=6$ 帧）训练后，我们的模型在所有评估的前瞻预算下都优于其监督微调初始化和 Voxtral Realtime 骨干模型：在 80 毫秒结构延迟下相对降低 WER 30.8%，在 480 毫秒下相对降低 5.7%，同时将中位 AWED 从 1.17 秒降低到 1.04 秒。因此，延迟奖励的后训练在不改变架构的情况下推进了流式 ASR 的准确率-延迟帕累托前沿。

英文摘要

Streaming automatic speech recognition (ASR) must be judged jointly on what it transcribes and on how quickly it commits each word. Delayed streams modeling (DSM) has become the dominant paradigm for streaming large audio-language models, exposing a structural delay $τ$ that bounds the decoder's lookahead. We show that $τ$ is a poor proxy for user-perceived latency, and that the alignment-based supervision of DSM leaves latency on the table: the same forced-aligned transcript is used at every $τ$, forcing the model to withhold words it could already commit. We introduce AWED, a word-level emission-delay metric defined relative to the acoustic end of each word, and post-train a DSM recognizer with GRPO under a reward that scores transcription accuracy and measured delay jointly. Trained at a single operating point ($τ=6$ frames), our model dominates both its supervised fine-tuning initialization and the Voxtral Realtime backbone across all evaluated lookahead budgets: it cuts WER by 30.8\% relative at an 80\,ms structural delay, and by 5.7\% relative at 480\,ms while lowering median AWED from 1.17\,s to 1.04\,s. Latency-rewarded post-training thus advances the accuracy--latency Pareto frontier of streaming ASR without architectural change.

发表机构

• University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）

机构由 AI 辅助整理，请以论文原文为准。