基于梯度的任意语音识别模型的语音到文本对齐：从连接主义时间分类到语音大语言模型
Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs
AI总结：
研究基于梯度的任意可微语音识别模型的语音到文本对齐方法，通过对教师强制令牌对数概率取梯度并解码，无需训练、改模型及对齐头，适用于各模型家族，在多模型上评估，结果显示该方法能产生可用对齐，有优有劣。
摘要：
语音到文本对齐是指在音频中找到每个单词的时间边界。一些模型直接提供这种对齐，而其他模型则不然。连接主义时间分类（CTC）和变换器模型通过构建具有对齐，而基于注意力的编码器-解码器（AED）和语音大语言模型（LLMs）则没有，它们的单词时间通常从注意力权重中读取。所有这些信号都存在于编码器帧网格上，这限制了它们的时间精度。我们研究了一种适用于任何可微语音识别模型的基于梯度的通用对齐方法。我们对每个教师强制令牌对数概率相对于输入取梯度，将其简化为每帧显著性，并通过单次动态规划将所得矩阵解码为单词边界。该方法无需训练、无需修改模型且无需对齐头，适用于所有模型家族，包括语音大语言模型，并且在输入网格上对齐而非在较粗的编码器网格上。我们在四个家族的十六个模型上进行评估，包括阅读（TIMIT）和自发（Buckeye）语音，每个模型都与模型自身的原生或基于注意力的对齐进行对比。我们发现梯度为每个模型产生了可用的对齐，通常略落后于强大的原生对齐器，但在原生对齐较弱的地方（如流式模型）表现更好，其主要缺点是每个令牌需要一次反向传播的成本。
英文摘要：
Speech-to-text alignment means finding the temporal boundaries of each word in the audio. Some models provide such an alignment directly and others do not. Connectionist temporal classification (CTC) and transducer models have an alignment by construction, whereas attention-based encoder-decoders (AED) and speech large language models (LLMs) do not, and their word timings are usually read off the attention weights instead. All of these signals live on the encoder frame grid, which bounds their temporal precision. We study a generic gradient-based alignment that applies to any differentiable ASR model. We take the gradient of each teacher-forced token log probability with respect to the input, reduce it to a per-frame saliency, and decode the resulting matrix into word boundaries with a single dynamic-programming pass. The method needs no training, no model modification and no alignment heads, works across all model families including the speech LLMs, and aligns on the input grid rather than on the coarser encoder grid. We evaluate it on sixteen models from four families, on read (TIMIT) and spontaneous (Buckeye) speech, each against the model's own native or attention-based alignment. We find that the gradient yields a usable alignment for every model, that it is usually somewhat behind a strong native aligner but better where the native alignment is weak, as for the streaming models, and that its main disadvantage is the cost of one backward pass per token.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
