使用Whisper的印地语带时间戳语音转录

中文摘要 AI 辅助

时间准确的语音转录在关键应用中至关重要，例如医疗对话转录、阅读评估和非典型语音识别。诸如Whisper之类的编码器-解码器自动语音识别模型，并非对所有语言都能直接生成词级时间戳。现有的词级时间戳估计方法要么使用能够进行帧级音素/字符预测的外部CTC模型（计算成本高昂），要么使用模型内部的交叉注意力分数（提供粗略的时间戳）。本文考虑印地语的带时间戳语音转录。我们提出了一种基于CTC的方法，直接在Whisper编码器输出上训练一个低复杂度的字符级CTC头，以解决现有工作在复杂性和粒度上的局限性。我们使用人工标注的词级时间戳评估该系统。实验表明，所提出的内部CTC方法在性能上与使用外部CTC模型相当，且优于基于交叉注意力分数的方法。此外，我们研究了利用CTC头输出来检测和缓解对话语音中的幻觉。我们的结果显示，由CTC输出得出的简单令牌长度准则可以改善幻觉更可能发生的短片段转录质量。

英文摘要

Time accurate speech transcription is essential in critical appli- cations, such as medical conversation transcription, reading as- sessment, and atypical speech recognition. Encoder-decoder auto- matic speech recognition models, such as Whisper, do not produce word-level timestamps out-of-the-box for all languages. Existing approaches to word-level time stamp estimation either use an exter- nal CTC model capable of frame level phoneme/character prediction which is computationally expensive, or use the cross attention scores internal to the model which gives coarse time stamps. We consider timestamped speech transcription for Hindi in this paper. We pro- pose a CTC-based approach, in which we train a low-complexity character level CTC head directly on the Whisper encoder output, to address the complexity and granularity limitations of existing works. We evaluate the system using manually-annotated word-level times- tamps. Experiments show that the proposed internal-CTC based approach performs similar to using an external CTC model and better than the cross attention score-based approaches. Further, we study the usage of CTC-head output to detect and mitigate hallu- cinations in conversational speech. Our results show that a simple token length criterion derived from the CTC output can improve the transcription quality for short segments where hallucinations are more probable.

发表机构

• Indian Institute of Technology Bombay（印度理工学院孟买分校）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

Submitted to ICASSP 2027