WhisperRT -- 将Whisper转化为因果流式模型
WhisperRT -- Turning Whisper into a Causal Streaming Model
AI总结：
本文提出将Whisper模型转化为低延迟流式模型的方法，通过使编码器因果化并生成与可用时间上下文对齐的token，实现更高效的流式语音识别。
摘要：
自动语音识别（ASR）已取得显著进展，如OpenAI Whisper和NVIDIA Canary等模型在离线转录中达到最先进的性能。然而，这些模型不适用于流式（在线或实时）转录，因受限于其架构和训练方法。本文提出将Transformer编码器-解码器模型转化为低延迟流式模型的方法。编码器被设计为因果以逐步处理音频，而解码器则根据部分编码器状态生成与可用时间上下文对齐的token。这需要显式同步编码输入帧与token生成。由于token只能在观察到足够的语音证据后生成，因此存在固有的延迟，需要对编码器-解码器对齐机制进行微调。我们提出了一种更新的推理机制，利用微调后的因果编码器和解码器以获得贪婪和束搜索解码，并证明其在局部上最优。在低延迟分块大小（小于300毫秒）的实验中，我们的微调模型在大多数情况下优于现有非微调的流式方法，同时使用更低的复杂度。我们发布了训练和推理代码，以及微调后的模型，以支持进一步的流式ASR研究和开发。
英文摘要：
Automatic Speech Recognition (ASR) has seen remarkable progress, with models like OpenAI Whisper and NVIDIA Canary achieving state-of-the-art (SOTA) performance in offline transcription. However, these models are not designed for streaming (online or real-time) transcription, due to limitations in their architecture and training methodology. We propose a method to turn the transformer encoder-decoder model into a low-latency streaming model. The encoder is made causal to process audio incrementally, while the decoder conditions on partial encoder states to generate tokens aligned with the available temporal context. This requires explicit synchronization between encoded input frames and token emissions. Since tokens are produced only after sufficient acoustic evidence is observed, an inherent latency arises, necessitating fine-tuning of the encoder-decoder alignment mechanism. We propose an updated inference mechanism that utilizes the fine-tuned causal encoder and decoder to yield greedy and beam-search decoding, and is shown to be locally optimal. Experiments on low-latency chunk sizes (less than 300 msec) show that our fine-tuned model outperforms existing non-fine-tuned streaming approaches in most cases, while using a lower complexity. We release our training and inference code, along with the fine-tuned models, to support further research and development in streaming ASR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
