发表机构
Epoch Learn（Epoch Learn）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文系统评估了Token合并方法在Whisper多语言语音识别模型上的效果，涵盖16种语言和3种模型规模，发现该方法能显著提升计算效率且几乎不损失准确性，并可与微调兼容。
AI 中文摘要
像 Whisper 这样的领先多语言语音识别模型无需特定语言的训练即可转录多种低资源语言，但部署时计算成本高昂。Token 合并通过动态组合冗余特征，在推理期间缩短序列长度，而无需重新训练，从而缓解了这一低效问题。在本文中，我们系统地评估了 Token 合并方法在 Whisper 模型家族上的表现，涵盖十六种不同语言和三种不同模型规模。我们还测试了 Token 合并与低资源语言微调（DoRA）的交互作用。我们的研究结果表明，在大多数低资源语言和模型规模下，合并 Token 能提高计算效率，且转录准确性几乎没有损失，并且即使在模型经过微调后仍然有效。我们的结果证明，Token 合并是一种非常实用的方法，可以使多语言语音识别部署更快、更便宜。
英文摘要
Leading multilingual speech recognition models like Whisper transcribe diverse, low-resource languages without language-specific training but are computationally expensive to deploy. Token merging mitigates this inefficiency by dynamically combining redundant features, shortening the sequence length during inference without requiring retraining. In this paper, we systematically evaluate token merging on the Whisper model family across sixteen diverse languages and three different model sizes. We also test how token merging interacts with fine-tuning (DoRA) on low-resource languages. Our findings show that merging tokens increases computational efficiency with almost no loss in transcription accuracy across most low-resource languages and model sizes, and it works even after the model has been fine-tuned. Our results demonstrate that token merging is a highly practical method for making multilingual speech recognition faster and cheaper to deploy.
Comments11 pages, 3 figures
