通过知识蒸馏实现高效的交错式语音建模
Efficient Interleaved Speech Modeling through Knowledge Distillation
AI总结：
针对语音语言模型规模大、延迟高难以部署的问题，提出层对齐知识蒸馏方法，推出2B参数的TinyWave模型家族，在多项基准上接近教师模型性能，优于同规模基线，适配消费级硬件并开源相关资源。
摘要：
当前语音语言模型的规模和延迟超出了许多部署环境的限制。我们通过层对齐蒸馏构建紧凑且具表达能力的语音生成模型，通过匹配隐藏状态、注意力图和软化logits，将大型多模态Transformer压缩3倍，且性能损失极小。我们推出TinyWave，这是一类参数量为2B的模型，支持语音到语音生成与交错式语音-文本生成，在5万小时公开音频数据上训练。TinyWave支持两种模式：（i）使用语音或表达性token的纯语音生成，（ii）混合语音-文本续写。在Libri-Light上的评估显示，TinyWave的归一化困惑度仅比其教师模型高1.4个点以内。在口语StoryCloze和SALMon基准上的准确率达到教师模型性能的93%-97%，优于同等规模的基线模型。这些模型针对消费级硬件部署进行了优化，可应用于实时对话智能体、辅助技术及低资源环境。我们开源了模型、训练代码和评估脚本，以支持关于紧凑且具表达能力的语音生成的可复现研究。
英文摘要：
Current speech language models exceed the size and latency constraints of many deployment environments. We build compact, expressive speech generation models through layer-aligned distillation, matching hidden states, attention maps, and softened logits to compress large multimodal transformers by 3x with minimal loss in performance. We introduce TinyWave, a family of 2B-parameter models for speech-to-speech and interleaved speech-text generation, trained on 50,000 hours of public audio. TinyWave supports (i) speech-only generation using phonetic or expressive tokens and (ii) mixed speech-text continuations. Evaluation on Libri-Light shows TinyWave within 1.4 normalized perplexity points of its teacher. Accuracy on spoken StoryCloze and SALMon reaches 93-97% of the teacher's performance, outperforming size-matched baselines. These models are optimized for deployment on commodity hardware, enabling applications in real-time conversational agents, assistive technologies, and low-resource environments. We release models, training code, and evaluation scripts to support reproducible research on compact, expressive speech generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
