Raon-Speech 技术报告
Raon-Speech Technical Report
AI总结：
本文提出 Raon-Speech，一个 9B 参数的语音语言模型，通过多阶段训练实现英语和韩语的语音理解、回答与生成，并扩展为全双工对话模型 Raon-SpeechChat，在语音任务上超越同类模型。
摘要：
我们提出了 Raon-Speech，一个在英语和韩语语音理解、回答和生成方面表现优异的 9B 参数语音语言模型（SpeechLM），以及 Raon-SpeechChat，一个用于自然实时对话的高性能全双工扩展。Raon-Speech 成功地将预训练的大语言模型（LLM）转换为既能理解又能生成语音的 SpeechLM，同时保留了强大的文本能力。它在 138 万小时精心策划的英语和韩语语音及文本数据集上训练，训练阶段包括：(1) 语音模块对齐，(2) 基于知识蒸馏的端到端 SpeechLM 预训练，以及 (3) 基于多任务偏好优化的后训练。在 42 个英语和韩语语音及文本基准测试中，与包括 Qwen2.5-Omni 和 Fun-Audio-Chat 在内的八个近期类似规模的音频基础模型相比，Raon-Speech 在语音中心任务上建立了最强的整体表现，同时保留了强大的文本问答性能。在此基础上，Raon-SpeechChat 通过在 119K 小时的时间对齐的真实和合成对话数据上进行持续训练，实现了自然的全双工对话。它通过三个互补的训练阶段进行：(1) 因果编码器适应，(2) 全双工预训练，(3) 用于语音和角色控制的全双工微调。在多个全双工基准测试中，Raon-SpeechChat 在 FDB v1.0 涵盖的轮流发言和中断敏感行为上显示出最明显的优势，并在更广泛的全双工评估套件中保持竞争力。我们开源了所有模型检查点、训练和推理流程以及交互式演示。
英文摘要：
We present Raon-Speech, a top-performing 9B-parameter speech language model (SpeechLM) for English and Korean speech understanding, answering, and generation, and Raon-SpeechChat, a high-performing full-duplex extension for natural real-time conversation. Raon-Speech successfully transforms a pre-trained LLM into a SpeechLM that both understands and generates speech while preserving strong text capabilities. It trains on 1.38M hours of highly curated English and Korean speech and text datasets with the following training stages: (1) speech modules alignment, (2) end-to-end SpeechLM pre-training with knowledge distillation, and (3) multi-task preference optimization-based post-training. Across 42 English and Korean speech and text benchmarks, Raon-Speech establishes the strongest overall profile on speech-centric tasks in our comparison against eight similarly sized recent audio foundation models, including Qwen2.5-Omni and Fun-Audio-Chat, while preserving strong text question answering performance. Building upon it, Raon-SpeechChat enables natural full-duplex conversation by continual training on 119K hours of time-aligned real and synthetic dialogue data. It proceeds through three complementary training stages: (1) causal encoder adaptation, (2) full-duplex pre-training, (3) full-duplex fine-tuning for voice and role-control. On multiple full-duplex benchmarks, Raon-SpeechChat shows its clearest strengths on the turn-taking and interruption-sensitive behaviors covered by FDB v1.0, and remains competitive across the broader full-duplex evaluation suite. We open-source all model checkpoints, the training and inference pipeline, and an interactive demo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
