Moshi：一种面向实时对话的语音-文本基础模型
Moshi: a speech-text foundation model for real-time dialogue
AI总结：
本文提出全双工语音-文本基础模型 Moshi，通过语音到语音生成、并行双流建模和 Inner Monologue 时间对齐文本前缀，实现低延迟实时口语对话。
摘要：
我们介绍 Moshi，这是一个语音-文本基础模型和全双工口语对话框架。当前的口语对话系统依赖于由独立组件构成的流水线，即语音活动检测、语音识别、文本对话和文本转语音。此类框架无法模拟真实对话体验。首先，它们的复杂性导致交互之间存在数秒的延迟。其次，由于文本是对话的中间模态，会改变意义的非语言信息——例如情绪或非语音声音——会在交互中丢失。最后，它们依赖于按说话人轮次进行的切分，而这没有考虑重叠语音、打断和感叹插话。Moshi 通过将口语对话转化为语音到语音生成，一并解决了这些彼此独立的问题。从文本语言模型骨干出发，Moshi 将语音生成为神经音频编解码器残差量化器产生的 token，同时将其自身语音和用户语音分别建模为并行流。这使得去除显式说话人轮次以及建模任意对话动态成为可能。此外，我们将先前工作中分层的语义到声学 token 生成扩展为：首先预测时间对齐的文本 token，作为音频 token 的前缀。这种“Inner Monologue”方法不仅显著提高了生成语音的语言质量，我们还说明了它如何能够提供流式语音识别和文本转语音。我们所得模型是首个实时全双工口语大语言模型，理论延迟为160ms，实际延迟为200ms，并可在 https://github.com/kyutai-labs/moshi 获取。
英文摘要：
We introduce Moshi, a speech-text foundation model and full-duplex spoken dialogue framework. Current systems for spoken dialogue rely on pipelines of independent components, namely voice activity detection, speech recognition, textual dialogue and text-to-speech. Such frameworks cannot emulate the experience of real conversations. First, their complexity induces a latency of several seconds between interactions. Second, text being the intermediate modality for dialogue, non-linguistic information that modifies meaning -- such as emotion or non-speech sounds -- is lost in the interaction. Finally, they rely on a segmentation into speaker turns, which does not take into account overlapping speech, interruptions and interjections. Moshi solves these independent issues altogether by casting spoken dialogue as speech-to-speech generation. Starting from a text language model backbone, Moshi generates speech as tokens from the residual quantizer of a neural audio codec, while modeling separately its own speech and that of the user into parallel streams. This allows for the removal of explicit speaker turns, and the modeling of arbitrary conversational dynamics. We moreover extend the hierarchical semantic-to-acoustic token generation of previous work to first predict time-aligned text tokens as a prefix to audio tokens. Not only this "Inner Monologue" method significantly improves the linguistic quality of generated speech, but we also illustrate how it can provide streaming speech recognition and text-to-speech. Our resulting model is the first real-time full-duplex spoken large language model, with a theoretical latency of 160ms, 200ms in practice, and is available at https://github.com/kyutai-labs/moshi.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
