分析并缓解神经编解码器语言模型中离散音频令牌的不一致性
Analyzing and Mitigating Inconsistency in Discrete Audio Tokens for Neural Codec Language Models
AI总结：
针对离散音频令牌的离散表示不一致性（DRI）导致神经编解码器语言模型语音生成出现遗漏重复的问题，该研究定量分析DRI现象并提出缓解方法，经多数据集验证有效且通用。
摘要：
基于大语言模型（LLMs）的进展，音频处理领域对使用离散音频令牌序列训练音频生成任务的兴趣日益浓厚。然而，通过神经音频编解码器直接对音频进行离散化处理，得到的序列往往与文本序列存在本质差异。文本令牌序列具有确定性，而离散音频令牌则会因上下文因素表现出显著的可变性，却仍能生成感知上完全相同的音频片段。我们将这一现象称为**离散表示不一致性（Discrete Representation Inconsistency, DRI）**。这种不一致性会导致单个音频片段被多个差异明显的序列表示，给神经编解码器语言模型造成混淆，进而在语音生成过程中出现遗漏和重复问题。本文对EnCodec等主流音频令牌化器中的DRI现象开展了定量分析。我们提出的方法可有效缓解神经音频编解码器的DRI现象。此外，我们在LibriTTS和大规模MLS数据集（44000小时）上的神经编解码器语言模型实验，充分验证了该方法的有效性与通用性。音频样本演示可在线获取。
英文摘要：
Building upon advancements in Large Language Models (LLMs), the field of audio processing has seen increased interest in training audio generation tasks with discrete audio token sequences. However, directly discretizing audio by neural audio codecs often results in sequences that fundamentally differ from text sequences. Unlike text, where text token sequences are deterministic, discrete audio tokens can exhibit significant variability based on contextual factors, while still producing perceptually identical audio segments. We refer to this phenomenon as \textbf{Discrete Representation Inconsistency (DRI)}. This inconsistency can lead to a single audio segment being represented by multiple divergent sequences, which creates confusion in neural codec language models and results in omissions and repetitions during speech generation. In this paper, we quantitatively analyze the DRI phenomenon within popular audio tokenizers such as EnCodec. Our approach effectively mitigates the DRI phenomenon of the neural audio codec. Furthermore, extensive experiments on the neural codec language model over LibriTTS and large-scale MLS datases (44,000 hours) demonstrate the effectiveness and generality of our method. The demo of audio samples is available online~\footnote{\url{https://consistencyinneuralcodec.github.io}}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
