TASTE: 用于语音语言建模的文本对齐语音标记化与嵌入
TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling
AI总结：
TASTE通过语音重建目标实现文本对齐的语音标记化与嵌入，提升语音语言建模的性能。
摘要：
近期的研究致力于开发能够听且能说的语音语言模型（SLMs），以实现更自然的人与大语言模型（LLM）的交互。联合语音-文本建模是实现这一目标的一个有前景的方向。然而，最近语音标记在联合建模中的有效性仍处于探索阶段。为了解决这个问题，我们引入了文本对齐的语音标记化与嵌入（TASTE），一种在标记化阶段通过将语音标记与对应的文本转录对齐来直接解决模态差距的方法。我们提出了一种通过基于注意力的聚合机制和语音重建作为训练目标来实现这一目标的方法。我们进行了广泛的实验，证明TASTE能够在大幅减少标记序列长度的同时保留关键的非语言信息。通过TASTE，我们可以通过在预训练文本LLM上使用低秩适应进行简单的联合语音语言建模。实验结果表明，基于TASTE的SLMs在SALMON和StoryCloze任务上与先前工作表现相当；而在语音延续任务上，在主观和客观评估中均显著优于其他预训练的SLMs。据我们所知，TASTE是首个利用重建目标自动学习适合语音语言建模的文本对齐语音标记化和嵌入的端到端方法。我们的演示、代码和模型可在https://mtkresearch.github.io/TASTE-SpokenLM.github.io上找到。
英文摘要：
Recent efforts target spoken language models (SLMs) that not only listen but also speak for more natural human-LLM interaction. Joint speech-text modeling is a promising direction to achieve this. However, the effectiveness of recent speech tokens for joint modeling remains underexplored. To address this, we introduce Text-Aligned Speech Tokenization and Embedding (TASTE), a method that directly addresses the modality gap by aligning speech token with the corresponding text transcription during the tokenization stage. We propose a method that can achieve this through a attention-based aggregation mechanism and with speech reconstruction as the training objective. We conduct extensive experiments and show that TASTE can preserve essential paralinguistic information while dramatically reducing the token sequence length. With TASTE, we perform straightforward joint spoken language modeling by using Low-Rank Adaptation on the pre-trained text LLM. Experimental results show that TASTE-based SLMs perform comparable to previous work on SALMON and StoryCloze; while significantly outperform other pre-trained SLMs on speech continuation across subjective and objective evaluations. To our knowledge, TASTE is the first end-to-end approach that utilizes a reconstruction objective to automatically learn a text-aligned speech tokenization and embedding suitable for spoken language modeling. Our demo, code, and model are available at https://mtkresearch.github.io/TASTE-SpokenLM.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
