arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.23068cs.SD
收藏

TADA: 一种通过文本-语音双模对齐生成语音模型的框架

TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment

Hume AI;Dartmouth College(达特茅斯学院)

Trung Dang, Sharath Rao, Ananya Gupta, Christopher Gagne, Panagiotis Tzirakis, Alice Baird, Jakub Piotr Cłapa, Peter Chin, Alan Cowen

AI总结:

TADA通过文本-语音双模对齐生成框架,实现语音与文本的一对一同步建模,提升语音生成的保真度和效率,减少幻觉并降低推理成本。

摘要:

现代的文本到语音(TTS)系统越来越多地利用大语言模型(LLM)架构来实现可扩展、高保真、零样本生成。然而,这些系统通常依赖于固定帧率的语音tokenization,导致生成的语音序列显著长于,并且与对应的文本不同步。除了计算效率低下外,这种序列长度差异往往会在TTS中触发幻觉,并加剧语音语言模型(SLM)中的模态差距。在本文中,我们提出了一种新的tokenization方案,建立连续语音特征与文本token之间的一对一同步,使可以在LLM中实现统一的单流建模。我们证明这些同步token能够保持高保真的音频重建,并且可以有效地在大语言模型的潜在空间中通过流匹配头进行建模。此外,能够在上下文中无缝切换语音模态的能力使文本-only指导成为可能——这是一种将文本-only和文本-语音模式的logits混合以灵活弥合向文本-only LLM智能的差距的技术。实验结果表明,我们的方法在性能上与最先进的TTS和SLM系统相当,同时几乎消除了内容幻觉并保持了语言的完整性,所有这些都在显著降低推理成本的情况下实现。

英文摘要:

Modern Text-to-Speech (TTS) systems increasingly leverage Large Language Model (LLM) architectures to achieve scalable, high-fidelity, zero-shot generation. However, these systems typically rely on fixed-frame-rate acoustic tokenization, resulting in speech sequences that are significantly longer than, and asynchronous with their corresponding text. Beyond computational inefficiency, this sequence length disparity often triggers hallucinations in TTS and amplifies the modality gap in spoken language modeling (SLM). In this paper, we propose a novel tokenization scheme that establishes one-to-one synchronization between continuous acoustic features and text tokens, enabling unified, single-stream modeling within an LLM. We demonstrate that these synchronous tokens maintain high-fidelity audio reconstruction and can be effectively modeled in a latent space by a large language model with a flow matching head. Moreover, the ability to seamlessly toggle speech modality within the context enables text-only guidance--a technique that blends logits from text-only and text-speech modes to flexibly bridge the gap toward text-only LLM intelligence. Experimental results indicate that our approach achieves performance competitive with state-of-the-art TTS and SLM systems while virtually eliminating content hallucinations and preserving linguistic integrity, all at a significantly reduced inference cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑