arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.00767cs.SDeess.AS
收藏

从连续文本流进行零样本文本到语音合成

Zero-Shot Text-to-Speech from Continuous Text Streams

Applied Sciences Group(应用科学组);Microsoft Corporation(微软公司)

Trung Dang, David Aponte, Dung Tran, Tianyi Chen, Kazuhito Koishida

AI总结:

本文提出LiveSpeech 2,一个基于Mamba和旋转位置嵌入的流感知零样本TTS模型,通过滑动窗口和语义引导解码实现无限长度语音生成与文本-音频流同步。

摘要:

现有的零样本文本到语音(TTS)系统通常被设计用于处理完整句子,并受限于其训练时的最大时长。然而,在许多流式应用中,文本以短块形式连续到达,要求系统即时响应。我们识别了块级流式处理所需的关键能力,并引入了LiveSpeech 2,这是一个具备流感知能力的模型,支持无限长度的语音生成、文本-音频流同步以及短语音块之间的无缝过渡。为实现这些能力,我们提出:(1)采用Mamba,这是一类以线性时间解码为特征的序列建模方法,并通过交叉注意力机制进行条件增强;(2)在交叉注意力计算中使用旋转位置嵌入,使模型能够通过滑动窗口处理无限文本流;(3)使用语义引导进行解码,该技术在推理过程中以最小开销将语音与转录文本对齐。实验结果表明,我们的模型与基于语言模型的最先进零样本TTS模型相比具有竞争力,同时还能灵活支持广泛的流式场景。

英文摘要:

Existing zero-shot text-to-speech (TTS) systems are typically designed to process complete sentences and are constrained by the maximum duration for which they have been trained. However, in many streaming applications, texts arrive continuously in short chunks, necessitating instant responses from the system. We identify the essential capabilities required for chunk-level streaming and introduce LiveSpeech 2, a stream-aware model that supports infinitely long speech generation, text-audio stream synchronization, and seamless transitions between short speech chunks. To achieve these, we propose (1) adopting Mamba, a class of sequence modeling distinguished by linear-time decoding, which is augmented by cross-attention mechanisms for conditioning, (2) utilizing rotary positional embeddings in the computation of cross-attention, enabling the model to process an infinite text stream by sliding a window, and (3) decoding with semantic guidance, a technique that aligns speech with the transcript during inference with minimal overhead. Experimental results demonstrate that our models are competitive with state-of-the-art language model-based zero-shot TTS models, while also providing flexibility to support a wide range of streaming scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑