用于端侧语音到语音转换的流式 Parrotron
Streaming Parrotron for on-device speech-to-speech conversion
AI总结：
本文提出端侧流式语音到语音转换模型，通过编码器实时流式处理、解码器与声码器流式生成，并设计前瞻特征堆叠与自注意力混合方法，在 Pixel4 CPU 上实现近 2 倍实时速度且质量损失极小。
摘要：
我们提出了一种完全端侧运行的流式语音到语音（Speech2Speech）转换模型，该模型可将给定的输入语音直接归一化为合成的输出语音。在移动设备上部署此类模型在内存占用和计算需求方面面临重大挑战。与参考的最先进非流式方法相比，我们提出了一种基于流式处理的方法，可在语音转换质量损失最小的情况下实现可接受的延迟。我们的方法包括：首先在说话人讲话时实时流式处理编码器；然后，说话人一停止讲话，我们就在流式声码器旁以流式模式运行频谱图解码器以生成输出语音。为了实现可接受的延迟与质量权衡，我们提出了一种新颖的编码器前瞻混合方法，该方法将前瞻特征堆叠器与前瞻自注意力相结合。我们表明，我们的流式方法在 Pixel4 CPU 上的运行速度几乎是实时的 2 倍。
英文摘要：
We present a fully on-device streaming Speech2Speech conversion model that normalizes a given input speech directly to synthesized output speech. Deploying such a model on mobile devices pose significant challenges in terms of memory footprint and computation requirements. We present a streaming-based approach to produce an acceptable delay, with minimal loss in speech conversion quality, when compared to a reference state of the art non-streaming approach. Our method consists of first streaming the encoder in real time while the speaker is speaking. Then, as soon as the speaker stops speaking, we run the spectrogram decoder in streaming mode along the side of a streaming vocoder to generate output speech. To achieve an acceptable delay-quality trade-off, we propose a novel hybrid approach for look-ahead in the encoder which combines a look-ahead feature stacker with a look-ahead self-attention. We show that our streaming approach is almost 2x faster than real time on the Pixel4 CPU.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
