Non-autoregressive text to speech (NAR-TTS) models have attracted much attention from both academia and industry due to their fast generation speed. One limitation of NAR-TTS models is that they ignore the correlation in time and frequency domains while generating speech mel-spectrograms, and thus cause blurry and over-smoothed results. In this work, we revisit this over-smoothing problem from a novel perspective: the degree of over-smoothness is determined by the gap between the complexity of data distributions and the capability of modeling methods. Both simplifying data distributions and improving modeling methods can alleviate the problem. Accordingly, we first study methods reducing the complexity of data distributions. Then we conduct a comprehensive study on NAR-TTS models that use some advanced modeling methods. Based on these studies, we find that 1) methods that provide additional condition inputs reduce the complexity of data distributions to model, thus alleviating the over-smoothing problem and achieving better voice quality. 2) Among advanced modeling methods, Laplacian mixture loss performs well at modeling multimodal distributions and enjoys its simplicity, while GAN and Glow achieve the best voice quality while suffering from increased training or model complexity. 3) The two categories of methods can be combined to further alleviate the over-smoothness and improve the voice quality. 4) Our experiments on the multi-speaker dataset lead to similar conclusions as above and providing more variance information can reduce the difficulty of modeling the target data distribution and alleviate the requirements for model capacity.
Revisiting End-to-End Speech-to-Text Translation From Scratch
University of Edinburgh(爱丁堡大学);University of Zurich(苏黎世大学)
端到端(E2 E)语音到文本翻译(ST)通常依赖于通过语音识别或文本翻译任务使用源转录来预训练其编码器和/或解码器,没有这些任务,翻译性能会大幅下降。然而,成绩单并不总是可用的,以及如何显着这样的预训练是E2 E ST很少在文献中研究。在本文中,我们重新审视这个问题,并探讨在何种程度上的语音翻译对单独训练的E2 E ST的质量可以提高。我们重新审视了之前被证明对ST有益的几种技术,并提供了一组最佳实践,使基于Transformer的E2 E ST系统偏向于从头开始训练。此外,我们提出了参数化的距离惩罚,以促进局部性的语音自注意模型的建模。在涵盖23种语言的四个基准上,我们的实验表明,在不使用任何成绩单或预训练的情况下,所提出的系统达到甚至超过了以前采用预训练的研究,尽管在(极)低资源环境中仍然存在差距。最后,我们讨论了神经声学特征建模,其中神经模型被设计为直接从原始语音信号中提取声学特征,其目标是简化归纳偏差并在描述语音时增加模型的自由度。我们第一次证明了它的可行性,并在ST任务上显示出令人鼓舞的结果。
Revisiting Interpolation Augmentation for Speech-to-Text Generation
College of Computer Science and Technology, Harbin Engineering University(哈尔滨工程大学计算机科学与技术学院);School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);ByteDance(字节跳动);NiuTrans Research(NiuTrans研究院)