Zero-Shot Mono-to-Binaural 语音合成
Zero-Shot Mono-to-Binaural Speech Synthesis
AI总结：
提出首个零样本神经方法 ZeroBAS，利用无参数几何变换与预训练去噪声码器，从单耳音频和位置信息合成双耳音频，在分布外数据上超越监督方法。
摘要：
我们提出了 ZeroBAS，一种无需在任何双耳数据上进行训练，仅从单耳录音和位置信息合成双耳音频的神经方法。据我们所知，这是首个公开发表的用于单耳到双耳音频合成的零样本神经方法。具体而言，我们展示了基于声源位置的无参数几何时间规整和幅度缩放足以获得初始双耳合成结果，该结果可通过迭代应用预训练去噪声码器进行细化。此外，我们发现该方法能跨房间条件泛化，我们通过引入新数据集 TUT Mono-to-Binaural 来评估最先进的单耳到双耳合成方法在未见条件下的表现来测量这一点。我们的零样本方法在标准单耳到双耳数据集上感知表现与监督方法相当，甚至在我们分布外的 TUT Mono-to-Binaural 数据集上超越了它们。我们的结果突显了预训练生成音频模型和零样本学习在实现稳健双耳音频合成方面的潜力。
英文摘要：
We present ZeroBAS, a neural method to synthesize binaural audio from monaural audio recordings and positional information without training on any binaural data. To our knowledge, this is the first published zero-shot neural approach to mono-to-binaural audio synthesis. Specifically, we show that a parameter-free geometric time warping and amplitude scaling based on source location suffices to get an initial binaural synthesis that can be refined by iteratively applying a pretrained denoising vocoder. Furthermore, we find this leads to generalization across room conditions, which we measure by introducing a new dataset, TUT Mono-to-Binaural, to evaluate state-of-the-art monaural-to-binaural synthesis methods on unseen conditions. Our zero-shot method is perceptually on-par with the performance of supervised methods on the standard mono-to-binaural dataset, and even surpasses them on our out-of-distribution TUT Mono-to-Binaural dataset. Our results highlight the potential of pretrained generative audio models and zero-shot learning to unlock robust binaural audio synthesis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
