交互式乐器声音生成中的谱图修复
Spectrogram Inpainting for Interactive Generation of Instrument Sounds
摘要：
使用深度神经网络进行声音合成的现代方法很难控制，特别是当细粒度的条件信息不可用时，这阻碍了音乐家采用它们。在本文中，我们将单个乐器音符的生成作为一个基于inpainting的任务，引入新颖独特的方法来迭代地塑造声音。为此，我们提出了一个两步的方法：首先，我们适应的VQ-VAE-2图像生成架构的频谱图，以转换为紧凑的离散codemap的实值频谱图，然后我们实现标记屏蔽的Transformers这些codemap的inpainting为基础的生成。我们将所提出的架构应用于NSynth数据集上的掩码恢复任务。最重要的是，我们开源了一个交互式Web界面，通过修复来转换声音，为艺术家和从业者提供了新的创造性用途。
英文摘要：
Modern approaches to sound synthesis using deep neural networks are hard to control, especially when fine-grained conditioning information is not available, hindering their adoption by musicians. In this paper, we cast the generation of individual instrumental notes as an inpainting-based task, introducing novel and unique ways to iteratively shape sounds. To this end, we propose a two-step approach: first, we adapt the VQ-VAE-2 image generation architecture to spectrograms in order to convert real-valued spectrograms into compact discrete codemaps, we then implement token-masked Transformers for the inpainting-based generation of these codemaps. We apply the proposed architecture on the NSynth dataset on masked resampling tasks. Most crucially, we open-source an interactive web interface to transform sounds by inpainting, for artists and practitioners alike, opening up to new, creative uses.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
