使用单阶段非自回归Transformer的掩码音频生成
Masked Audio Generation using a Single Non-Autoregressive Transformer
AI总结：
本研究提出单阶段非自回归掩码音频生成模型MAGNeT，搭配重打分机制与混合解码策略，在文本生成音乐、文本生成音频任务上性能媲美基线且速度快7倍。
摘要：
我们提出了MAGNeT，这是一种掩码生成式序列建模方法，可直接在多个音频token流上运行。与现有研究不同，MAGNeT由单阶段非自回归Transformer构成。训练阶段，我们预测由掩码调度器生成的掩码token跨度；推理阶段，我们通过多个解码步骤逐步构建输出序列。为进一步提升生成音频的质量，我们提出了一种新颖的重打分方法：利用外部预训练模型对MAGNeT的预测结果进行重打分与排序，再将其用于后续解码步骤。最后，我们探索了MAGNeT的混合版本：融合自回归与非自回归模型，前几秒音频以自回归方式生成，序列其余部分则并行解码。我们验证了MAGNeT在文本生成音乐、文本生成音频任务上的有效性，并结合客观指标与人类研究开展了全面的实证评估。所提方法的性能与评估基线相当，但速度显著提升（比自回归基线快7倍）。通过消融实验与分析，我们阐明了MAGNeT各组成部分的重要性，同时指出了自回归与非自回归建模在延迟、吞吐量和生成质量之间的权衡。样本可在我们的演示页面https://pages.cs.huji.ac.il/adiyoss-lab/MAGNeT 获取。
英文摘要：
We introduce MAGNeT, a masked generative sequence modeling method that operates directly over several streams of audio tokens. Unlike prior work, MAGNeT is comprised of a single-stage, non-autoregressive transformer. During training, we predict spans of masked tokens obtained from a masking scheduler, while during inference we gradually construct the output sequence using several decoding steps. To further enhance the quality of the generated audio, we introduce a novel rescoring method in which, we leverage an external pre-trained model to rescore and rank predictions from MAGNeT, which will be then used for later decoding steps. Lastly, we explore a hybrid version of MAGNeT, in which we fuse between autoregressive and non-autoregressive models to generate the first few seconds in an autoregressive manner while the rest of the sequence is being decoded in parallel. We demonstrate the efficiency of MAGNeT for the task of text-to-music and text-to-audio generation and conduct an extensive empirical evaluation, considering both objective metrics and human studies. The proposed approach is comparable to the evaluated baselines, while being significantly faster (x7 faster than the autoregressive baseline). Through ablation studies and analysis, we shed light on the importance of each of the components comprising MAGNeT, together with pointing to the trade-offs between autoregressive and non-autoregressive modeling, considering latency, throughput, and generation quality. Samples are available on our demo page https://pages.cs.huji.ac.il/adiyoss-lab/MAGNeT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
