ReGen：用于高效波形扩散模型的分层多提示表示生成

中文摘要 AI 辅助

表示对齐（REPA）已被研究用于加速扩散训练，但在扩散Transformer（DiT）中对中间表示进行正则化可能会隐式地纠缠潜在变量并限制生成能力。为解决此问题，我们提出了ReGen，这是一个分层多提示表示生成框架，可在单个扩散模型中联合估计表示和数据的多个向量场。我们进一步引入广义流匹配（GFM）来提高条件流匹配（CFM）的泛化能力。我们在包括神经音频编解码器和Wave-VAE在内的单阶段波形扩散模型上验证了ReGen。ReGen显著提高了从12.5Hz高度压缩潜在表示生成的波形质量。我们还展示了ReGenVoice，这是一个基于潜在扩散模型（LDM）的文本到语音模型，在小数据集上实现了很强的语音清晰度（WER）和说话者相似度（SIM）。此外，以6.25Hz运行LDM并具有丰富的语义和声学潜在表示可实现高效训练和采样，在4个GPU上仅需1天训练且推理快速，RTF为0.08。音频样本可在该https URL获取。

英文摘要

Representation alignment (REPA) has been investigated to accelerate diffusion training, but we observe that regularizing intermediate representations in diffusion Transformers (DiT) may implicitly entangle latents and limit generative capacity. To address this issue, we propose ReGen, a hierarchical multi-prompt representation generation framework that jointly estimates multiple vector fields for both representations and data within a single diffusion model. We further introduce generalized flow matching (GFM) to improve the generalization of conditional flow matching (CFM). We validate ReGen on single-stage waveform diffusion models including neural audio codec and Wave-VAE. ReGen significantly improves waveform generation quality from highly compressed latent representations at 12.5 Hz. We also present ReGenVoice, a latent diffusion model (LDM)-based text-to-speech model that achieves strong speech intelligibility (WER) and speaker similarity (SIM) with a small dataset. Moreover, operating the LDM at 6.25 Hz with rich semantic and acoustic latent representation enables efficient training and sampling, requiring only 1 day of training on 4 GPUs and fast inference with an RTF of 0.08. Audio samples are available at https://regenvoice.github.io/demo/.

发表机构

• Department of Artificial Intelligence, Ajou University, Suwon, Korea（人工智能系，全州大学）
• KT Corp., Seoul, Korea（KT公司）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

Accepted to ICML 2026