DriftAudio：一步式文生音频生成器的边际漂移分布后训练
DriftAudio: Marginal Drifting for Distributional Post-Training of One-Step Text-to-Audio Generators
浏览论文内容
中文总结 AI 辅助
提出DriftAudio，通过边际漂移后训练改善一步式文生音频生成器的分布质量，在AudioCaps上显著降低FAD和FD等指标。
中文摘要 AI 辅助
近期的一步式文生音频（TTA）模型大幅降低了推理成本，但其生成的分布仍可通过后训练得到改善。我们提出DriftAudio，一种分布后训练方法，将Drifting适配到预训练的一步式TTA生成器上。在自由形式文本条件下，按条件应用Drifting具有挑战性，因为特定条件通常仅有一个或少数几个真实样本可用。DriftAudio改为在边际音频分布上执行Drifting，同时保留文本条件生成。漂移场在冻结的音频特征空间中使用重采样的真实样本、滚动生成的样本库以及当前批次的生成样本来估计。所得的漂移场为仅更新生成器提供了分离的训练目标，保持了原始的一步推理过程。在AudioCaps上，从MeanAudio出发，DriftAudio将FAD和FD分别降低了33.9%和17.6%，同时改善了KL和CLAP。从FdAudio出发，它进一步降低了FAD、FD和KL，但在IS和CLAP上有所权衡。这些结果证明了边际分布后训练对一步式TTA生成的有效性。
英文摘要
Recent one-step text-to-audio (TTA) models substantially reduce inference cost, yet their generated distributions can still be improved through post-training. We propose DriftAudio, a distributional post-training method that adapts Drifting to pretrained one-step TTA generators. Applying Drifting condition-wise is challenging under free-form text conditioning, where only one or a few real samples are typically available for a particular condition. DriftAudio instead performs Drifting on the marginal audio distribution while retaining text-conditioned generation. The drifting field is estimated in a frozen audio feature space using resampled real samples, a rolling bank of generated samples, and the current batch of generated samples. The resulting Drifting field provides a detached training target for updating only the generator, keeping the original one-step inference procedure. On AudioCaps, starting from MeanAudio, DriftAudio reduces FAD and FD by 33.9% and 17.6%, respectively, while also improving KL and CLAP. Starting from FdAudio, it further reduces FAD, FD, and KL, with trade-offs in IS and CLAP. These results demonstrate the effectiveness of marginal distributional post-training for one-step TTA generation.
