SALAD-VAE:基于语言-音频蒸馏的语义音频压缩
SALAD-VAE: Semantic Audio Compression with Language-Audio Distillation
AI总结:
提出SALAD-VAE,一种频域连续紧凑语义音频变分自编码器,以7.8 Hz低帧率实现高压缩比和高质量重建,通过对比学习与CLAP蒸馏增强语义,在分类基准上优于同类模型,并支持零样本音频字幕生成。
摘要:
现代生成模型和多模态模型越来越依赖于紧凑的潜在表示,这些表示在语义丰富性与高保真重建之间进行权衡和平衡。我们提出了SALAD-VAE,一种连续且高度紧凑的语义音频变分自编码器,它在频域中运行,以极低的潜在帧率(7.8 Hz)实现了最先进的压缩,同时展现出语义结构并产生高质量的音频。我们增强了标准VAE的语义损失和数据增强,特别是对比学习和基于CLAP的嵌入蒸馏,使其能够泛化到不同的音频领域。与同类最先进的VAE相比,SALAD-VAE的计算复杂度显著降低,其重建质量与之相当,同时在一系列广泛的分类基准上持续优于它们。此外,所提出的额外损失函数提供了一个经过训练的CLAP投影层,可用于零样本音频字幕生成和分类,与预训练的CLAP音频-文本嵌入相匹配。
英文摘要:
Modern generative and multimodal models increasingly rely on compact latent representations that trade and balance semantic richness with high-fidelity reconstruction. We introduce SALAD-VAE, a continuous and highly compact semantic Audio Variational Autoencoder, which operates in the frequency domain and achieves state-of-the-art compression with very low latent frame rate (7.8 Hz) while surfacing semantic structure and producing high audio quality. We enhance the standard VAE semantic losses and augmentation, specifically contrastive learning and CLAP-based embedding distillation, enabling it to generalize across diverse audio domains. With a significantly less computational complex architecture than comparable state-of-the-art VAEs, SALAD-VAE matches their reconstruction quality while it consistently outperforms them on a wide range of classification benchmarks. Furthermore, the proposed additional loss function provides a trained CLAP projection layer, which can be used zero-shot audio captioning and classification matching pretrained CLAP audio-text embeddings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。