arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.10604eess.AScs.SD
收藏

生成还是复制:听诊音频潜在扩散模型

Generation or Replication: Auscultating Audio Latent Diffusion Models

MERL(三菱电机研究实验室)

Dimitrios Bralios, Gordon Wichern, François G. Germain, Zexu Pan, Sameer Khurana, Chiori Hori, Jonathan Le Roux

AI总结:

本文通过比较输出与训练数据,系统分析音频潜在扩散模型的记忆行为,发现梅尔频谱图相似性比嵌入向量更有效,并揭示AudioCaps数据集中存在大量重复片段。

摘要:

具有根据文本描述按需生成逼真声音片段能力的音频潜在扩散模型的引入,有可能彻底改变我们处理音频的方式。在这项工作中,我们初步尝试理解音频潜在扩散模型的内部工作机制,通过调查其音频输出与训练数据的比较情况,类似于医生通过聆听患者器官的声音来听诊患者。使用在AudioCaps数据集上训练的文本到音频潜在扩散模型,我们系统地分析了记忆行为随训练集大小的变化。我们还评估了不同的检索指标以寻找训练数据记忆的证据,发现梅尔频谱图之间的相似性在检测匹配方面比学习到的嵌入向量更稳健。在分析音频潜在扩散模型记忆性的过程中,我们还发现了AudioCaps数据库中存在大量重复的音频片段。

英文摘要:

The introduction of audio latent diffusion models possessing the ability to generate realistic sound clips on demand from a text description has the potential to revolutionize how we work with audio. In this work, we make an initial attempt at understanding the inner workings of audio latent diffusion models by investigating how their audio outputs compare with the training data, similar to how a doctor auscultates a patient by listening to the sounds of their organs. Using text-to-audio latent diffusion models trained on the AudioCaps dataset, we systematically analyze memorization behavior as a function of training set size. We also evaluate different retrieval metrics for evidence of training data memorization, finding the similarity between mel spectrograms to be more robust in detecting matches than learned embedding vectors. In the process of analyzing memorization in audio latent diffusion models, we also discover a large amount of duplicated audio clips within the AudioCaps database.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑