arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.10025cs.SDcs.IReess.AS
收藏

DiffATR:基于扩散的生成式建模用于音频-文本检索

DiffATR: Diffusion-based Generative Modeling for Audio-Text Retrieval

School of ECE, Peking University(北京大学电子学院)

Yifei Xin, Xuxin Cheng, Zhihong Zhu, Xusheng Yang, Yuexian Zou

AI总结:

针对现有判别式音频-文本检索方法难以辨别分布外数据的问题,提出基于扩散的生成式框架DiffATR,通过建模联合概率并融合生成与对比损失进行优化,在多个数据集上取得优异性能且在域外检索中表现稳健。

摘要:

现有的音频-文本检索(ATR)方法本质上是判别式模型,旨在最大化条件似然,表示为 p(candidates|query)。然而,这种方法未能考虑数据的内在分布 p(query),导致难以辨别分布外数据。在本研究中,我们尝试通过生成式视角来解决这一限制,并将音频与文本之间的关系建模为它们的联合概率 p(candidates,query)。为此,我们提出了一个基于扩散的 ATR 框架(DiffATR),该框架将 ATR 建模为一个迭代过程,从噪声中逐步生成联合分布。在其训练阶段,DiffATR 从生成式和判别式两个视角进行优化:生成器通过生成损失进行优化,而特征提取器则受益于对比损失,从而结合了两种方法的优势。在 AudioCaps 和 Clotho 数据集上的实验取得了优异的性能,验证了我们方法的有效性。值得注意的是,无需任何改动,我们的 DiffATR 在域外检索设置中始终表现出强大的性能。

英文摘要:

Existing audio-text retrieval (ATR) methods are essentially discriminative models that aim to maximize the conditional likelihood, represented as p(candidates|query). Nevertheless, this methodology fails to consider the intrinsic data distribution p(query), leading to difficulties in discerning out-of-distribution data. In this work, we attempt to tackle this constraint through a generative perspective and model the relationship between audio and text as their joint probability p(candidates,query). To this end, we present a diffusion-based ATR framework (DiffATR), which models ATR as an iterative procedure that progressively generates joint distribution from noise. Throughout its training phase, DiffATR is optimized from both generative and discriminative viewpoints: the generator is refined through a generation loss, while the feature extractor benefits from a contrastive loss, thus combining the merits of both methodologies. Experiments on the AudioCaps and Clotho datasets with superior performances, verify the effectiveness of our approach. Notably, without any alterations, our DiffATR consistently exhibits strong performance in out-of-domain retrieval settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑