多模态对比学习联合框架用于鲁棒的语音术语检测和关键词 spotting
Joint Multimodal Contrastive Learning for Robust Spoken Term Detection and Keyword Spotting
AI总结：
本文提出一种联合多模态对比学习框架，通过统一音频和跨模态监督提升语音术语检测和关键词 spotting 的鲁棒性。
摘要：
声学词嵌入（AWEs）提高了语音检索任务如语音术语检测（STD）和关键词 spotting（KWS）的效率。然而，现有方法存在局限性，包括单模态监督、音频-音频和音频-文本对齐的离散优化，以及需要任务特定模型。为了解决这些不足，我们提出了一种联合多模态对比学习框架，将音频和跨模态监督统一在共享嵌入空间中。我们的方法同时优化：（i）受CLAP损失启发的音频-文本对比学习，以对齐音频和文本表示；（ii）通过深度词判别（DWD）损失实现的音频-音频对比学习，以增强类内紧凑性和类间分离。所提出的方法在词判别任务上优于现有AWE基线，同时灵活支持STD和KWS。据我们所知，这是此类全面方法的首次尝试。
英文摘要：
Acoustic Word Embeddings (AWEs) improve the efficiency of speech retrieval tasks such as Spoken Term Detection (STD) and Keyword Spotting (KWS). However, existing approaches suffer from limitations, including unimodal supervision, disjoint optimization of audio-audio and audio-text alignment, and the need for task-specific models. To address these shortcomings, we propose a joint multimodal contrastive learning framework that unifies both acoustic and cross-modal supervision in a shared embedding space. Our approach simultaneously optimizes: (i) audio-text contrastive learning, inspired by the CLAP loss, to align audio and text representations and (ii) audio-audio contrastive learning, via Deep Word Discrimination (DWD) loss, to enhance intra-class compactness and inter-class separation. The proposed method outperforms existing AWE baselines on word discrimination task while flexibly supporting both STD and KWS. To our knowledge, this is the first comprehensive approach of its kind.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
