arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.13119cs.CLcs.SDeess.AS
收藏

从粗到细的对齐实现更优的语音-图像检索

Coarse-to-fine Alignment Makes Better Speech-image Retrieval

Netease Yidun AI Lab(网易易盾AI实验室)

Lifeng Zhou, Yuke Li

AI总结:

本研究提出一种粗到细对齐的语音-图像检索框架,联合语音图像对比与匹配任务,结合嵌入队列和动量蒸馏优化训练,在两个基准数据集上R@1超现有最优方法4%以上,泛化性优异。

摘要:

本文提出了一种用于语音-图像检索的新型框架。我们利用语音-图像对比(speech-image contrastive, SIC)学习任务在粗粒度层面对齐语音与图像表示,再通过语音-图像匹配(speech-image matching, SIM)学习任务进一步细化细粒度跨模态对齐。SIC与SIM学习任务以统一方式联合训练。为优化学习过程,我们采用嵌入队列,在SIC学习期间助力高效采样高质量且多样化的负样本表示;此外,该队列还能基于SIC任务中计算得到的对比相似度有效挖掘难负样本,从而强化SIM任务的学习。为进一步优化含噪监督下的学习效果,我们在训练流程中引入动量蒸馏。实验结果表明,在两个语音-图像检索任务的基准数据集上,本文框架的R@1指标较当前最优方法提升超过4%。此外,零样本实验结果显示,该框架具备出色的泛化能力。

英文摘要:

In this paper, we propose a novel framework for speech-image retrieval. We utilize speech-image contrastive (SIC) learning tasks to align speech and image representations at a coarse level and speech-image matching (SIM) learning tasks to further refine the fine-grained cross-modal alignment. SIC and SIM learning tasks are jointly trained in a unified manner. To optimize the learning process, we utilize an embedding queue that facilitates efficient sampling of high-quality and diverse negative representations during SIC learning. Additionally, it enhances the learning of SIM tasks by effectively mining hard negatives based on contrastive similarities calculated in SIC tasks. To further optimize learning under noisy supervision, we incorporate momentum distillation into the training process. Experimental results show that our framework outperforms the state-of-the-art method by more than 4% in R@1 on two benchmark datasets for the speech-image retrieval tasks. Moreover, as observed in zero-shot experiments, our framework demonstrates excellent generalization capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑