arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2207.00282cs.CVcs.IRcs.MM
收藏

(非)似然训练用于可解释嵌入

(Un)likelihood Training for Interpretable Embedding

City University of Hong Kong(香港城市大学);Singapore Management University(新加坡管理大学)

Jiaxin Wu, Chong-Wah Ngo, Wing-Kwong Chan, Zhijian Hou

AI总结:

本文提出似然与非似然训练目标及编码器-解码器网络,实现可解释跨模态视频表示,在TRECVid和MSR-VTT上显著优于现有检索模型。

摘要:

跨模态表示学习已成为弥合文本与视觉数据之间语义鸿沟的新常态。然而,在连续潜在空间中学习模态无关的表示,通常被视为一种黑箱数据驱动的训练过程。众所周知,表示学习的有效性在很大程度上依赖于训练数据的质量和规模。对于视频表示学习,要获得一套完整标注视频内容全谱的标签用于训练,即使并非不可能,也是极其困难的。这些问题——黑箱训练和数据集偏差——使得表示学习在视频理解中的部署面临实际挑战,因为其结果难以解释且不可预测。在本文中,我们提出了两种新颖的训练目标——似然函数和非似然函数——以在解决训练中标签稀疏问题的同时,展开嵌入背后的语义。似然训练旨在解释训练标签之外的嵌入语义,而非似然训练则利用先验知识进行正则化,以确保语义连贯的解释。结合这两种训练目标,我们提出了一种新的编码器-解码器网络,用于学习可解释的跨模态表示,并应用于即席视频搜索。在TRECVid和MSR-VTT数据集上的大量实验表明,所提出的网络在统计显著性的性能优势上优于几种最先进的检索模型。

英文摘要:

Cross-modal representation learning has become a new normal for bridging the semantic gap between text and visual data. Learning modality agnostic representations in a continuous latent space, however, is often treated as a black-box data-driven training process. It is well-known that the effectiveness of representation learning depends heavily on the quality and scale of training data. For video representation learning, having a complete set of labels that annotate the full spectrum of video content for training is highly difficult if not impossible. These issues, black-box training and dataset bias, make representation learning practically challenging to be deployed for video understanding due to unexplainable and unpredictable results. In this paper, we propose two novel training objectives, likelihood and unlikelihood functions, to unroll semantics behind embeddings while addressing the label sparsity problem in training. The likelihood training aims to interpret semantics of embeddings beyond training labels, while the unlikelihood training leverages prior knowledge for regularization to ensure semantically coherent interpretation. With both training objectives, a new encoder-decoder network, which learns interpretable cross-modal representation, is proposed for ad-hoc video search. Extensive experiments on TRECVid and MSR-VTT datasets show the proposed network outperforms several state-of-the-art retrieval models with a statistically significant performance margin.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑