arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.00586cs.CV
收藏

ClipSitu:在情境识别中有效利用CLIP进行条件预测

ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognition

Debaditya Roy, Dhruv Verma, Basura Fernando

AI总结:

本文提出ClipSitu XTF,一种基于CLIP视觉令牌的交叉注意力Transformer模型,通过有效利用CLIP的视觉-语言知识进行条件预测,在imSitu数据集的语义角色标注和情境定位任务上大幅超越现有最先进方法。

摘要:

情境识别是使用一个活动动词以及参与者和物体所扮演的语义角色,对图像中发生的事件生成结构化摘要的任务。在此任务中,同一个活动动词可以描述多种不同的情境,而相同的参与者或物体类别也可以根据图像所描绘的情境扮演多种不同的语义角色。因此,情境识别模型需要理解图像的上下文以及语义角色的视觉-语言含义。为此,我们利用了通过语言描述学习图像上下文的CLIP基础模型。我们表明,通过使用CLIP的图像和文本嵌入特征,更深更宽的多层感知机(MLP)块在情境识别任务中取得了显著的结果,得益于CLIP封装的外部隐式视觉-语言知识以及现代MLP块设计的表达能力,它甚至超越了基于Transformer的最先进模型CoFormer。受此启发,我们设计了一种基于交叉注意力的Transformer,它使用CLIP视觉令牌来建模文本角色与视觉实体之间的关系。我们这种被称为ClipSitu XTF的基于交叉注意力的Transformer,在使用imSitu数据集的语义角色标注(值)top-1准确率上,以14.1%的巨大优势超越了现有的最先进方法。同样地,我们的ClipSitu XTF获得了最先进的情境定位性能。我们将公开代码。

英文摘要:

Situation Recognition is the task of generating a structured summary of what is happening in an image using an activity verb and the semantic roles played by actors and objects. In this task, the same activity verb can describe a diverse set of situations as well as the same actor or object category can play a diverse set of semantic roles depending on the situation depicted in the image. Hence a situation recognition model needs to understand the context of the image and the visual-linguistic meaning of semantic roles. Therefore, we leverage the CLIP foundational model that has learned the context of images via language descriptions. We show that deeper-and-wider multi-layer perceptron (MLP) blocks obtain noteworthy results for the situation recognition task by using CLIP image and text embedding features and it even outperforms the state-of-the-art CoFormer, a Transformer-based model, thanks to the external implicit visual-linguistic knowledge encapsulated by CLIP and the expressive power of modern MLP block designs. Motivated by this, we design a cross-attention-based Transformer using CLIP visual tokens that model the relation between textual roles and visual entities. Our cross-attention-based Transformer known as ClipSitu XTF outperforms existing state-of-the-art by a large margin of 14.1\% on semantic role labelling (value) for top-1 accuracy using imSitu dataset. {Similarly, our ClipSitu XTF obtains state-of-the-art situation localization performance.} We will make the code publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑