Large-scale vision-language pre-training has achieved significant performance in multi-modal understanding and generation tasks. However, existing methods often perform poorly on image-text matching tasks that require structured representations, i.e., representations of objects, attributes, and relations. As illustrated in Fig.~reffig:case (a), the models cannot make a distinction between ``An astronaut rides a horse" and ``A horse rides an astronaut". This is because they fail to fully leverage structured knowledge when learning representations in multi-modal scenarios. In this paper, we present an end-to-end framework Structure-CLIP, which integrates Scene Graph Knowledge (SGK) to enhance multi-modal structured representations. Firstly, we use scene graphs to guide the construction of semantic negative examples, which results in an increased emphasis on learning structured representations. Moreover, a Knowledge-Enhance Encoder (KEE) is proposed to leverage SGK as input to further enhance structured representations. To verify the effectiveness of the proposed framework, we pre-train our model with the aforementioned approaches and conduct experiments on downstream tasks. Experimental results demonstrate that Structure-CLIP achieves state-of-the-art (SOTA) performance on VG-Attribution and VG-Relation datasets, with 12.5% and 4.1% ahead of the multi-modal SOTA model respectively. Meanwhile, the results on MSCOCO indicate that Structure-CLIP significantly enhances the structured representations while maintaining the ability of general representations. Our code is available at https://github.com/zjukg/Structure-CLIP.
Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features
Mohamed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
仇恨模因是社交媒体上日益增长的威胁。虽然模因中的图像及其对应的文本是相关的,但当单独查看时,它们不一定传达相同的含义。因此,检测仇恨模因需要仔细考虑视觉和文本信息。多模态预训练对于这项任务是有益的,因为它通过在相似的特征空间中表示图像和文本来有效地捕捉它们之间的关系。此外,它是必要的,通过中间融合的图像和文本特征之间的相互作用建模。大多数现有的方法要么采用多模式预训练,要么采用中间融合,但不是两者兼而有之。在这项工作中,我们提出了Hate-CLIPper架构,该架构通过一个特征交互矩阵(CLIP)明确地对使用对比图像预训练(CLIP)编码器获得的图像和文本表示之间的跨模态交互进行建模。一个简单的分类器,基于Hateful Memes Challenge(HMC)数据集,能够达到最先进的性能,AUROC为85.8,甚至超过了人类的82.65。在Propaganda Memes和TamilMemes等模因数据集上的实验也证明了该方法的通用性。最后,我们分析的可解释性的跨模态表示,并表明,跨模态的相互作用确实可以促进有意义的概念的学习。这个工作的代码可以在https://github.com/gokulkarthik/hateclipper上找到。
What Vision-Language Models `See' when they See Scenes
University of Malta, Institute of Linguistics and Language Technology(马耳他大学语言学与语言技术研究所);Universiteit Utrecht, Information and Computing Sciences(乌得勒支大学信息与计算科学系)
A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
DialCLIP: Empowering CLIP as Multi-Modal Dialog Retriever
University of Science and Technology of China(中国科学技术大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院)
NAU-QMUL: Utilizing BERT and CLIP for Multi-modal AI-Generated Image Detection
School of Accounting, Nanjing Audit University (NAU), Nanjing, Jiangsu, China(会计学院,南京审计大学(NAU));School of Electronic Engineering and Computer Science, Queen Mary University of London (QMUL), London, UK(电子工程与计算机科学学院,女王玛丽大学(QMUL))
Jailbreaking Multimodal Large Language Models using Multi-Clip Video
Department of Applied Artificial Intelligence, Sungkyunkwan University(应用人工智能系,成均馆大学);Department of Human-Artificial Intelligence Interaction, Sungkyunkwan University(人机交互系,成均馆大学)