arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.06152cs.CLcs.AIcs.MM
收藏

Structure-CLIP:引入场景图知识以增强多模态结构化表示

Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations

Zhejiang University(浙江大学);Netease Inc.(网易公司)

Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, Wen Zhang

AI总结:

针对现有多模态预训练模型结构化表示能力不足的问题,提出整合场景图知识的Structure-CLIP框架,通过场景图引导负样本构建和知识增强编码器,在相关数据集上取得SOTA性能并兼顾通用表示能力。

摘要:

大规模视觉-语言预训练已在多模态理解和生成任务中取得显著性能。然而,现有方法在需要结构化表示(即对象、属性和关系的表示)的图文匹配任务上往往表现不佳。如图1(a)所示,模型无法区分“一名宇航员骑着一匹马”和“一匹马骑着一名宇航员”。这是因为它们在多模态场景下学习表示时未能充分利用结构化知识。本文提出一种端到端框架Structure-CLIP,该框架整合场景图知识(Scene Graph Knowledge, SGK)以增强多模态结构化表示。首先,我们利用场景图引导语义负样本的构建,从而提升对结构化表示学习的重视程度。此外,我们提出知识增强编码器(Knowledge-Enhance Encoder, KEE),将SGK作为输入以进一步增强结构化表示。为验证所提框架的有效性,我们采用上述方法对模型进行预训练,并在下游任务上开展实验。实验结果表明,Structure-CLIP在VG-Attribution和VG-Relation数据集上达到了当前最优(SOTA)性能,分别领先多模态SOTA模型12.5%和4.1%。同时,在MSCOCO上的结果显示,Structure-CLIP在保持通用表示能力的同时,显著增强了结构化表示能力。我们的代码可在https://github.com/zjukg/Structure-CLIP获取。

英文摘要:

Large-scale vision-language pre-training has achieved significant performance in multi-modal understanding and generation tasks. However, existing methods often perform poorly on image-text matching tasks that require structured representations, i.e., representations of objects, attributes, and relations. As illustrated in Fig.~reffig:case (a), the models cannot make a distinction between ``An astronaut rides a horse" and ``A horse rides an astronaut". This is because they fail to fully leverage structured knowledge when learning representations in multi-modal scenarios. In this paper, we present an end-to-end framework Structure-CLIP, which integrates Scene Graph Knowledge (SGK) to enhance multi-modal structured representations. Firstly, we use scene graphs to guide the construction of semantic negative examples, which results in an increased emphasis on learning structured representations. Moreover, a Knowledge-Enhance Encoder (KEE) is proposed to leverage SGK as input to further enhance structured representations. To verify the effectiveness of the proposed framework, we pre-train our model with the aforementioned approaches and conduct experiments on downstream tasks. Experimental results demonstrate that Structure-CLIP achieves state-of-the-art (SOTA) performance on VG-Attribution and VG-Relation datasets, with 12.5% and 4.1% ahead of the multi-modal SOTA model respectively. Meanwhile, the results on MSCOCO indicate that Structure-CLIP significantly enhances the structured representations while maintaining the ability of general representations. Our code is available at https://github.com/zjukg/Structure-CLIP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑