arXiv 2308.128982023-08-25cs.MMcs.AIcs.CLcs.CV

语言知识能否提升视觉-语言预训练中的多模态对齐？

Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?

South China University of Technology（华南理工大学）；University of Sydney（悉尼大学）；Harbin Institute of Technology（哈尔滨工业大学）；JD Explore Academy at JD.com（京东探索研究院）

Fei Wang, Liang Ding, Jun Rao, Ye Liu, Li Shen, Changxing Ding

备注：: [TL;DR] we design and release the SNARE, the first large-scale multimodal alignment probing benchmark for current vision-language pretrained models

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究针对VLP中语言知识对多模态对齐的影响问题，发布首个大规模探测基准SNARE，通过四项任务分析5个先进VLP模型，揭示其在句法、否定逻辑、视觉关系判断等方面的能力缺陷。

摘要：

多媒体领域对使用多模态预训练神经网络模型感知和表征物理世界展现出浓厚兴趣，其中视觉-语言预训练（VLP）是当前最受关注的方向。但目前鲜有研究探讨两个问题：1）VLP过程中能否提取出语义、句法等核心语言知识；2）这类语言知识如何影响或增强多模态对齐。针对这一空白，本文旨在阐明语义表达、句法结构等综合语言知识对多模态对齐的影响。具体而言，我们设计并发布了首个大规模多模态对齐探测基准SNARE，用于检测词汇、语义、句法知识等关键语言成分，该基准包含四项任务：语义结构、否定逻辑、属性归属和关系组合。基于我们提出的探测基准，我们对5个先进VLP模型开展了全面分析，结果表明VLP模型：i）对复杂句法结构不敏感，依赖实义词理解句子；ii）对句子与否定词的组合理解能力有限；iii）在判断视觉信息中是否存在动作或空间关系方面面临挑战，且难以验证三元组组合的正确性。我们已将基准和代码发布于https://github.com/WangFei-2019/SNARE/。

英文摘要：

The multimedia community has shown a significant interest in perceiving and representing the physical world with multimodal pretrained neural network models, and among them, the visual-language pertaining (VLP) is, currently, the most captivating topic. However, there have been few endeavors dedicated to the exploration of 1) whether essential linguistic knowledge (e.g., semantics and syntax) can be extracted during VLP, and 2) how such linguistic knowledge impact or enhance the multimodal alignment. In response, here we aim to elucidate the impact of comprehensive linguistic knowledge, including semantic expression and syntactic structure, on multimodal alignment. Specifically, we design and release the SNARE, the first large-scale multimodal alignment probing benchmark, to detect the vital linguistic components, e.g., lexical, semantic, and syntax knowledge, containing four tasks: Semantic structure, Negation logic, Attribute ownership, and Relationship composition. Based on our proposed probing benchmarks, our holistic analyses of five advanced VLP models illustrate that the VLP model: i) shows insensitivity towards complex syntax structures and relies on content words for sentence comprehension; ii) demonstrates limited comprehension of combinations between sentences and negations; iii) faces challenges in determining the presence of actions or spatial relationships within visual information and struggles with verifying the correctness of triple combinations. We make our benchmark and code available at \url{https://github.com/WangFei-2019/SNARE/}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。