视觉基础语言模型中的语义组合
Semantic Composition in Visually Grounded Language Models
AI总结：
本文针对视觉基础语言模型在语义组合表征上的不足，提出基准、蒸馏度量、因果追踪、句法池化和跨模态正则化五种方法，以提升视觉-语言组合能力。
摘要：
什么是句子意义及其理想表征？人类语言的大量表达能力源于语义组合，即心智在成分之上以层级和关系方式表征意义的能力。与此同时，许多句子意义位于文本之外，需要在感觉、运动和体验模态中进行基础化才能被充分学习。尽管大型语言模型表现出相当强的组合能力，近期研究表明，视觉基础语言模型在表征组合结构方面严重失败。在本论文中，我们探讨模型是否以及如何组合视觉基础语义，以及我们如何提升它们这样做的能力。具体而言，我们提出：1）WinogroundVQA，一个新的组合式视觉问答基准；2）句法神经模块蒸馏，一种衡量句子嵌入模型组合能力的方法；3）面向图像描述模型的因果追踪，用于定位对视觉-语言组合至关重要的神经表征；4）句法平均池化，用于向句子嵌入注入组合归纳偏置；5）跨模态注意力一致性正则化，一种用于视觉-语言关系对齐的自监督目标函数。最后，我们讨论了本工作与神经科学、心理语言学、形式语义学和哲学的联系。
英文摘要：
What is sentence meaning and its ideal representation? Much of the expressive power of human language derives from semantic composition, the mind's ability to represent meaning hierarchically & relationally over constituents. At the same time, much sentential meaning is outside the text and requires grounding in sensory, motor, and experiential modalities to be adequately learned. Although large language models display considerable compositional ability, recent work shows that visually-grounded language models drastically fail to represent compositional structure. In this thesis, we explore whether & how models compose visually grounded semantics, and how we might improve their ability to do so. Specifically, we introduce 1) WinogroundVQA, a new compositional visual question answering benchmark, 2) Syntactic Neural Module Distillation, a measure of compositional ability in sentence embedding models, 3) Causal Tracing for Image Captioning Models to locate neural representations vital for vision-language composition, 4) Syntactic MeanPool to inject a compositional inductive bias into sentence embeddings, and 5) Cross-modal Attention Congruence Regularization, a self-supervised objective function for vision-language relation alignment. We close by discussing connections of our work to neuroscience, psycholinguistics, formal semantics, and philosophy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
