从图像描述中发现具有视觉基础语义的有意义单元
Discovering Meaningful Units with Visually Grounded Semantics from Image Captions
AI总结：
针对现有视觉-语言模型细粒度对齐单元缺乏实际语义的问题，提出内置token分组架构的模型，结合物体发现图像编码器对齐表示，提升了模型细粒度跨模态理解能力，所得token组与可落地短语高度相似。
摘要：
细粒度知识对于视觉-语言模型更好地理解现实世界至关重要。尽管已有研究尝试在视觉和语言领域获取这类知识，但大多数工作聚焦于将图像块与语言侧的token进行对齐。然而，图像块对人眼而言不具备任何意义，单个token也未必承载图像中可落地的信息，描述场景不同维度的是token组。本研究提出一种模型，将描述token分组作为其架构的一部分，以捕获细粒度语言表示。我们期望所得表示能对应图像中存在的物体层级，因此将该表示与经训练可发现物体的图像编码器输出进行对齐。研究表明，通过学习对token分组，视觉-语言模型对视觉和语言的细粒度理解能力得到提升。此外，本模型发现的token组在定性和定量层面均与文本中可落地的短语高度相似。
英文摘要：
Fine-grained knowledge is crucial for vision-language models to obtain a better understanding of the real world. While there has been work trying to acquire this kind of knowledge in the space of vision and language, it has mostly focused on aligning the image patches with the tokens on the language side. However, image patches do not have any meaning to the human eye, and individual tokens do not necessarily carry groundable information in the image. It is groups of tokens which describe different aspects of the scene. In this work, we propose a model which groups the caption tokens as part of its architecture in order to capture a fine-grained representation of the language. We expect our representations to be at the level of objects present in the image, and therefore align our representations with the output of an image encoder trained to discover objects. We show that by learning to group the tokens, the vision-language model has a better fine-grained understanding of vision and language. In addition, the token groups that our model discovers are highly similar to groundable phrases in text, both qualitatively and quantitatively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
