arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.05657cs.CV
收藏

Tag2Text:通过图像标记引导视觉-语言模型

Tag2Text: Guiding Vision-Language Model via Image Tagging

Fudan University(复旦大学);OPPO Research Institute(OPPO研究院);International Digital Economy Academy (IDEA)(国际数字经济学院)

Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, Lei Zhang

AI总结:

Tag2Text通过显式学习图像标记器,利用无标注标记引导视觉-语言模型,提升生成与对齐任务性能,达到最先进水平。

摘要:

本文提出了Tag2Text,一种视觉语言预训练(VLP)框架,它将图像标记引入视觉-语言模型,以引导视觉-语言特征的学习。与先前使用手动标记或通过现成检测器自动检测且性能有限的物体标记的工作不同,我们的方法利用从图像配对文本中解析出的标记来显式地学习一个图像标记器,从而为视觉-语言模型提供强大的语义引导。通过这种方式,Tag2Text能够利用与图像-文本对一致的大规模无标注图像标记,并提供超越物体的更多样化的标记类别。因此,Tag2Text展示了作为基础图像标记模型的能力,其零样本性能甚至可与全监督模型相媲美。此外,通过利用标记引导,Tag2Text有效增强了视觉-语言模型在基于生成和基于对齐的任务上的性能。在广泛的下游基准测试中,Tag2Text在相似模型大小和数据规模下取得了最先进的结果,证明了所提出的标记引导的有效性。代码、演示和预训练模型可在https://github.com/xinyu1205/recognize-anything获取。

英文摘要:

This paper presents Tag2Text, a vision language pre-training (VLP) framework, which introduces image tagging into vision-language models to guide the learning of visual-linguistic features. In contrast to prior works which utilize object tags either manually labeled or automatically detected with an off-the-shelf detector with limited performance, our approach explicitly learns an image tagger using tags parsed from image-paired text and thus provides a strong semantic guidance to vision-language models. In this way, Tag2Text can utilize large-scale annotation-free image tags in accordance with image-text pairs, and provides more diverse tag categories beyond objects. As a result, Tag2Text demonstrates the ability of a foundational image tagging model, with superior zero-shot performance even comparable to fully supervised models. Moreover, by leveraging the tagging guidance, Tag2Text effectively enhances the performance of vision-language models on both generation-based and alignment-based tasks. Across a wide range of downstream benchmarks, Tag2Text achieves state-of-the-art results with similar model sizes and data scales, demonstrating the efficacy of the proposed tagging guidance. Code, demo and pre-trained models are available at https://github.com/xinyu1205/recognize-anything.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑