arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.10093cs.CVcs.AIcs.CLcs.LG
收藏

探究属性上下文在视觉-语言模型中用于目标识别与检测的作用

Investigating the Role of Attribute Context in Vision-Language Models for Object Recognition and Detection

Intelligent Systems Program(智能系统项目);Department of Computer Science, University of Pittsburgh(匹兹堡大学计算机科学系)

Kyle Buettner, Adriana Kovashka

AI总结:

该研究针对视觉-语言模型中属性上下文的利用不足问题,通过多任务实验分析其作用,发现属性上下文常被浪费、语义未被充分利用,并提出基于形容词负样本字幕的对比训练策略以提升属性收益。

摘要:

从图像-字幕对中学习的视觉-语言对齐已被证明有益于目标识别与检测等任务。现有方法大多围绕目标类别名称的学习效果进行评估,但字幕中还包含丰富的属性上下文,在学习目标对齐时应予以考虑。目前尚不清楚各方法在学习过程中如何利用这类上下文,也不清楚当任务需要属性与目标理解能力时模型是否能胜任。为填补这一空白,我们针对视觉-语言模型中属性的作用开展了大量分析。我们专门衡量了模型对属性上下文的存在性与语义的敏感度,通过无监督短语定位(phrase grounding)和基于描述的分类方法评估其对目标嵌入的影响。我们进一步评估了属性上下文在开放词汇目标检测、细粒度文本-区域检索和属性归因任务训练中的效用。结果表明,在学习检测任务的对齐时属性上下文可能被浪费,嵌入中未充分考虑属性语义,仅通过属性描述类别是无效的。我们发现,一种能提升属性收益的可行策略是采用基于形容词的负样本字幕进行对比训练。

英文摘要:

Vision-language alignment learned from image-caption pairs has been shown to benefit tasks like object recognition and detection. Methods are mostly evaluated in terms of how well object class names are learned, but captions also contain rich attribute context that should be considered when learning object alignment. It is unclear how methods use this context in learning, as well as whether models succeed when tasks require attribute and object understanding. To address this gap, we conduct extensive analysis of the role of attributes in vision-language models. We specifically measure model sensitivity to the presence and meaning of attribute context, gauging influence on object embeddings through unsupervised phrase grounding and classification via description methods. We further evaluate the utility of attribute context in training for open-vocabulary object detection, fine-grained text-region retrieval, and attribution tasks. Our results show that attribute context can be wasted when learning alignment for detection, attribute meaning is not adequately considered in embeddings, and describing classes by only their attributes is ineffective. A viable strategy that we find to increase benefits from attributes is contrastive training with adjective-based negative captions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑