arXiv 2307.048382023-07-19cs.CVcs.LG

CREPE: 结合CLIP的可学习提示改进视觉关系预测

CREPE: Learnable Prompting With CLIP Improves Visual Relationship Prediction

Arizona State University（亚利桑那州立大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）

Rakshith Subramanyam, T. S. Jayram, Rushil Anirudh, Jayaraman J. Thiagarajan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究针对视觉关系预测任务，基于UVTransE框架结合CLIP设计表示方案，提出带对比训练策略的CREPE方法，在Visual Genome基准上取得了性能提升，验证了视觉语言模型在该任务的有效性。

摘要：

本文探究了视觉语言模型（VLMs）——具体而言是CLIP——在视觉对象关系预测中的潜力，该任务需要将图像中的视觉特征解读为基于语言的关系。当前最优方法采用复杂的图模型，利用语言线索与视觉特征来应对这一挑战。我们假设CLIP嵌入中强大的语言先验能够简化这些图模型，为更简洁的方法铺平道路。我们采用UVTransE关系预测框架，该框架将关系学习为一种平移嵌入，结合场景中的主体、客体和并集框嵌入。我们在UVTransE框架内系统探究了基于CLIP的主体、客体和并集框表示的设计，并提出CREPE（CLIP表示增强谓词估计）。CREPE对所有三个边界框都采用基于文本的表示，并引入一种新颖的对比训练策略来自动推断并集框的文本提示。我们的方法在谓词估计任务上取得了最优性能，在Visual Genome基准上mR@5达到27.79，mR@20达到31.95，相较于近期最优方法在mR@20指标上提升了15.3%。本工作证明了CLIP在对象关系预测中的有效性，也推动了视觉语言模型在这一具有挑战性领域的进一步研究。

英文摘要：

In this paper, we explore the potential of Vision-Language Models (VLMs), specifically CLIP, in predicting visual object relationships, which involves interpreting visual features from images into language-based relations. Current state-of-the-art methods use complex graphical models that utilize language cues and visual features to address this challenge. We hypothesize that the strong language priors in CLIP embeddings can simplify these graphical models paving for a simpler approach. We adopt the UVTransE relation prediction framework, which learns the relation as a translational embedding with subject, object, and union box embeddings from a scene. We systematically explore the design of CLIP-based subject, object, and union-box representations within the UVTransE framework and propose CREPE (CLIP Representation Enhanced Predicate Estimation). CREPE utilizes text-based representations for all three bounding boxes and introduces a novel contrastive training strategy to automatically infer the text prompt for union-box. Our approach achieves state-of-the-art performance in predicate estimation, mR@5 27.79, and mR@20 31.95 on the Visual Genome benchmark, achieving a 15.3\% gain in performance over recent state-of-the-art at mR@20. This work demonstrates CLIP's effectiveness in object relation prediction and encourages further research on VLMs in this challenging domain.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。