arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.08998cs.CV
收藏

基于视觉与语言模型的统一视觉关系检测

Unified Visual Relationship Detection with Vision and Language Models

Google Research(谷歌研究院)

Long Zhao, Liangzhe Yuan, Boqing Gong, Yin Cui, Florian Schroff, Ming-Hsuan Yang, Hartwig Adam, Ting Liu

AI总结:

针对多数据集标签体系不一致导致视觉关系检测难统一的问题,提出基于VLM的自底向上UniVRD方法,在HICO-DET等任务上性能优异,可媲美单数据集模型且扩模后效果更佳。

摘要:

本研究聚焦于训练单个视觉关系检测器,使其能在多个数据集的标签空间并集上进行预测。由于分类体系不一致,合并不同数据集的标签颇具挑战,而在视觉关系检测中,当引入对象对之间的二阶视觉语义时,这一问题会更加突出。为应对该挑战,我们提出UniVRD,这是一种利用视觉与语言模型(VLMs)实现统一视觉关系检测的新型自底向上方法。VLMs可提供对齐良好的图像与文本嵌入,其中相似关系经优化后在语义空间中相互靠近,从而实现语义统一。我们的自底向上设计让模型能够同时从目标检测数据集和视觉关系数据集中获益。在人-物交互检测和场景图生成两项任务上的实验结果表明,该模型具备极具竞争力的性能。UniVRD在HICO-DET数据集上达到38.07的mAP,较当前最优的自底向上HOI检测器高出14.26 mAP。更重要的是,我们的统一检测器在mAP指标上表现与针对特定数据集训练的模型相当,且在扩大模型规模后性能可进一步提升。我们的代码将在GitHub上公开发布。

英文摘要:

This work focuses on training a single visual relationship detector predicting over the union of label spaces from multiple datasets. Merging labels spanning different datasets could be challenging due to inconsistent taxonomies. The issue is exacerbated in visual relationship detection when second-order visual semantics are introduced between pairs of objects. To address this challenge, we propose UniVRD, a novel bottom-up method for Unified Visual Relationship Detection by leveraging vision and language models (VLMs). VLMs provide well-aligned image and text embeddings, where similar relationships are optimized to be close to each other for semantic unification. Our bottom-up design enables the model to enjoy the benefit of training with both object detection and visual relationship datasets. Empirical results on both human-object interaction detection and scene-graph generation demonstrate the competitive performance of our model. UniVRD achieves 38.07 mAP on HICO-DET, outperforming the current best bottom-up HOI detector by 14.26 mAP. More importantly, we show that our unified detector performs as well as dataset-specific models in mAP, and achieves further improvements when we scale up the model. Our code will be made publicly available on GitHub.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑