arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.17702cs.CV
收藏

CVPR 2023 第一届基础模型挑战赛 Track2 的解决方案

The Solution for the CVPR 2023 1st foundation model challenge-Track2

Nanjing University of Science and Technology(南京理工大学);Dalian University of Technology(大连理工大学)

Haonan Xu, Yurui Huang, Sishun Pan, Zhihao Guan, Yi Xu, Yang Yang

AI总结:

本文针对跨模态交通检索提出解决方案,将任务分为行人和车辆检索,分别基于 IRRA 和 BLIP 设计属性挖掘、包含关系匹配及颜色增强等策略,最终在 CVPR 2023 基础模型挑战赛 Track2 中以 70.9 分获得第一名。

摘要:

在本文中,我们提出了一种跨模态交通检索的解决方案。由于交通图像存在跨域问题,我们通过一个简单策略将问题划分为行人检索和车辆检索两个子任务。在行人检索任务中,我们使用 IRRA 作为基础模型,并专门设计了一个属性分类模块来挖掘属性标签所隐含的知识。更重要的是,我们采用包含关系匹配策略,使具有包含关系的图像-文本对在特征空间中具有相似的表示。对于车辆检索任务,我们使用 BLIP 作为基础模型。由于对齐车辆的颜色属性具有挑战性,我们引入基于属性的目标检测技术,向车辆图像中添加颜色补丁块以进行颜色数据增强。这作为强先验信息,帮助模型完成图像-文本对齐。同时,我们将标注属性纳入图像-文本对齐损失中,以学习细粒度对齐,防止相似图像和文本被错误分离。我们的方法在最终 B-board 测试中以 70.9 的分数排名第一。

英文摘要:

In this paper, we propose a solution for cross-modal transportation retrieval. Due to the cross-domain problem of traffic images, we divide the problem into two sub-tasks of pedestrian retrieval and vehicle retrieval through a simple strategy. In pedestrian retrieval tasks, we use IRRA as the base model and specifically design an Attribute Classification to mine the knowledge implied by attribute labels. More importantly, We use the strategy of Inclusion Relation Matching to make the image-text pairs with inclusion relation have similar representation in the feature space. For the vehicle retrieval task, we use BLIP as the base model. Since aligning the color attributes of vehicles is challenging, we introduce attribute-based object detection techniques to add color patch blocks to vehicle images for color data augmentation. This serves as strong prior information, helping the model perform the image-text alignment. At the same time, we incorporate labeled attributes into the image-text alignment loss to learn fine-grained alignment and prevent similar images and texts from being incorrectly separated. Our approach ranked first in the final B-board test with a score of 70.9.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑