arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2308.03475cs.MM
收藏

COPA:通过协同对象-补丁-文本对齐实现高效视觉-语言预训练

COPA: Efficient Vision-Language Pre-training Through Collaborative Object- and Patch-Text Alignment

Chaoya Jiang, Haiyang Xu, Wei Ye, Qinghao Ye, Chenliang Li, Ming Yan, Bin Bi, Shikun Zhang, Ji Zhang, Fei Huang

AI总结:

COPA提出一种基于ViT的视觉-语言预训练方法,通过将对象级信号转换为补丁级信号并设计补丁-文本对齐任务,在仅5%训练图像标注下实现近88%的推理加速,同时保持有竞争力的下游任务性能。

摘要:

基于目标检测的视觉-语言预训练(VLP)方法享有细粒度对象-文本对齐的丰富知识,但代价是计算开销高昂的推理。近期基于视觉Transformer(ViT)的方法规避了这一问题,但在缺乏详细跨模态对齐信息的情况下难以处理长视觉序列。本文提出一种基于ViT的VLP技术,通过一种新颖的补丁-文本对齐机制高效地融入对象信息。具体而言,我们将对象级信号转换为补丁级信号,并设计了一个补丁-文本对齐预训练任务(PTA)来学习一个文本感知的补丁检测器。通过仅在5%的训练图像中使用现成的精细对象标注,我们将PTA与其他常规VLP目标以端到端方式联合训练,绕开了目标检测的高计算成本,并得到一个有效的补丁检测器,能够准确检测与文本相关的补丁,从而大幅减少补丁序列并加速ViT主干内的计算。我们在多种广泛使用的基准上的实验表明,与先前的VLP模型相比,我们的方法实现了近88%的加速,同时在相似模型规模和数据的下游任务上保持了有竞争力或更优的性能。

英文摘要:

Vision-Language Pre-training (VLP) methods based on object detection enjoy the rich knowledge of fine-grained object-text alignment but at the cost of computationally expensive inference. Recent Visual-Transformer (ViT)-based approaches circumvent this issue while struggling with long visual sequences without detailed cross-modal alignment information. This paper introduces a ViT-based VLP technique that efficiently incorporates object information through a novel patch-text alignment mechanism. Specifically, we convert object-level signals into patch-level ones and devise a Patch-Text Alignment pre-training task (PTA) to learn a text-aware patch detector. By using off-the-shelf delicate object annotations in 5\% training images, we jointly train PTA with other conventional VLP objectives in an end-to-end manner, bypassing the high computational cost of object detection and yielding an effective patch detector that accurately detects text-relevant patches, thus considerably reducing patch sequences and accelerating computation within the ViT backbone. Our experiments on a variety of widely-used benchmarks reveal that our method achieves a speedup of nearly 88\% compared to prior VLP models while maintaining competitive or superior performance on downstream tasks with similar model size and data scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑