arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2103.08784cs.CLcs.CV
收藏

LightningDOT:用于实时的图像-文本检索的预训练视觉语义嵌入

LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval

Microsoft Corporation(微软公司)

Siqi Sun, Yen-Chun Chen, Linjie Li, Shuohang Wang, Yuwei Fang, Jingjing Liu

摘要:

多模式预训练推动了视觉和语言研究的巨大进步。这些大规模的预训练模型虽然成功,但由于Transformer架构中的跨模态注意力导致的巨大计算成本,导致推理速度缓慢。当应用于实际应用时,这种延迟和计算需求严重阻碍了预训练模型的实际使用。在本文中,我们研究了图像文本检索(ITR),V+L应用程序中最成熟的场景,甚至在最近的预训练模型出现之前就已经被广泛研究。我们提出了一种简单而高效的方法,LightningDOT,它可以将ITR的推理时间加快数千倍,而不会牺牲准确性。LightningDOT通过对三个新的学习目标进行预训练,离线提取特征索引,并采用即时点积匹配和进一步重新排序来消除耗时的跨模态注意力,从而显着加快检索过程。事实上,LightningDOT在多个ITR基准测试(如Flickr 30 k、COCO和Multi 30 K)中达到了最新的技术水平,优于现有的预训练模型,这些模型消耗了1000倍的计算时间。代码和培训前检查点可在https://github.com/intersun/LightningDOT上找到。

英文摘要:

Multimodal pre-training has propelled great advancement in vision-and-language research. These large-scale pre-trained models, although successful, fatefully suffer from slow inference speed due to enormous computation cost mainly from cross-modal attention in Transformer architecture. When applied to real-life applications, such latency and computation demand severely deter the practical use of pre-trained models. In this paper, we study Image-text retrieval (ITR), the most mature scenario of V+L application, which has been widely studied even prior to the emergence of recent pre-trained models. We propose a simple yet highly effective approach, LightningDOT that accelerates the inference time of ITR by thousands of times, without sacrificing accuracy. LightningDOT removes the time-consuming cross-modal attention by pre-training on three novel learning objectives, extracting feature indexes offline, and employing instant dot-product matching with further re-ranking, which significantly speeds up retrieval process. In fact, LightningDOT achieves new state of the art across multiple ITR benchmarks such as Flickr30k, COCO and Multi30K, outperforming existing pre-trained models that consume 1000x magnitude of computational hours. Code and pre-training checkpoints are available at https://github.com/intersun/LightningDOT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑