arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.12646cs.IR
收藏

MAKE:基于视觉语言预训练的淘宝商品检索

MAKE: Vision-Language Pre-training based Product Retrieval in Taobao Search

Xiaoyang Zheng, Zilong Wang, Ke Xu, Sen Li, Tao Zhuang, Qingwen Liu, Xiaoyi Zeng

摘要:

淘宝搜索分为两个阶段:检索阶段和排名阶段。给定一个用户查询,检索阶段返回一个候选产品的子集,用于下一个排名阶段。最近,预训练和微调的范式已经显示出它的潜力,将视觉线索的检索任务。本文主要研究淘宝搜索中文本到多模态检索的问题。我们认为,用户对标题或图像的关注因产品而异。因此,我们提出了一种新的跨模态融合的模态自适应模块,它有助于跨产品的文本和图像分配适当的权重。此外,在电子商务搜索中,用户查询往往是简短的,从而导致显着的语义不平衡之间的用户查询和产品标题。因此,我们设计了一个单独的文本编码器和关键字增强机制,以丰富查询表示和提高文本到多模态匹配。为此,我们提出了一种新的视觉语言(V+L)预训练方法,以利用(用户查询,产品标题,产品图像)的多模态信息。大量的实验表明,我们的检索特定的预训练模型(简称为MAKE)优于现有的V+L预训练方法的文本到多模态检索任务。MAKE已经上线,并对淘宝搜索的检索系统进行了重大改进。

英文摘要:

Taobao Search consists of two phases: the retrieval phase and the ranking phase. Given a user query, the retrieval phase returns a subset of candidate products for the following ranking phase. Recently, the paradigm of pre-training and fine-tuning has shown its potential in incorporating visual clues into retrieval tasks. In this paper, we focus on solving the problem of text-to-multimodal retrieval in Taobao Search. We consider that users' attention on titles or images varies on products. Hence, we propose a novel Modal Adaptation module for cross-modal fusion, which helps assigns appropriate weights on texts and images across products. Furthermore, in e-commerce search, user queries tend to be brief and thus lead to significant semantic imbalance between user queries and product titles. Therefore, we design a separate text encoder and a Keyword Enhancement mechanism to enrich the query representations and improve text-to-multimodal matching. To this end, we present a novel vision-language (V+L) pre-training methods to exploit the multimodal information of (user query, product title, product image). Extensive experiments demonstrate that our retrieval-specific pre-training model (referred to as MAKE) outperforms existing V+L pre-training methods on the text-to-multimodal retrieval task. MAKE has been deployed online and brings major improvements on the retrieval system of Taobao Search.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑