arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.02853cs.CV
收藏

学习实例级表示用于电商领域大规模多模态预训练

Learning Instance-Level Representation for Large-Scale Multi-Modal Pretraining in E-commerce

Peking University(北京大学);ByteDance Inc.(字节跳动公司)

Yang Jin, Yongzhi Li, Zehuan Yuan, Yadong Mu

AI总结:

本文提出了一种名为 ECLIP 的以实例为中心的多模态预训练范式,通过引入可学习的实例查询和两个无需昂贵标注的 pretext 任务,在 1 亿电商数据上预训练,在多项下游任务中无需微调即大幅超越现有方法,展现出强大的可迁移性。

摘要:

本文旨在建立一个通用的多模态基础模型,该模型具备扩展至电商领域海量下游应用的能力。近期,大规模视觉-语言预训练方法在通用领域取得了显著进展。然而,由于自然图像与商品图像之间存在显著差异,直接将这些用于建模图像级表示的框架应用于电商领域将不可避免地是次优的。为此,我们在本工作中提出了一种以实例为中心的多模态预训练范式,称为 ECLIP。具体而言,我们精心设计了一个解码器架构,引入了一组可学习的实例查询,以显式地聚合实例级语义。此外,为了使模型能够在不依赖昂贵人工标注的情况下聚焦于目标商品实例,我们进一步提出了两个专门配置的 pretext 任务。在 1 亿条电商相关数据上进行预训练后,ECLIP 成功提取了更具通用性、语义丰富且鲁棒的表示。大量实验结果表明,无需进一步微调,ECLIP 在广泛的下游任务上大幅超越了现有方法,展示了对真实世界电商应用的强大可迁移性。

英文摘要:

This paper aims to establish a generic multi-modal foundation model that has the scalable capability to massive downstream applications in E-commerce. Recently, large-scale vision-language pretraining approaches have achieved remarkable advances in the general domain. However, due to the significant differences between natural and product images, directly applying these frameworks for modeling image-level representations to E-commerce will be inevitably sub-optimal. To this end, we propose an instance-centric multi-modal pretraining paradigm called ECLIP in this work. In detail, we craft a decoder architecture that introduces a set of learnable instance queries to explicitly aggregate instance-level semantics. Moreover, to enable the model to focus on the desired product instance without reliance on expensive manual annotations, two specially configured pretext tasks are further proposed. Pretrained on the 100 million E-commerce-related data, ECLIP successfully extracts more generic, semantic-rich, and robust representations. Extensive experimental results show that, without further fine-tuning, ECLIP surpasses existing methods by a large margin on a broad range of downstream tasks, demonstrating the strong transferability to real-world E-commerce applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑