arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.03544cs.CV
收藏

PinCLIP:Pinterest的规模化多模态基础表示

PinCLIP: Large-scale Foundational Multimodal Representation at Pinterest

Pinterest Inc.(Pinterest公司)

Josh Beal, Eric Kim, Jinfeng Rao, Rex Wu, Dmitry Kislyuk, Charles Rosenberg

AI总结:

PinCLIP通过多模态表示学习提升Pinterest的检索和排序模型,解决冷启动问题并提高用户参与度。

摘要:

尽管多模态视觉语言模型(VLMs)在各个领域都取得了显著成功,但将VLMs整合到推荐和检索系统中仍是一个挑战,原因包括训练目标不一致和服务效率瓶颈。本文介绍了PinCLIP,一种大规模的视觉表示学习方法,旨在通过利用VLMs来学习图像-文本对齐,从而增强Pinterest的检索和排序模型。我们提出了一种新的混合视觉Transformer架构,该架构利用VLM骨干网络和混合融合机制,以不同粒度捕捉多模态内容表示。除了标准的图像到文本对齐目标外,我们还引入了一个邻居对齐目标,以建模Pinterest Pin-Board图中的多模态表示交叉融合。离线评估显示,PinCLIP在多模态检索任务中优于最先进的基线,如Qwen,提高了20%。在线A/B测试显示了显著的商业影响,包括在Pinterest的所有主要表面上实现了显著的参与度提升。值得注意的是,PinCLIP显著解决了“冷启动”问题,通过增加15%的有机内容重新发布和新广告的8.7%更高的点击率,增强了新鲜内容的分发。

英文摘要:

While multi-modal Visual Language Models (VLMs) have demonstrated significant success across various domains, the integration of VLMs into recommendation and retrieval systems remains a challenge, due to issues like training objective discrepancies and serving efficiency bottlenecks. This paper introduces PinCLIP, a large-scale visual representation learning approach developed to enhance retrieval and ranking models at Pinterest by leveraging VLMs to learn image-text alignment. We propose a novel hybrid Vision Transformer architecture that utilizes a VLM backbone and a hybrid fusion mechanism to capture multi-modality content representation at varying granularities. Beyond standard image-to-text alignment objectives, we introduce a neighbor alignment objective to model the cross-fusion of multi-modal representations within the Pinterest Pin-Board graph. Offline evaluations show that PinCLIP outperforms state-of-the-art baselines, such as Qwen, by 20% in multi-modal retrieval tasks. Online A/B testing demonstrates significant business impact, including substantial engagement gains across all major surfaces in Pinterest. Notably, PinCLIP significantly addresses the "cold-start" problem, enhancing fresh content distribution with a 15% Repin increase in organic content and 8.7% higher click for new Ads.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑