arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.08802cs.CLcs.CVcs.IR
收藏

jina-clip-v2:面向文本和图像的多语言多模态嵌入

jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images

Jina AI GmbH(Jina AI公司)

Andreas Koukounas, Georgios Mastrapas, Sedigheh Eslami, Bo Wang, Mohammad Kalim Akram, Michael Günther, Isabelle Mohr, Saba Sturua, Nan Wang, Han Xiao

AI总结:

提出jina-clip-v2多语言多模态嵌入模型,通过多任务多阶段对比学习,支持文本和图像任务,在零样本检索和语义相似性上超越现有CLIP模型。

摘要:

对比语言-图像预训练(CLIP)已被广泛用于跨模态信息检索和多模态理解任务。然而,CLIP模型主要针对跨模态视觉-语言任务进行优化,在单模态文本任务中表现不佳。此外,这些模型通常在英文数据集上训练,因此缺乏多语言理解能力。另外,从视觉理解的角度来看,以往的基于CLIP的模型对视觉丰富的文档理解不足。在这项工作中,我们提出了jina-clip-v2,一种对比视觉-语言模型,通过多任务和多阶段的对比学习范式,在文本对、三元组和图像-文本对上训练,以支持纯文本和跨模态任务。我们采用多语言文本编码器,并将训练数据集扩展到包含来自29种非英语语言的多语言文本,包括印地语、中文、德语、法语等,以及视觉丰富文档的图像。我们评估了模型的性能,并表明jina-clip-v2在零样本纯文本检索、语义文本相似性和跨模态检索任务中,在英语和多语言设置下,相较于最先进的基于CLIP的模型取得了显著改进。jina-clip-v2还提供了嵌入维度的灵活性,使用户能够选择表示的粒度。jina-clip-v2可在https://huggingface.co/jinaai/jina-clip-v2公开获取。

英文摘要:

Contrastive Language-Image Pretraining (CLIP) has been widely used for crossmodal information retrieval and multimodal understanding tasks. However, CLIP models are mainly optimized for crossmodal vision-language tasks and underperform in single-mode text tasks. Moreover, these models are often trained on English datasets and therefore lack multilingual understanding. Additionally, from a visual understanding perspective, previous CLIP-based models exhibit insufficient understanding of visually rich documents. In this work, we propose jina-clip-v2, a contrastive vision-language model trained on text pairs, triplets and image-text pairs via a multi-task and multi-stage contrastive learning paradigm in order to support both text-only and crossmodal tasks. We employ a multilingual text encoder and expand the training dataset to include multilingual texts from 29 non-English languages, including Hindi, Chinese, German, French, and others, as well as images of visually rich documents. We evaluate the model's performance and show that jina-clip-v2 achieves notable improvements over state-of-the-art CLIP-based models in zero-shot text-only retrieval, semantic textual similarity, and crossmodal retrieval tasks in both English and multilingual settings. jina-clip-v2 also provides for flexibility in embedding dimensionality, enabling users to select the granularity of the representations. jina-clip-v2 is publicly available at https://huggingface.co/jinaai/jina-clip-v2.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑