arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.17016cs.CLcs.AIcs.IR
收藏

面向8192 Token双语文本嵌入的多任务对比学习

Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings

Jina AI GmbH(吉纳人工智能有限公司)
Isabelle Mohr, Markus Krimmel, Saba Sturua, Mohammad Kalim Akram, Andreas Koukounas, Michael Günther, Georgios Mastrapas, Vinit Ravishankar, Joan Fontanals Mart… 展开作者

Isabelle Mohr, Markus Krimmel, Saba Sturua, Mohammad Kalim Akram, Andreas Koukounas, Michael Günther, Georgios Mastrapas, Vinit Ravishankar, Joan Fontanals Martínez, Feng Wang, Qi Liu, Ziniu Yu, Jie Fu, Saahil Ognawala, Susana Guzman, Bo Wang, Maximilian Werk, Nan Wang, Han Xiao

AI总结:

本文提出支持8192 token长文本的双语嵌入模型,通过多任务对比学习提升跨语言语义相似度性能,并扩展MTEB基准以覆盖德语和西班牙语。

摘要:

我们提出了一套新颖的最先进的双语文本嵌入模型,旨在支持英语和另一种目标语言。这些模型能够处理长达8192个token的长文本输入,使其在文本检索、聚类和语义文本相似度(STS)计算等一系列自然语言处理任务中具有高度通用性。通过专注于双语模型并引入独特的多任务学习目标,我们显著提升了模型在STS任务上的性能,在目标语言理解和跨语言评估任务中均优于现有 multilingual 模型的能力。此外,我们的双语模型更加高效,由于词汇表需求更小,所需参数更少、内存占用更低。我们还扩展了 Massive Text Embedding Benchmark(MTEB),纳入了德语和西班牙语嵌入模型的基准测试。这一整合旨在推动这些语言的文本嵌入技术进一步研究与发展。

英文摘要:

We introduce a novel suite of state-of-the-art bilingual text embedding models that are designed to support English and another target language. These models are capable of processing lengthy text inputs with up to 8192 tokens, making them highly versatile for a range of natural language processing tasks such as text retrieval, clustering, and semantic textual similarity (STS) calculations. By focusing on bilingual models and introducing a unique multi-task learning objective, we have significantly improved the model performance on STS tasks, which outperforms the capabilities of existing multilingual models in both target language understanding and cross-lingual evaluation tasks. Moreover, our bilingual models are more efficient, requiring fewer parameters and less memory due to their smaller vocabulary needs. Furthermore, we have expanded the Massive Text Embedding Benchmark (MTEB) to include benchmarks for German and Spanish embedding models. This integration aims to stimulate further research and advancement in text embedding technologies for these languages.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑