arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.15710cs.CL
收藏

Conan-embedding:基于更多更优负样本的通用文本嵌入

Conan-embedding: General Text Embedding with More and Better Negative Samples

Peking University(北京大学);Tencent Platform and Content Group(腾讯平台与内容事业群)

Shiyu Li, Yang Tang, Shizhe Chen, Xi Chen

AI总结:

本文提出 conan-embedding 模型,通过动态困难负样本挖掘、跨GPU平衡损失及利用LLM的prompt-response对,最大化利用更多更优负样本,有效提升文本嵌入能力并登顶MTEB中文榜首。

摘要:

随着 RAG 的日益普及,embedding 模型的能力正受到越来越多的关注。embedding 模型主要通过对比损失学习进行训练,其中负样本是关键组成部分。以往的工作提出了各种困难负样本挖掘策略,但这些策略通常作为预处理步骤使用。在本文中,我们提出了 conan-embedding 模型,以最大化利用更多且更高质量的负样本。具体而言,由于模型处理预处理负样本的能力在训练过程中不断演变,我们提出了动态困难负样本挖掘方法,使模型在整个训练过程中接触到更具挑战性的负样本。其次,对比学习需要尽可能多的负样本,但受到 GPU 显存的限制。因此,我们使用 Cross-GPU balancing Loss 为 embedding 训练提供更多负样本,并平衡多个任务间的 batch size。此外,我们还发现 LLMs 的 prompt-response 对可用于 embedding 训练。我们的方法有效增强了 embedding 模型的能力,目前在 Massive text embedding benchmark 的中文排行榜上排名第一。

英文摘要:

With the growing popularity of RAG, the capabilities of embedding models are gaining increasing attention. Embedding models are primarily trained through contrastive loss learning, with negative examples being a key component. Previous work has proposed various hard negative mining strategies, but these strategies are typically employed as preprocessing steps. In this paper, we propose the conan-embedding model, which maximizes the utilization of more and higher-quality negative examples. Specifically, since the model's ability to handle preprocessed negative examples evolves during training, we propose dynamic hard negative mining method to expose the model to more challenging negative examples throughout the training process. Secondly, contrastive learning requires as many negative examples as possible but is limited by GPU memory constraints. Therefore, we use a Cross-GPU balancing Loss to provide more negative examples for embedding training and balance the batch size across multiple tasks. Moreover, we also discovered that the prompt-response pairs from LLMs can be used for embedding training. Our approach effectively enhances the capabilities of embedding models, currently ranking first on the Chinese leaderboard of Massive text embedding benchmark

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑