MTEB-French：用于法语句嵌入评估与分析的资源
MTEB-French: Resources for French Sentence Embedding Evaluation and Analysis
AI总结：
本文扩展 MTEB 构建首个法语大规模句嵌入基准，整合并新建数据集，评估 51 个模型在 8 类任务上的表现并分析模型特征与性能的关系。
摘要：
近来，大量嵌入模型已经出现，并被广泛用于各类 NLP 任务。Massive Text Embedding Benchmark（MTEB，大规模文本嵌入基准）主要简化了选择在英语多项任务上表现良好的模型的过程，但其向其他语言的扩展仍具挑战。为此，我们扩展 MTEB，提出首个面向法语的大规模句嵌入基准。我们在一个易于使用的界面中汇集 15 个现有数据集，并新建 3 个法语数据集，以覆盖 8 个任务类别的综合评估。我们大规模比较了 51 个经过精心筛选的嵌入模型，开展全面的统计检验，并分析模型性能与其众多特征之间的相关性。我们发现，即使没有模型在所有任务上均为最佳，在句相似度任务上预训练的大型多语言模型表现格外出色。本工作附带开源代码、新数据集和公开排行榜。
英文摘要：
Recently, numerous embedding models have been made available and widely used for various NLP tasks. The Massive Text Embedding Benchmark (MTEB) has primarily simplified the process of choosing a model that performs well for several tasks in English, but extensions to other languages remain challenging. This is why we expand MTEB to propose the first massive benchmark of sentence embeddings for French. We gather 15 existing datasets in an easy-to-use interface and create three new French datasets for a global evaluation of 8 task categories. We compare 51 carefully selected embedding models on a large scale, conduct comprehensive statistical tests, and analyze the correlation between model performance and many of their characteristics. We find out that even if no model is the best on all tasks, large multilingual models pre-trained on sentence similarity perform exceptionally well. Our work comes with open-source code, new datasets and a public leaderboard.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
