在多语言嵌入中评估跨语言语义对齐的基准测试
Benchmarking Cross-Lingual Semantic Alignment in Multilingual Embeddings
AI总结：
本研究提出语义亲和力指标，评估多语言嵌入的跨语言语义对齐，发现训练目标而非规模决定对齐效果，强调显式翻译监督的重要性。
摘要：
在数百种多语言嵌入模型可用的情况下，从业者缺乏明确的指导来区分哪些模型提供了真正的跨语言语义对齐，而不是通过语言特定的模式实现任务性能。任务驱动的基准测试（MTEB）可能会掩盖基本的对齐缺陷。我们引入了语义亲和力（SA），一种介于0和1之间的度量标准，通过余弦距离测量跨语言到内语言的扩散比，并结合PHATE可视化在我们的Semanscope框架中使用。在13种模型上跨4个数据集（52次实验）进行基准测试，揭示出三级结构：（1）顶级BERT模型（LaBSE SA = 0.70，USE SA = 0.68，S-BERT SA = 0.68）通过翻译对监督实现强大的对齐；（2）LLM嵌入在SA介于0.55到0.61之间稳定，无论模型规模在0.6 B到8 B之间；（3）仅使用MLM的BERT模型（mBERT、XLM-R、SA < 0.50）即使在超过100种语言训练后也失败。训练目标，而不是架构或规模，决定了对齐。Oracle Bone原始（1200 BCE）暴露了语义漂移——模型学习的是语料库模式而不是认知原始。本工作提供了语义基准测试，帮助从业者从数百种可用模型中选择高质量的多语言嵌入，显示跨语言对齐需要显式的翻译监督，而不是仅仅模型规模或多语言数据。
英文摘要：
With hundreds of multilingual embedding models available, practitioners lack clear guidance on which provide genuine cross-lingual semantic alignment versus task performance through language-specific patterns. Task-driven benchmarks (MTEB) may mask fundamental alignment shortcomings. We introduce Semantic Affinity (SA), a bounded (between 0 and 1) metric measuring inter-lingual to intra-lingual spread ratio using cosine distance, combined with PHATE visualization in our Semanscope framework. Benchmarking 13 models across 4 datasets (52 experiments) reveals a three-tier structure: (1) Top BERT models (LaBSE SA = 0.70, USE SA = 0.68, S-BERT SA = 0.68) achieve strong alignment via translation-pair supervision; (2) LLM embeddings plateau at SA between 0.55 and 0.61 regardless of 0.6 B to 8 B scale; (3) MLM-only BERT models (mBERT, XLM-R, SA < 0.50) fail despite more than 100 language training. Training objective, not architecture or scale, determines alignment. Oracle Bone primitives (1200 BCE) expose semantic drift-models learn corpus patterns rather than cognitive primitives. This work provides semantic benchmarking to help practitioners select quality multilingual embeddings from hundreds of available models, showing cross-lingual alignment requires explicit translation supervision, not merely model scale or multilingual data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
