通过多方向平行文本对齐增强多语言嵌入
Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment
AI总结：
通过多方向平行文本对齐提升多语言嵌入，增强跨语言表示性能。
摘要：
多语言预训练通常缺乏显式的对齐信号，导致表示空间中的跨语言对齐效果不佳。在本工作中，我们展示出通过在多种语言的多样化语料库中训练标准预训练模型以实现跨语言对齐，可以显著提升多语言和跨语言表示，以用于自然语言理解任务。我们使用一种现成的NMT模型将英文文本翻译成六种目标语言，构建了一个多方向平行数据集，并通过对比学习实现了强大的跨语言对齐。这在MTEB基准测试中，对于XLM-Roberta和多语言BERT基础模型，无论是已见还是未见语言，都带来了显著的性能提升。使用多方向平行语料进行对比训练，在双语文本挖掘（21.3%）、语义相似性（5.3%）和分类（28.4%）方面，相比以英语为中心（En-X）的双语平行数据，表现更优，其中X是从多种目标语言中采样的。此外，对mE5模型在小数据集上进行微调，使用多方向平行性显著提升了双语文本挖掘的效果，这凸显了即使对于已经预训练出高质量句子嵌入的模型，多方向跨语言监督的重要性。
英文摘要：
Multilingual pretraining typically lacks explicit alignment signals, leading to suboptimal cross-lingual alignment in the representation space. In this work, we show that training standard pretrained models for cross-lingual alignment with a multi-way parallel corpus in a diverse pool of languages can substantially improve multilingual and cross-lingual representations for NLU tasks. We construct a multi-way parallel dataset using translations of English text from an off-the-shelf NMT model for a pool of six target languages and achieve strong cross-lingual alignment through contrastive learning. This leads to substantial performance gains across both seen and unseen languages for multiple tasks from the MTEB benchmark evaluated for XLM-Roberta and multilingual BERT base models. Using a multi-way parallel corpus for contrastive training yields substantial gains on bitext mining (21.3%), semantic similarity (5.3%), and classification (28.4%) compared to English-centric (En-X) bilingually parallel data, where X is sampled from a pool of multiple target languages. Furthermore, finetuning mE5 model on a small dataset with multi-way parallelism significantly improves bitext mining compared to one without, underscoring the importance of multi-way cross-lingual supervision even for models already pretrained for high-quality sentence embeddings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
