建模序列句子关系以改进跨语言稠密检索
Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrieval
AI总结：
针对跨语言稠密检索缺乏定制化多语言预训练模型的问题，该研究提出MSM模型，通过建模跨语言通用的序列句子关系，结合掩码句子预测任务与分层对比损失训练，显著提升了跨语言检索性能。
摘要：
近年来，mBERT、XLM-R等多语言预训练语言模型（PLM）在跨语言稠密检索领域取得了显著进展。尽管成效显著，但这类模型属于通用PLM，针对跨语言检索定制的多语言PLM仍有待探索。受平行文档中句子顺序大致一致这一跨语言普遍现象的启发，我们提出对这种序列句子关系进行建模，以助力跨语言表示学习。具体而言，我们提出了名为masked sentence model（掩码句子模型，MSM）的多语言PLM，它由用于生成句子表示的句子编码器，以及作用于文档中句子向量序列的文档编码器组成。文档编码器在所有语言间共享，以建模跨语言的通用序列句子关系。为训练该模型，我们提出了掩码句子预测任务，通过带采样负例的分层对比损失来掩码并预测句子向量。在四项跨语言检索任务上的综合实验表明，MSM显著优于现有的先进预训练模型，证明了我们方法的有效性及更强的跨语言检索能力。代码和模型可在https://github.com/shunyuzh/MSM获取。
英文摘要：
Recently multi-lingual pre-trained language models (PLM) such as mBERT and XLM-R have achieved impressive strides in cross-lingual dense retrieval. Despite its successes, they are general-purpose PLM while the multilingual PLM tailored for cross-lingual retrieval is still unexplored. Motivated by an observation that the sentences in parallel documents are approximately in the same order, which is universal across languages, we propose to model this sequential sentence relation to facilitate cross-lingual representation learning. Specifically, we propose a multilingual PLM called masked sentence model (MSM), which consists of a sentence encoder to generate the sentence representations, and a document encoder applied to a sequence of sentence vectors from a document. The document encoder is shared for all languages to model the universal sequential sentence relation across languages. To train the model, we propose a masked sentence prediction task, which masks and predicts the sentence vector via a hierarchical contrastive loss with sampled negatives. Comprehensive experiments on four cross-lingual retrieval tasks show MSM significantly outperforms existing advanced pre-training models, demonstrating the effectiveness and stronger cross-lingual retrieval capabilities of our approach. Code and model are available at https://github.com/shunyuzh/MSM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
