跨语言句子嵌入的双重对齐预训练
Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
AI总结：
本文提出双重对齐预训练（DAP）框架，通过结合句子级和词元级对齐，引入表示翻译学习（RTL）任务，显著提升了跨语言句子嵌入的效果。
摘要：
最近的研究表明，使用句子级翻译排序任务训练的双编码器模型是跨语言句子嵌入的有效方法。然而，我们的研究表明，词元级对齐在多语言场景中同样至关重要，而这一点在先前的研究中尚未得到充分探索。基于我们的发现，我们提出了一种用于跨语言句子嵌入的双重对齐预训练（DAP）框架，该框架同时结合了句子级和词元级对齐。为此，我们引入了一种新颖的表示翻译学习（RTL）任务，模型学习使用单侧上下文化的词元表示来重构其对应的翻译。这种重构目标促使模型将翻译信息嵌入到词元表示中。与翻译语言建模等其他词元级对齐方法相比，RTL更适合双编码器架构，并且计算效率高。在三个句子级跨语言基准上的广泛实验表明，我们的方法能够显著改善句子嵌入。
英文摘要：
Recent studies have shown that dual encoder models trained with the sentence-level translation ranking task are effective methods for cross-lingual sentence embedding. However, our research indicates that token-level alignment is also crucial in multilingual scenarios, which has not been fully explored previously. Based on our findings, we propose a dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding that incorporates both sentence-level and token-level alignment. To achieve this, we introduce a novel representation translation learning (RTL) task, where the model learns to use one-side contextualized token representation to reconstruct its translation counterpart. This reconstruction objective encourages the model to embed translation information into the token representation. Compared to other token-level alignment methods such as translation language modeling, RTL is more suitable for dual encoder architectures and is computationally efficient. Extensive experiments on three sentence-level cross-lingual benchmarks demonstrate that our approach can significantly improve sentence embedding. Our code is available at https://github.com/ChillingDream/DAP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
