arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.12737cs.CLcs.AI
收藏

MEXMA:Token级别目标改进句子表示

MEXMA: Token-level objectives improve sentence representations

Meta AI;Sorbonne Université(索邦大学);Criteo AI Lab(Criteo人工智能实验室)

João Maria Janeiro, Benjamin Piwowarski, Patrick Gallinari, Loïc Barrault

AI总结:

提出MEXMA方法,结合句子级和token级目标,通过跨语言掩码token预测直接更新编码器,显著提升跨语言句子表示质量,在双语文本挖掘等任务上优于现有方法。

摘要:

当前的预训练跨语言句子编码器方法仅使用句子级别目标。这可能导致信息丢失,特别是对于token,进而降低句子表示质量。我们提出MEXMA,一种结合句子级别和token级别目标的新方法。一种语言的句子表示用于预测另一种语言的掩码token,句子表示和所有token都直接更新编码器。我们表明,添加token级别目标在多项任务中显著提升了句子表示质量。我们的方法在双语文本挖掘及多个下游任务上优于当前的预训练跨语言句子编码器。我们还分析了token中编码的信息,以及句子表示如何从中构建。

英文摘要:

Current pre-trained cross-lingual sentence encoders approaches use sentence-level objectives only. This can lead to loss of information, especially for tokens, which then degrades the sentence representation. We propose MEXMA, a novel approach that integrates both sentence-level and token-level objectives. The sentence representation in one language is used to predict masked tokens in another language, with both the sentence representation and all tokens directly updating the encoder. We show that adding token-level objectives greatly improves the sentence representation quality across several tasks. Our approach outperforms current pre-trained cross-lingual sentence encoders on bi-text mining as well as several downstream tasks. We also analyse the information encoded in our tokens, and how the sentence representation is built from them.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑