EMMA-X：一种用于跨语言表示学习的类EM多语言预训练算法
EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learning
AI总结：
本文提出类EM多语言预训练算法EMMA-X，通过在海量非平行语料上统一跨语言表示学习与语义关系预测任务并相互监督，在XRETE基准上取得最先进性能。
摘要：
表达所有语言共有的通用语义有助于理解复杂且具有文化特定性的句子的含义。该场景下的研究主题聚焦于利用大规模平行语料学习跨语言的通用表示。然而，由于平行数据的稀疏性和稀缺性，学习任意两种语言之间真实的“通用语义”仍然面临巨大挑战。本文提出EMMA-X：一种类EM的多语言预训练算法，借助海量多语言非平行数据学习跨语言通用表示。EMMA-X在EM框架内统一了跨语言表示学习任务与一个额外的语义关系预测任务。额外的语义分类器和跨语言句子编码器共同近似两个句子的语义关系，并相互监督直至收敛。为评估EMMA-X，我们在XRETE上进行了实验，XRETE是一个新提出的基准，包含12个被广泛研究的、完全依赖句子级表示的跨语言任务。结果表明EMMA-X取得了最先进的性能。对构建的表示空间进行进一步几何分析，并以三项要求加以验证，证明了EMMA-X相对于先进模型的优越性。
英文摘要：
Expressing universal semantics common to all languages is helpful in understanding the meanings of complex and culture-specific sentences. The research theme underlying this scenario focuses on learning universal representations across languages with the usage of massive parallel corpora. However, due to the sparsity and scarcity of parallel data, there is still a big challenge in learning authentic ``universals'' for any two languages. In this paper, we propose EMMA-X: an EM-like Multilingual pre-training Algorithm, to learn (X)Cross-lingual universals with the aid of excessive multilingual non-parallel data. EMMA-X unifies the cross-lingual representation learning task and an extra semantic relation prediction task within an EM framework. Both the extra semantic classifier and the cross-lingual sentence encoder approximate the semantic relation of two sentences, and supervise each other until convergence. To evaluate EMMA-X, we conduct experiments on XRETE, a newly introduced benchmark containing 12 widely studied cross-lingual tasks that fully depend on sentence-level representations. Results reveal that EMMA-X achieves state-of-the-art performance. Further geometric analysis of the built representation space with three requirements demonstrates the superiority of EMMA-X over advanced models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
