基于去噪的UNMT比基于MASS的UNMT对词序分歧具有更好的鲁棒性
Denoising-based UNMT is more robust to word-order divergence than MASS-based UNMT
摘要：
我们的目标是研究具有自我监督预训练的UNMT方法是否对语言对之间的语序分歧具有鲁棒性。我们通过比较两个具有相同自监督预训练目标的预训练模型来实现这一点。第一个模型在具有不同词序的语言对上训练，第二个模型在相同的语言对上训练，其中源语言被重新排序以匹配目标语言的词序。理想情况下，对词序分歧具有鲁棒性的UNMT方法在两种配置之间不应表现出明显的性能差异。在本文中，我们研究了两种基于自监督预训练的UNMT方法，即掩蔽序列到序列预训练（MASS）（没有洗牌噪声）和去噪自动编码器（DAE）（有洗牌噪声）。我们用五个英语$\rightarrow$印度语语言对进行实验，即，en-hi、en-bn、en-gu、en-kn和en-ta），其中源语言的词序是SVO（主语-动词-宾语），而目标语言的词序是SOV（主语-宾语-动词）。我们观察到，对于这些语言对，基于DAE的UNMT方法在翻译准确性方面始终优于MASS。此外，使用重新排序来弥合词序差距提高了基于MASS的UNMT模型的翻译准确性，而它不能提高基于DAE的UNMT模型的翻译准确性。这一观察结果表明，基于DAE的UNMT比基于MASS的UNMT对词序分歧更鲁棒。DAE方法中的混洗噪声可能是该方法对词序发散具有鲁棒性的原因。
英文摘要：
We aim to investigate whether UNMT approaches with self-supervised pre-training are robust to word-order divergence between language pairs. We achieve this by comparing two models pre-trained with the same self-supervised pre-training objective. The first model is trained on language pairs with different word-orders, and the second model is trained on the same language pairs with source language re-ordered to match the word-order of the target language. Ideally, UNMT approaches which are robust to word-order divergence should exhibit no visible performance difference between the two configurations. In this paper, we investigate two such self-supervised pre-training based UNMT approaches, namely Masked Sequence-to-Sequence Pre-Training, (MASS) (which does not have shuffling noise) and Denoising AutoEncoder (DAE), (which has shuffling noise). We experiment with five English$\rightarrow$Indic language pairs, i.e., en-hi, en-bn, en-gu, en-kn, and en-ta) where word-order of the source language is SVO (Subject-Verb-Object), and the word-order of the target languages is SOV (Subject-Object-Verb). We observed that for these language pairs, DAE-based UNMT approach consistently outperforms MASS in terms of translation accuracies. Moreover, bridging the word-order gap using reordering improves the translation accuracy of MASS-based UNMT models, while it cannot improve the translation accuracy of DAE-based UNMT models. This observation indicates that DAE-based UNMT is more robust to word-order divergence than MASS-based UNMT. Word-shuffling noise in DAE approach could be the possible reason for the approach being robust to word-order divergence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
