arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.04727cs.CLcs.AI
收藏

一种简单有效的消除多语言表征中自语言偏差的方法

A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations

Stanford University(斯坦福大学);Google Research(谷歌研究院)

Ziyi Yang, Yinfei Yang, Daniel Cer, Eric Darve

摘要:

语言无关和语义语言信息隔离是多语言表示模型的一个新兴研究方向。我们从几何代数和语义空间的角度来探讨这个问题。一种简单但高效的方法“语言信息去除(LIR)”从多语言表示中的语义相关组件中提取语言身份信息。LIR是一种后训练和模型不可知的方法,只使用简单的线性运算,例如矩阵分解和正交投影。LIR揭示了弱对齐多语言系统中,语义空间的主成分主要编码语言身份信息。我们首先评估LIR的跨语言问题答案检索任务(LAReQA),这需要多语言嵌入空间的强对齐。实验表明,LIR在此任务上非常有效,对于弱对齐模型,MAP的相对改善几乎达到100%。然后,我们在Amazon Reviews和XEVAL数据集上评估LIR,观察到去除语言信息能够提高跨语言迁移性能。

英文摘要:

Language agnostic and semantic-language information isolation is an emerging research direction for multilingual representations models. We explore this problem from a novel angle of geometric algebra and semantic space. A simple but highly effective method "Language Information Removal (LIR)" factors out language identity information from semantic related components in multilingual representations pre-trained on multi-monolingual data. A post-training and model-agnostic method, LIR only uses simple linear operations, e.g. matrix factorization and orthogonal projection. LIR reveals that for weak-alignment multilingual systems, the principal components of semantic spaces primarily encodes language identity information. We first evaluate the LIR on a cross-lingual question answer retrieval task (LAReQA), which requires the strong alignment for the multilingual embedding space. Experiment shows that LIR is highly effectively on this task, yielding almost 100% relative improvement in MAP for weak-alignment models. We then evaluate the LIR on Amazon Reviews and XEVAL dataset, with the observation that removing language information is able to improve the cross-lingual transfer performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑