arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.12847cs.CL
收藏

Morph Call:探索多语言Transformers的形态句法内容

Morph Call: Probing Morphosyntactic Content of Multilingual Transformers

SberDevices, Sberbank(SberDevices,俄罗斯联邦储蓄银行);HSE University(高等经济大学);Moscow Institute of Physics and Technology(莫斯科物理技术学院)

Vladislav Mikhailov, Oleg Serikov, Ekaterina Artemova

摘要:

基于transformer的语言模型在各种NLP和NLU任务上的出色表现激发了人们对探索其内部工作原理的兴趣。最近的研究主要集中在更高层次和复杂的语言现象,如句法,语义,世界知识和常识。大多数的研究都是以英语为中心的,对其他语言的了解很少,确切地说是他们的形态句法特性。为此,我们的工作提出了Morph Call,这是一套针对四种不同形态的印欧语言(英语,法语,德语和俄语)的46个探测任务。我们提出了一种新型的探测任务的基础上检测指导句子扰动。我们使用的神经元,层和代表级的内省技术相结合,分析四种多语言的Transformers,包括其较少探索蒸馏版本的形态句法内容。此外,我们还研究了POS标记的微调如何影响模型知识。结果表明,微调可以提高和降低探测性能,并改变如何在模型中分布的形态句法知识。代码和数据是公开的,我们希望填补Transformers研究较少方面的空白。

英文摘要:

The outstanding performance of transformer-based language models on a great variety of NLP and NLU tasks has stimulated interest in exploring their inner workings. Recent research has focused primarily on higher-level and complex linguistic phenomena such as syntax, semantics, world knowledge, and common sense. The majority of the studies are anglocentric, and little remains known regarding other languages, precisely their morphosyntactic properties. To this end, our work presents Morph Call, a suite of 46 probing tasks for four Indo-European languages of different morphology: English, French, German and Russian. We propose a new type of probing task based on the detection of guided sentence perturbations. We use a combination of neuron-, layer- and representation-level introspection techniques to analyze the morphosyntactic content of four multilingual transformers, including their less explored distilled versions. Besides, we examine how fine-tuning for POS-tagging affects the model knowledge. The results show that fine-tuning can improve and decrease the probing performance and change how morphosyntactic knowledge is distributed across the model. The code and data are publicly available, and we hope to fill the gaps in the less studied aspect of transformers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑