神经语言习得与自然语言习得相似吗？按时间顺序的探索性研究
Is neural language acquisition similar to natural? A chronological probing study
摘要：
探测方法允许人们使用外部分类器和统计分析来获得存储在神经网络的内层中的语言现象的部分表示。预训练的基于transformer的语言模型被广泛用于自然语言理解（NLU）和自然语言生成（NLG）任务，使得它们最常用于下游应用。然而，几乎没有进行分析，无论是模型预先训练足够或包含与语言理论相关的知识。我们正在介绍Transformer英文模型的时间顺序探索研究，如MultiBERT和T5。在语料库的训练过程中，我们依次比较了模型学习到的语言信息。结果表明：1）语言信息是在训练的早期阶段获得的; 2）两种语言模型都表现出从语言的各个层面捕捉各种特征的能力，包括词法、句法甚至话语，但它们也可能在被认为很容易的任务上失败。我们还介绍了开放源代码框架的时间顺序探测研究，与其他基于transformer的模型兼容。https://github.com/EkaterinaVoloshina/chronological_probing
英文摘要：
The probing methodology allows one to obtain a partial representation of linguistic phenomena stored in the inner layers of the neural network, using external classifiers and statistical analysis. Pre-trained transformer-based language models are widely used both for natural language understanding (NLU) and natural language generation (NLG) tasks making them most commonly used for downstream applications. However, little analysis was carried out, whether the models were pre-trained enough or contained knowledge correlated with linguistic theory. We are presenting the chronological probing study of transformer English models such as MultiBERT and T5. We sequentially compare the information about the language learned by the models in the process of training on corpora. The results show that 1) linguistic information is acquired in the early stages of training 2) both language models demonstrate capabilities to capture various features from various levels of language, including morphology, syntax, and even discourse, while they also can inconsistently fail on tasks that are perceived as easy. We also introduce the open-source framework for chronological probing research, compatible with other transformer-based models. https://github.com/EkaterinaVoloshina/chronological_probing
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
