arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.11978cs.LGcs.CL
收藏

深度学习能学会抽象吗?一个系统的探究框架

Does Deep Learning Learn to Abstract? A Systematic Probing Framework

Microsoft Corporation(微软公司)

Shengnan An, Zeqi Lin, Bei Chen, Qiang Fu, Nanning Zheng, Jian-Guang Lou

摘要:

抽象是深度学习模型的理想能力,这意味着从具体实例中归纳抽象概念,并在学习上下文之外灵活地应用它们。与此同时,人们对深度学习模型中这种能力的存在和进一步特征缺乏清晰的了解。在本文中,我们引入了一个系统的探测框架,从可移植性的角度探索深度学习模型的抽象能力。基于该框架进行了一组对照实验,有力地证明了两种预训练语言模型(PLMs)T5和GPT 2具有抽象能力。我们还进行了深入的分析,从而进一步揭示:(1)整个训练阶段表现出一个“记忆-然后-抽象”的两阶段过程:(2)学习的抽象概念集中在几个中间层的注意头,而不是均匀分布在整个模型;(3)被探测的抽象能力对概念突变具有鲁棒性,并且对低层/源端突变的鲁棒性强于对高层/目标端突变的鲁棒性;(4)通用预训练对于抽象能力的出现至关重要,并且PLM在更大的模型大小和数据规模下表现出更好的抽象。

英文摘要:

Abstraction is a desirable capability for deep learning models, which means to induce abstract concepts from concrete instances and flexibly apply them beyond the learning context. At the same time, there is a lack of clear understanding about both the presence and further characteristics of this capability in deep learning models. In this paper, we introduce a systematic probing framework to explore the abstraction capability of deep learning models from a transferability perspective. A set of controlled experiments are conducted based on this framework, providing strong evidence that two probed pre-trained language models (PLMs), T5 and GPT2, have the abstraction capability. We also conduct in-depth analysis, thus shedding further light: (1) the whole training phase exhibits a "memorize-then-abstract" two-stage process; (2) the learned abstract concepts are gathered in a few middle-layer attention heads, rather than being evenly distributed throughout the model; (3) the probed abstraction capabilities exhibit robustness against concept mutations, and are more robust to low-level/source-side mutations than high-level/target-side ones; (4) generic pre-training is critical to the emergence of abstraction capability, and PLMs exhibit better abstraction with larger model sizes and data scales.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑