通过持续学习研究预训练表示中的遗忘现象
Investigating Forgetting in Pre-Trained Representations Through Continual Learning
AI总结：
本文通过三个指标研究持续学习中预训练语言模型的表示遗忘，发现通用性受损且句法语义知识遗忘，并提出先训练通用任务及混合持续学习两种缓解策略。
摘要：
表示遗忘是指在持续训练过程中上下文表示发生的漂移。直观上，表示遗忘可能影响预训练语言模型（LMs）中存储的通用知识，但具体影响尚不清楚。本文研究表示遗忘对预训练语言模型通用性的影响，即应对未来下游任务的潜在能力。具体而言，我们设计了三个指标，包括整体通用性破坏（GD）、句法知识遗忘（SynF）和语义知识遗忘（SemF），以衡量持续学习中通用知识的演变。通过大量实验，我们发现各种预训练语言模型的通用性遭到破坏，且句法和语义知识在持续学习中被遗忘。基于实验与分析，我们进一步获得两个缓解通用知识遗忘的见解：1）首先在通用语言任务上训练可以缓解通用知识遗忘；2）与仅考虑重放或正则化的方法相比，混合持续学习方法可以缓解通用性破坏并保留更多通用知识。
英文摘要：
Representation forgetting refers to the drift of contextualized representations during continual training. Intuitively, the representation forgetting can influence the general knowledge stored in pre-trained language models (LMs), but the concrete effect is still unclear. In this paper, we study the effect of representation forgetting on the generality of pre-trained language models, i.e. the potential capability for tackling future downstream tasks. Specifically, we design three metrics, including overall generality destruction (GD), syntactic knowledge forgetting (SynF), and semantic knowledge forgetting (SemF), to measure the evolution of general knowledge in continual learning. With extensive experiments, we find that the generality is destructed in various pre-trained LMs, and syntactic and semantic knowledge is forgotten through continual learning. Based on our experiments and analysis, we further get two insights into alleviating general knowledge forgetting: 1) training on general linguistic tasks at first can mitigate general knowledge forgetting; 2) the hybrid continual learning method can mitigate the generality destruction and maintain more general knowledge compared with those only considering rehearsal or regularization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
