解构和缓解任务相似性对持续学习的影响
Disentangling and Mitigating the Impact of Task Similarity for Continual Learning
AI总结：
该研究通过线性教师-学生模型分析了任务相似性对持续学习的影响，发现基于Fisher信息度量的权重正则化能显著提升知识保留且不损害迁移，为缓解持续学习困难提供了理论见解。
摘要：
部分相似任务的持续学习对人工神经网络构成了挑战，因为任务相似性既提供了知识迁移的机会，也带来了干扰和灾难性遗忘的风险。然而，目前尚不清楚输入特征和读出模式中的任务相似性如何影响知识迁移和遗忘，以及它们如何与常见的持续学习算法相互作用。在此，我们开发了一个具有潜在结构的线性教师-学生模型，并解析地表明，高输入特征相似性加上低读出相似性对知识迁移和保留都是灾难性的。相反，相反的情况则相对无害。我们的分析进一步揭示，依赖于任务的活动门控以牺牲迁移为代价提高了知识保留，而依赖于任务的可塑性门控在过参数化极限下既不影响保留也不影响迁移性能。相比之下，基于Fisher信息度量的权重正则化显著改善了保留，无论任务相似性如何，都不会损害迁移性能。然而，其对角近似和欧几里得空间中的正则化对任务相似性的鲁棒性要差得多。我们在带有潜在变量的置换MNIST任务中展示了这些一致的结果。总体而言，这项工作为持续学习何时困难以及如何缓解它提供了见解。
英文摘要：
Continual learning of partially similar tasks poses a challenge for artificial neural networks, as task similarity presents both an opportunity for knowledge transfer and a risk of interference and catastrophic forgetting. However, it remains unclear how task similarity in input features and readout patterns influences knowledge transfer and forgetting, as well as how they interact with common algorithms for continual learning. Here, we develop a linear teacher-student model with latent structure and show analytically that high input feature similarity coupled with low readout similarity is catastrophic for both knowledge transfer and retention. Conversely, the opposite scenario is relatively benign. Our analysis further reveals that task-dependent activity gating improves knowledge retention at the expense of transfer, while task-dependent plasticity gating does not affect either retention or transfer performance at the over-parameterized limit. In contrast, weight regularization based on the Fisher information metric significantly improves retention, regardless of task similarity, without compromising transfer performance. Nevertheless, its diagonal approximation and regularization in the Euclidean space are much less robust against task similarity. We demonstrate consistent results in a permuted MNIST task with latent variables. Overall, this work provides insights into when continual learning is difficult and how to mitigate it.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
