Kolmogorov-Arnold网络中的灾难性遗忘
Catastrophic Forgetting in Kolmogorov-Arnold Networks
AI总结：
针对持续学习中KANs灾难性遗忘行为不明确的问题，本研究通过理论框架（关联遗忘与激活支持重叠、内在数据维度）和系统实验，揭示其在低维场景保留知识但高维领域仍易遗忘的特性，并提出KAN-LoRA适配器提升语言模型持续微调效率。
摘要：
灾难性遗忘是持续学习中的长期挑战，模型在学习新任务时会丢失早期任务的知识。尽管针对多层感知机（MLPs）提出了多种缓解策略，但Kolmogorov-Arnold网络（KANs）等近期架构进展被认为通过利用局部样条激活具有内在抗遗忘性。然而，KANs在持续学习下的实际表现仍不明确，其局限性也未被充分理解。为此，我们对KANs中的灾难性遗忘展开全面研究，并开发了将遗忘与激活支持重叠及内在数据维度关联的理论框架。我们通过在合成任务和视觉任务上的系统实验验证这些分析，测量不同模型配置和数据复杂度下的遗忘动态。此外，我们提出KAN-LoRA——一种用于语言模型参数高效持续微调的新型适配器设计，并评估其在知识编辑任务中的有效性。研究发现，KANs在低维算法场景中表现出良好的知识保留，但在图像分类和语言建模等高维领域仍易受遗忘影响。这些结果增进了对KANs优缺点的理解，为持续学习系统设计提供实用见解。
英文摘要：
Catastrophic forgetting is a longstanding challenge in continual learning, where models lose knowledge from earlier tasks when learning new ones. While various mitigation strategies have been proposed for Multi-Layer Perceptrons (MLPs), recent architectural advances like Kolmogorov-Arnold Networks (KANs) have been suggested to offer intrinsic resistance to forgetting by leveraging localized spline-based activations. However, the practical behavior of KANs under continual learning remains unclear, and their limitations are not well understood. To address this, we present a comprehensive study of catastrophic forgetting in KANs and develop a theoretical framework that links forgetting to activation support overlap and intrinsic data dimension. We validate these analyses through systematic experiments on synthetic and vision tasks, measuring forgetting dynamics under varying model configurations and data complexity. Further, we introduce KAN-LoRA, a novel adapter design for parameter-efficient continual fine-tuning of language models, and evaluate its effectiveness in knowledge editing tasks. Our findings reveal that while KANs exhibit promising retention in low-dimensional algorithmic settings, they remain vulnerable to forgetting in high-dimensional domains such as image classification and language modeling. These results advance the understanding of KANs' strengths and limitations, offering practical insights for continual learning system design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
