基于知识蒸馏的结构迁移用于超越稀疏分布式记忆的持续学习
Distillation-Guided Structural Transfer for Continual Learning Beyond Sparse Distributed Memory
AI总结：
本文提出Selective Subnetwork Distillation (SSD)，通过结构引导的持续学习框架，利用知识蒸馏实现跨任务知识重用，提升稀疏持续学习的准确率和保留率。
摘要：
稀疏神经网络因其模块化和低干扰特性，在高效持续学习中受到关注。例如，稀疏分布式记忆多层感知机（SDMLP）通过Top-K激活构建任务特定的子网络，并已显示出对灾难性遗忘的韧性。然而，其刚性的模块化限制了跨任务知识的重用，并在高稀疏度下导致性能下降。我们提出了选择性子网络蒸馏（SSD），一种结构引导的持续学习框架，将蒸馏不视为正则化器，而是视为与拓扑对齐的信息通道。SSD识别激活频率高的神经元，并选择性地在之前的Top-K子网络和输出logits中蒸馏知识，无需回放或任务标签。这使得结构重新对齐的同时保持稀疏模块化。在Split CIFAR-10、CIFAR-100和MNIST上的实验表明，SSD提高了准确率、保留率和表示覆盖度，为稀疏持续学习提供了结构化的解决方案。
英文摘要：
Sparse neural systems are gaining traction for efficient continual learning due to their modularity and low interference. Architectures such as Sparse Distributed Memory Multi-Layer Perceptrons (SDMLP) construct task-specific subnetworks via Top-K activation and have shown resilience against catastrophic forgetting. However, their rigid modularity limits cross-task knowledge reuse and leads to performance degradation under high sparsity. We propose Selective Subnetwork Distillation (SSD), a structurally guided continual learning framework that treats distillation not as a regularizer but as a topology-aligned information conduit. SSD identifies neurons with high activation frequency and selectively distills knowledge within previous Top-K subnetworks and output logits, without requiring replay or task labels. This enables structural realignment while preserving sparse modularity. Experiments on Split CIFAR-10, CIFAR-100, and MNIST demonstrate that SSD improves accuracy, retention, and representation coverage, offering a structurally grounded solution for sparse continual learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
