持续学习中的特征空间退化
Degradation of Feature Space in Continual Learning
AI总结：
本文研究了在持续学习中强制特征空间各向同性是否能提升表示质量，发现其可能降低模型性能，揭示了集中学习与持续学习在特征几何上的本质差异。
摘要：
集中训练是深度学习的标准范式，使模型能够在单一地点从统一的数据集中学习。在这种设置中，各向同性特征分布自然出现，作为支持良好结构和可推广表示的手段。相比之下，持续学习处理的是流式和非平稳数据，并逐步训练模型，本质上面临众所周知的可塑性-稳定性困境。在这种情况下，学习动态倾向于产生越来越各向异性的特征空间。这引发了一个根本性问题：是否应强制各向同性以在稳定性和可塑性之间取得更好的平衡，从而减轻灾难性遗忘？在本文中，我们研究了促进特征空间各向同性是否能提高持续学习中的表示质量。通过使用对比持续学习技术在CIFAR-10和CIFAR-100数据上进行实验，我们发现各向同性正则化未能提高，甚至可能降低持续学习中的模型准确性。我们的结果突显了集中学习和持续学习之间特征几何学的本质差异，表明各向同性虽然在集中设置中有益，但可能不是非平稳学习场景的合适归纳偏置。
英文摘要：
Centralized training is the standard paradigm in deep learning, enabling models to learn from a unified dataset in a single location. In such setup, isotropic feature distributions naturally arise as a mean to support well-structured and generalizable representations. In contrast, continual learning operates on streaming and non-stationary data, and trains models incrementally, inherently facing the well-known plasticity-stability dilemma. In such settings, learning dynamics tends to yield increasingly anisotropic feature space. This arises a fundamental question: should isotropy be enforced to achieve a better balance between stability and plasticity, and thereby mitigate catastrophic forgetting? In this paper, we investigate whether promoting feature-space isotropy can enhance representation quality in continual learning. Through experiments using contrastive continual learning techniques on CIFAR-10 and CIFAR-100 data, we find that isotropic regularization fails to improve, and can in fact degrade, model accuracy in continual settings. Our results highlight essential differences in feature geometry between centralized and continual learning, suggesting that isotropy, while beneficial in centralized setups, may not constitute an appropriate inductive bias for non-stationary learning scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
